Logging is indispensable when it’s needed, but too much logging can be a drain on performance and resources when everything is going smoothly and you don’t need really fine-grained log data.
Rather than adding more machines and buying fatter pipes, the largest apps I’ve worked on have all implemented some form of dynamic level control. In this system, logging usually defaults to something sensible (my preference is Information) and can be turned up or down from a central control panel when bugs need to be shaken out.
A service bus is usually the tool of choice for implementing it: publish a message like new LoggingLevelChange { App = "Ordering", Level = "Debug" } and the participating apps will notice, say “Hey, I do ordering!” and update their logging level accordingly.
There are always a few challenges to overcome before it all works smoothly. Ideally, every app would get this capability, but it’s not always high enough on the backlog to get done early on.
Along with the next Serilog version is a small change in the Seq sink that I’m really excited about. By sharing a LoggingLevelSwitch between the logging pipeline and Seq sink, it’s now possible to control the logging levels of connected apps easily — efficiently — from the Seq server.
Log.Logger = new LoggerConfiguration()
.MinimumLevel.ControlledBy(levelSwitch)
.WriteTo.LiterateConsole()
.WriteTo.Seq("http://localhost:5341",
apiKey: "yeEZyL3SMcxEKUijBjN",
controlLevelSwitch: levelSwitch)
.CreateLogger();
In this configuration, levelSwitch is passed to the MinimumLevel.ControlledBy() method and the WriteTo.Seq() statement.
Now, whenever a batch of events is sent to Seq, the logging level associated with the corresponding API key will be returned by Seq (or at two minute intervals, when things are quiet). The Seq sink will apply this to controlLevelSwitch, so the whole logging pipeline will be adjusted up or down.
Why is this so much better than what was possible before? Seq has had per-API key level control for a while, but this was only used as a filter: log events had to be generated at the highest required level and then discarded for the scheme to work effectively, using more CPU and memory in the client application than was strictly required. Now that the Seq sink can control the level of the logging pipeline, events that aren’t required are not generated in the first place.
Pre-release versions of the Seq sink including this feature work today on .NET 4.5+, but a pre-release Serilog 2.0 version is required as well. Depending on which other Serilog packages you use, this may or may not be possible to try out in your application. Some details of Serilog 2.0 status can be found in this post.
If you’re able to check it out I’d love to hear from you.