Python Logging 101 - Filters And Handlers

Python logging consists of a few core components that we’re required to interact with. While setting up a logger does not require us to interact with most of these, we eventually end up having to tweak the default configurations to get what we need for a production grade deployment.

The logging library comprises of a few modules namely,

1. Handlers (logging.Handler)

2. Filters (logging.Filter)

3. Formatters (logging.Formatter)

4. And the Logger itself. (logging.Logger)

Each log event generates a LogRecord (logging.LogRecord) which is the unit of operation. It’s the data that we want to publish to the logs. The goal is to publish the LogRecord object to the stream (or whatever output that is defined by the handler).

Handlers let you define a destination for the LogRecord with a few additional features. The logging module provides a few, out of the box like the StreamHandler, FileHandler, SocketHandlers, SysLogHandlers and several variants of RotatingFileHandlers.

So, the primary task for the Handlers is to format the event using a Formatter, identify if the emitted LogRecord should be published to the destination using a Filter and finally publishing it to the destination.

I try to imagine the handlers to be a set of independent, encapsulation of rules and attributes that are required to publish to a particular destination. The tricky bit to remember is that they function independent of each other. This is especially critical when you’re dealing with Filters

Filters

Filters are simple conditions that determine if an emitted LogRecord can be published to the destination. I think of it as logging middleware :) which decides whether the event (LogRecord) should be forwarded to the destination.

Filters associated with a Handler are attached exclusively to the Handler. Let’s look at an example

Setting Up A Basic Logger

Here we create a single handler that logs to STDOUT using the StreamHandler and we attach a custom formatter to it.

Filter All Uptime Checks From Our Logs

The most common problem statement that one would encounter is to avoid flooding production logs with uptime check messages. So to handle that we create a class that will check if the LogRecord object is logging an uptime check request. If so, it skips passing it forward to the destination. It forwards all other log events to the destination, which in our case is STDOUT via the StreamHandler.

Let’s Attach Our Filters To Our Handlers

Notice we added the filter to the stream_handler instead of the logger. Now, if we run a small loop, we can see our filter in action.

It skipped our /GET up calls as expected.

We can also add the same filter to the logger itself and see the same results.

Adding A Second Handler

Now, running the script will show the same logs on the file and on STDOUT.

When We Don’t Want To Filter FileLogs

In the scenario where we don’t want to filter the logs on the file, we attach the Filter to the Handler as before.

And, now we see the filtered logs on STDOUT and unfiltered logs on the file.