Python Decorators – A practical example

Hey guys,

I work in DevOps and as you can tell from my blog post I’ve been working with monitoring and system metrics lately. I came across a great python package called psutils. You can find it here: http://pythonhosted.org/psutil/

The psutil tools tend to give disk and memory metrics back in bytes. I don’t know about you but I don’t enjoy doing the conversion from bytes to human readable forms in my head. Their documentation points us to using the following function for conversion:

def bytes2human(n):
    # http://code.activestate.com/recipes/578019
    # >>> bytes2human(10000)
    # '9.8K'
    # >>> bytes2human(100001221)
    # '95.4M'
    symbols = ('K', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y')
    prefix = {}
    for i, s in enumerate(symbols):
        prefix[s] = 1 << (i + 1) * 10 for s in reversed(symbols): 
            if n >= prefix[s]:
                value = float(n) / prefix[s]
                return '%.1f%s' % (value, s)
    return "%sB" % n

I like this function a lot – quick and dirty, however I didn’t want to have to pipe the values that psutil outputs to this function. I instead would like a function that I can call and obtain human readable output without jumping into psutil’s source code.

The perfect tool in the python language for this is to create a decorator (without access to the source code)

I’ll be working with the function:
psutil.virtual_memory()

This function returns a named tuple with values, here’s an example:

>>> psutil.virtual_memory()
svmem(total=8048656384, available=2548318208, percent=68.3, used=7201034240, free=847622144, active=4398604288, inactive=2348535808, buffers=84848640, cached=1615847424)

Now what we want to do is extract the keys and the values from this output, convert the values to human readable, zip the keys and values back up into a dictionary and return said dictionary. For this we are going to create a decorator.

What is a decorator ?

  1. A decorator takes in a function.
  2. It then defines a function which manipulates the output of the original passed in function and returns the manipulated object.
  3. After the above function is defined, the enclosing function (the decorator itself) returns the defined function object.

That last part is key here – a decorator TAKES IN a function and RETURNS a function – a decorator does not directly give you the manipulated results.

So let’s see some code. In this example I’ve already declared “bytes2human” function.

def convert_humanreadable(virtualmemory): 
    def wrapper(): 
        tuple = virtualmemory()
        keys = tuple._fields
        values = [ bytes2human(value) for value in tuple ]
        return dict(zip(keys, values)) s
    return wrapper

So let’s compare the above example with my definition of a decorator.

The first thing we do here is declare our decorator function. It’s going to take in a “virtualmemory” object which will be the psutil.virtual_memory function.

Next we immediately define a function which is going to manipulate the output of the passed in function (psutil.virtual_memory). We take the tuple that we get when we run psutil.virtual_memory() and place it into a variable “tuple”. We then extract the keys from the tuple. We then use a list comprehension to take each value in tuple and convert it to a human readable form. We then give the instruction to return a dictionary of the keys and the new human readable values.

We wrap up our decorator by returning the function we defined above. Remember, the decorator itself returns a function, you use this new function to obtain the post-manipulated human-readable values.

So how do we use it?

virtual_memory_hr = convert_humanreadable(psutil.virtual_memory)
virtual_memory_hr() 
{'available': '2.6G', 'buffers': '82.8M', 'total': '7.5G', 'cached': '1.5G', 'inactive': '2.1G', 'active': '4.0G', 'free': '1.0G', 'percent': '65.1B', 'used': '6.5G'}

There you have it, now I have a function I can run to output everything in a human readable format.

Now if I felt so inclined, I could download the source code for psutil, define my decorator function along with the bytes2human function in the source and decorate each function. I’ll give a small example below (source at: https://github.com/giampaolo/psutil/blob/master/psutil/_pslinux.py#L169)

@convert_humanreadable
def virtual_memory():
    total, free, buffers, shared, _, _ = cext.linux_sysinfo()
    cached = active = inactive = None
    with open('/proc/meminfo', 'rb') as f:
        for line in f:
            if line.startswith(b"Cached:"):
                cached = int(line.split()[1]) * 1024
            elif line.startswith(b"Active:"):
                active = int(line.split()[1]) * 1024
            elif line.startswith(b"Inactive:"):
                inactive = int(line.split()[1]) * 1024
            if (cached is not None and
                    active is not None and
                    inactive is not None):
                break
        else:
            # we might get here when dealing with exotic Linux flavors, see:
            # https://github.com/giampaolo/psutil/issues/313
            msg = "'cached', 'active' and 'inactive' memory stats couldn't " \
                  "be determined and were set to 0"
            warnings.warn(msg, RuntimeWarning)
            cached = active = inactive = 0
    avail = free + buffers + cached
    used = total - free
    percent = usage_percent((total - avail), total, _round=1)
    return svmem(total, avail, percent, used, free,
                 active, inactive, buffers, cached)

The @convert_humanreadable is basically syntaxual sugar for:

virtual_memory_hr = convert_humanreadable(psutil.virtual_memory)
virtual_memory_hr()

The difference is the @decorator syntax re-maps the output of our decorator to the original (decorated) function name. Which would look like this:

virtual_memory = convert_humanreadable(virtual_memory)
virtual_memory()

Python takes the original function, places that in the decorator, and then maps the original function name to the function the decorator returns. This… is a little bit of a brain twister, but ultimately allows us to continue to call psutil.virtual_memory() in our programs and not the decorator itself.

Hope this gives a good example of when to use decorators and how they can elegantly solve specific problems.

Advertisements

One thought on “Python Decorators – A practical example

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s