SaltStack: Using Grains to reliably mount AWS Instance Block Devices | Part 1

Note: This post requires novice knowledge of terraform

I’m a huge fan of SaltStack and one of the reasons why is the ability to drop into Python when a more powerful solution is necessary. I used this ability in order to reliably mount EBS and Ephemeral volumes on AWS instances. If you’re not familiar with this problem let me explain.

When launching an AWS instance and supplying it with a block device configuration, AWS creates this mapping within their API. We can query what AWS believes this mapping to be via the meta-data service rest API. This API is accessible from the instance itself and can be used to query information about itself from the view of AWS’s api. If you want a quick example log into ssh on an aws instance and try this:

services/root@ip-10-10-1-168:/home/ubuntu# curl 169.254.169.254/latest/meta-data/block-device-mapping/
ami
ebs1
ephemeral0
ephemeral1
ephemeral2
ephemeral3
ephemeral4
ephemeral5
ephemeral6
ephemeral7

root@ip-10-10-1-168:/home/ubuntu# curl 169.254.169.254/latest/meta-data/block-device-mapping/ebs1
sdf

This provides a great way for us to query data about the instance, however this data is not always correct. When querying for block devices the AWS api has it’s perceived mapping, but the linux kernel can name the drive with a different prefix. We can always be sure however that the last character of the mapping is consistent. You can read more about block device mapping here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html

So with this information, wouldn’t it be great to be able to always know what the correct mapping was based on the instance’s perspective? The perfect tool for this is SaltStack’s grains.

Grains are facts provided by the instance\host itself. They are small python programs which return a dictionary to the salt-minion agent. When the master queries the host’s grains it converts the dictionary to a yaml representation, and we are free to use those keys and values in our states. Since this is all just python, we can easily create custom grains for our salt master to use.

In order to slip stream this tutorial, I created a git repo here: https://github.com/ldelossa/salt-grains-tutorial

This repo holds a terraform file, the grains file, and a sample salt bootstrap file just incase you were curious about boot-strapping salt from bash (we use curl in our provisioning steps). Since this post’s focus is on SaltStack, I will not explain the terraform file in details, but I will provide some instruction.

The terraform file creates a full test VPC and all the necessary components needed for connectivity EXCEPT for a key pair used for instance connectivity. I expect you to generate your own key pair, download the pem file, and place the path to the pem within this terraform configuration. You will need to fill in data at the following lines

#lines 10-14
provider "aws" {
    access_key = "" # your aws access key
    secret_key = "" # your aws secret key
    region = "" # region at which the vpc will be launched
}

#lines 170, 178, 258
private_key = "${file("")}" #Place the path to your downloaded key within the quotes

You may notice at the top of the file there’s  a variable declaration called local_public_ip. Terraform is going to ask you for your public IP-Address for security reasons. I’ve locked down connectivity to our instance based on your workstation’s public ip because… you know security. If you want to get fancy here’s how I run the terraform apply command.

salt-grains-tutorial$ TF_VAR_local_public_ip=$(curl -s ipinfo.io/ip) terraform apply

All this is doing is running the terraform apply command with an ad-hoc environment variable which maps to the defined variable local_public_ip in our salt-tutorial.tf file. The curl command hits a useful API end point which reports back your public IP address. If this is a tab bit over your head, just run terraform apply and google your public ip address when prompted.

Okay so now, once terraform is done doing it’s thing it should output two IP addresses for you, a salt master (salt-master01) and a client server (server01). Let’s log into salt-master01 to get things going.

Let’s make sure we see our minion and accept it’s key

root@ip-10-10-1-168:/home/ubuntu# salt-key
Accepted Keys:
Denied Keys:
Unaccepted Keys:
server01
Rejected Keys:

root@ip-10-10-1-168:/home/ubuntu# salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
server01
Proceed? [n/Y] Y
Key for minion server01 accepted.

Okay now with our minion registered with our salt mater, let’s take a look at the custom grain that we uploaded to our salt-master. All custom grains by default reside in /srv/salt/_grains. Let’s take a look at our custom grain block-devices.py

import httplib
import os

DEVICE_MAPPING_URI = '/latest/meta-data/block-device-mapping/'

def detect_devs():
    dev_list = [ x for x in os.listdir('/sys/block')
                 if x.startswith('xvd') or x.startswith('sd') ]
    return dev_list

def _metadata_call(url):
    try:
        conn = httplib.HTTPConnection("169.254.169.254", 80, timeout=1)
        conn.request('GET', url)
        response = conn.getresponse()
        if response.status != 200:
            return

        return response.read()
    except:
        return

def _get_block_devices():
    block_device_grain = { 'ephemeral': [], 'ebs': [] }
    detected_devs = detect_devs()

    for mapping in _metadata_call(DEVICE_MAPPING_URI).split('\n'):
        device = _metadata_call(DEVICE_MAPPING_URI + mapping)

        if mapping.startswith('ephemeral'):
            for dev in detected_devs:
                if dev[-1] == device[-1]:
                    block_device_grain['ephemeral'].append(dev)

        elif mapping.startswith('ebs'):
            for dev in detected_devs:
                if dev[-1] == device[-1]:
                    block_device_grain['ebs'].append(dev)
    return block_device_grain


def main():
    grains = {}
    grains['block_devices'] = _get_block_devices()
    return grains

So what’s going on here? First thing we do is set a constant variable indicating AWS’s meta-data URI that provides us the registered block device mappings for the instance. This is how we will find what the AWS API assumes is mounted on the host.

Next we define a function which simply returns a list. We use a list comprehension to parse out devices which begin with either xvd or sd from the directory /sys/block. We wind up with a list of devices from the instance’s perspective. We know that his list always contains the valid block device names!

We quickly define a wrapper for making http calls to the AWS Metadata instance, this should be self explanatory.

We then get to the actual work. The logic behind this grain is as follows:

• Create a dictionary for holding lists of ephemeral and ebs devices. This list will become the grain data presented to our salt master.
• Create the list of detected drives using the defined function above
• For ever block-device mapping that the meta-data service returns, do an additional lookup on this mapping giving us AWS’s perspective of what the block device name is
• If the mapping contained the word ‘ephemeral’ then check the last character of the drive name, if the last character matches a disk within the detected_devs list, append the detected_dev into the block_device_grains[‘ephemeral’] list.
• Do the same thing logic as above, but with ‘ebs’ mapping name.
• Return the block_device_grain

The last main() is the entry point for the grain. This is what the salt-minion agent will run when collecting grain information. We are using the top level tag of ‘block_devices’ for our grain, and then the dictionary information will populate underneath. I will show the yaml a little later in the post.

Okay! So hopefully that makes sense to you guys. Let’s take a look at how this works. The first thing we want to do is make sure our salt-master01 can talk to server01. Issue the following command:

root@ip-10-10-1-168:/home/ubuntu# salt 'server01' test.ping
server01:
    True

If you do not receive “True” then there is a connectivity issue and you will need to troubleshoot further. Next we need to sync the custom grain to our client

root@ip-10-10-1-168:/home/ubuntu# salt 'server01' saltutil.sync_all
server01:
    ----------
    beacons:
    grains:
        - grains.block-devices
    log_handlers:
    modules:
    output:
    proxymodules:
    renderers:
    returners:
    sdb:
    states:
    utils:

With our grain synced we can now try to receive the custom information!

root@ip-10-10-1-168:/home/ubuntu# salt 'server01' grains.get block_devices
server01:
    ----------
    ebs:
        - xvdf
    ephemeral:
        - xvdd
        - xvde
        - xvdf

Alright! We now have a consistent and reliable way to identify the correct naming of the block device from the instances\host perspective. This makes our state writing in the future a lot simpler, and equates to less jinja2 logic.

In part2 I’m going to go over how we can use these values and create reusable state which mount our block devices in a extensible fashion.

 

Python – A great class called defaultdict

A good practice that I’ve picked up in python (or any language) is to constantly refactor your code. This allows you to take a look at how you accomplished a task in the past and determine if today, you could accomplish this in a better way.

I was doing just this when I came along a snippet of code which was responsible for taking a list of dictionaries and printing out only the dictionaries which contained a specific value for a specific key.

My original idea was to run for loops. I will show the code below

First: A sample record:

{'threshold': 0, 'dev': 'sda', 'metric': 'wkB/s', 'time': 'Dec, Sun 2015 18:49:39', 'hostname': 'ldubuntu', 'current_value': 52.0}

Now now the code block

def dump_metrics(q, disk=None):
    alerts = [alert for alert in q]
    alerts.sort(key=itemgetter('time'))

    if any([disk == d for d in extract_disk_names()]):
        for alert in alerts:
            if alert['dev'] == disk:
                print(alert)
    else:
        for alert in alerts:
            print(alert)

The function ‘extract_disk_names()’ just returns a list of disks on the host system i.e [ ‘sda’, ‘sdb’, ‘sdc’ ]. If no disk was supplied in the function call, or the disk wasn’t recognized we move down to the else block. (I’m using the cmd module to handle the shell aspect of this program, so handling arguments is a little funky)

So this was all good, it worked, but it’s a little limiting. What if I wanted to dump all our metrics but keep them ordered by each device? Sure we can use more for loops and hack at it, but that’s not elegant.

In my pursuit of refactoring this code I found a really interesting class in the collections module. This class is called defaultdict.

I suggest you reference the class here:
https://docs.python.org/2/library/collections.html#collections.defaultdict

Let’s use their example for to explain what defaultdict is doing:

>>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
...     d[k].append(v)
...
>>> d.items()
[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]

So before anything lets look at our input and output. We start this block of code with a list of tuples. If we look at these tuples as a key value pair, it becomes apparent that we have duplicate keys.

Now what do we end up with? We end up with a consolidation. We wind up with a single unique key, and any values that were present in the duplicate are now present in a list accompanying the key as it’s value.

It’s not a coincidence that we wind up with a list as our value. As you can see we instantiated defaultdict with the factory object being a list.

Before we do a step by step walk through let’s go through a high level of how defaultdict works.

Defaultdict is very similar to a dictionary object but includes a specific handler for what happens when keys are not present. On a normal dictionary object, if you attempt a key lookup and there is no mapping you get an exception; this is not the case with a defaultdict. When using a defaultdict and presenting it with an unknown key, the defaultdict will create the key mapping along with an empty variant of the object you specified as the factor for the value. It does this by taking advantage of the fact that __getitem__() calls __missing__().

When we attempt to do a key lookup

value = dictionary['key']

Behind the scenes the dictionary calls __getitem__(). If the key is not mapped, once again behind the scenes, __missing__() is called. At this point, defaultdict has instructions in __missing__() to supply the empty variant of the object we instantiated it with, which would be the list object.

Okay so let’s walk through what’s going on here:

1) we have ‘s’ defined with a list of tuples, in what we assume is a key/value relationship

2) we create a defaultdict object which the factory object is a list. We are telling defaultdict that whenever we give you a key with no mapping, map an empty list to our supplied key

3) we enter a for loop over each item in ‘s’ with the key set to variable ‘k’, and the value set to variable ‘v’.

4) our first iteration of the loop attempts to do a lookup of key ‘yellow’ on our defaultdict ‘d’. This mapping is not there. What we return after this lookup is consequently this datastructure: { ‘yellow’: [] }. Now when we return from the lookup, we append v to this empty list, resulting in: { ‘yellow’: [1] }

5) the same above happens for blue

6) Now we get to a yellow again. When we do a the key lookup for ‘yellow’, we now have a valid lookup! We thus append ‘3’ to the list just like we would in any normal situation.

So let’s play around with the code:

#Let's see what happens if we don't issue the append statement
>>> from collections import defaultdict
>>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
...     d[k]
...     
[]
[]
[]
[]
[]
>>> d
defaultdict(<class 'list'>, {'blue': [], 'red': [], 'yellow': []})

The above is exactly what we would expect. Since we never did any appends, we wind up with our keys consolidated, but no values.

So now taking this idea and applying it to my problem above, we can infer that this gives us a very elegant way of displaying our data. My idea was to create multiple dictionaries which only held records which pertained to their key. For example, the dictionary ‘sda’ would have a value of a list, which held multiple metric records for that device.

I was able to accomplish this by the following code snip:

def dump_metrics(q, disk=None):
    alerts = [alert for alert in q]
    alerts.sort(key=itemgetter('time'))

    dev_dict = defaultdict(list)
    for alert in alerts:
        dev_dict[alert['dev']].append(alert)

    if any([disk == d for d in extract_disk_names()]):
        print(disk)
        for i in dev_dict[disk]:
            print('     ', i)

    else:
        for key in dev_dict:
            print(key)
            for i in dev_dict[key]:
                print('     ', i)

So what do we have here?

1) I dump all my metrics from my queue into an alerts list.
2) I sort all my metrics by datetime
4) I create a new defaultdict with the factory object being a list
5) I begin a for loop over all my alerts
6) I attempt a lookup of a key, if that key does not exist in dev_dict, a mapping for that key will be created with an empty list
5) the alert is appended to the empty list

The rest is just logic for making sure a valid disk is provided to with the command. If it’s not, I simply dump all my metrics.

Here’s a sample of the output:

(Cmd) dump_metrics
sdb
      {'time': 'Dec, Sun 2015 19:57:27', 'dev': 'sdb', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 92.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:57:34', 'dev': 'sdb', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 26.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:09', 'dev': 'sdb', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 156.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:23', 'dev': 'sdb', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 14.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:44', 'dev': 'sdb', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 60.0, 'hostname': 'ldubuntu'}
sda
      {'time': 'Dec, Sun 2015 19:57:27', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 6.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:57:34', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 6.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:57:41', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 10.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:02', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 44.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:16', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 4.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:23', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 6.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:31', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 2.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:38', 'dev': 'sda', 'threshold': 0, 'metric': 'avgqu-sz', 'current_value': 0.01, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:38', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 92.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:52', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 10.0, 'hostname': 'ldubuntu'}

(Cmd) dump_metrics sda
sda
      {'time': 'Dec, Sun 2015 19:58:23', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 6.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:31', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 2.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:38', 'dev': 'sda', 'threshold': 0, 'metric': 'avgqu-sz', 'current_value': 0.01, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:38', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 92.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:58:52', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 10.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:59:06', 'dev': 'sda', 'threshold': 0, 'metric': 'avgqu-sz', 'current_value': 0.01, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:59:06', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 46.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:59:13', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 6.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:59:20', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 14.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:59:27', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 6.0, 'hostname': 'ldubuntu'}
      {'time': 'Dec, Sun 2015 19:59:35', 'dev': 'sda', 'threshold': 0, 'metric': 'wkB/s', 'current_value': 6.0, 'hostname': 'ldubuntu'}

As you can see this is a much more elegant way of handling complex output verses using a bunch of for loops. We have both grouped by date, and by device.

What makes this module really interesting is the fact that you can supply any factory object to default disk. You could write your own factory object and have them mapped to each key. Pretty cool and powerful.

Hope this is informative for all you guys.

Docker Lab – Networking

As I begin to dig deeper into placing docker into production I find myself needing an elaborate lab. I personally use KVM on a Ubuntu system but this should apply to any hypverisor which uses Linux bridges to accomplish networking.

I wanted end-to-end connectivity from my host system to containers that were being hosted in a virtual machine. For example, I should be able to create a VM (from my KVM Host machine) named dockerhost01 and a containers in this machine. I should be able to ping the container from my KVM HOST machine. With the default routing table this will not work.

In order to aid in representing our configuration I have a crude hand-written diagram. I’m far to lazy to start up my windows VM just to get visio going so I hope this will do justice.

In the image you can see we have a workstation with a wlan adapter at 192.168.0.104. This workstation also has the default virtual bridge that KVM creates. We can think of a virtual bridge as a “switch” which lives on our host. We are, in essence, turning our workstation into a multi-port switch. Each virtual machine we power on literally “plugs” there interface into this virtual switch named virtbr0. I will return back to linux virtual bridges later in the post.

Next you can see that we have a KVM VM named DockerHost. This VM itself has another linux virtual bridge named Docker0. This is how containers communicate with their host machine. In the same fashion, our containers “plug” there interfaces directly into Docker0.

Now if this is your first introduction to linux bridges you may be slightly confused. I would suggest you play with them a little to see how they work. It’s a simple construct but conceptually can be slightly confusing. Let’s download the bridge-utils tools and inspect our switches

#Workstation machine
root@ldubuntu:/home/ldelossa# brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.5254007de659       yes             virbr0-nic
                                                        vnet0
                                                        vnet1
                                                        vnet2
                                                        vnet3
                                                        vnet4
root@ldubuntu:/home/ldelossa# 

What we see here is the default linux bride that KVM creates. You can see that each VM I have created on my workstation is a “plugged-in” interface on the bridge. The bridge has an IP address also

root@ldubuntu:/home/ldelossa# ip addr list
5: virbr0:  mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:54:00:7d:e6:59 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
6: virbr0-nic:  mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 500
    link/ether 52:54:00:7d:e6:59 brd ff:ff:ff:ff:ff:ff
7: vnet0:  mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 500
    link/ether fe:54:00:1a:57:13 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe1a:5713/64 scope link 
       valid_lft forever preferred_lft forever
8: vnet1:  mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 500
    link/ether fe:54:00:61:5d:99 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe61:5d99/64 scope link 
       valid_lft forever preferred_lft forever
10: vnet2:  mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 500
    link/ether fe:54:00:ba:3c:e3 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:feba:3ce3/64 scope link 
       valid_lft forever preferred_lft forever
11: vnet3:  mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 500
    link/ether fe:54:00:88:b8:44 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe88:b844/64 scope link 
       valid_lft forever preferred_lft forever
12: vnet4:  mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 500
    link/ether fe:54:00:63:42:aa brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe63:42aa/64 scope link 
       valid_lft forever preferred_lft forever

This IP address is used in NAT operations. Any node trying to access the 192.168.122.0/24 network OUTSIDE of our workstation, would need to first send packets to our workstation, then our workstation sends packets to our Linux bridge.

And here is where things get confusing if you are not used to Linux bridging. One would think that on our workstation, if we tried to trace the path to our DockerHostVM, we would see the Linux bridge’s interface IP as a hop. However we must keep in mind that our workstation IS the Linux bridge. There’s no hop necessary. We are directly connected to the 192.168.122.0/24 network simply by BEING the bridge.

Okay, if that doesn’t make too much sense don’t worry. The above can be demonstrated the same way for DockerHostVM. The same concepts are at play, however docker calls its Linux bridge docker0.

So if you look at our diagram, what is the main goal we need to accomplish? We need to get packets originating from HostWorkStation, destined for 172.17.0.1 to DockerHostVM, and then to the container. The reply packets must follow the same path up, through DockerHostVM, and back to HostWorkstation.

So let’s take a look at what we known. We know that we are going to need both Linux machines to act as a packet forwarding router. So let’s do this first, enable packet forwarding on both machines:

Run the following commands on both machines:

sysctl -w net.ipv4.ip_forward=1

Okay cool, so now we have the mechanisms which will allow us to forward packets based on our routing table entries. That last statement was a hint on where to go next. Let’s get an idea of what the routing tables look like on each machine in play.

#HostWorkStation
ldelossa@ldubuntu:~$ ip route
default via 192.168.0.1 dev wlan0  proto static  metric 600 
192.168.0.0/24 dev wlan0  proto kernel  scope link  src 192.168.0.104  metric 600 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 

#DockerHostVM
[root@dockerhost01 ~]# ip route
default via 192.168.122.1 dev ens3  proto static  metric 100 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.122.0/24 dev ens3  proto kernel  scope link  src 192.168.122.11  metric 100 

root@22e95e83a211:/# ip route
default via 172.17.0.1 dev eth0 
172.17.0.0/16 dev eth0  proto kernel  scope link  src 172.17.0.2 

Okay so let’s look at this information from bottom up. First the container’s routing table. Not much going on here, which is fine. All traffic leaving the container is going to head the only way it can – out it’s one interface which we know is virtually “plugged” into docker0.

Next we take a look at our DockerHostVM routing table. Here we have a little more complexity but still not bad. We know to direct packets heading to the container’s ip ranges (172.17.0.0/16) toward the docker0 bridge. We also know that we are directly connected to 192.168.122.0/24 via our ens3 interface. This interface is “plugged” into our virtb0 bridge on our HOST machine. So ANY packets that we aren’t sure where the destination is, we send up to our linux bridge on HostWorkStation.

Now we have our HostWorkStation. This is our point of interest. My workstation’s default route is going to send all unknown packets out my wireless lan interface; Which is appropriate, that’s where the internet is and consequently where we should send any unknown packets. Next we have a route for our VM network (192.168.122.0/24) directing any packets that need to go to our VMs to head to virbr0 interface. This is all great and dandy, but what are we missing?

We need to tell our HostWorkStation to send packets destined for our docker container somewhere other than out my wireless lan interface. Right now there’s no routes for 172.17.0.0/16, hence those packets will head out wlan0, and die. So where do we need to send those packets? My first inclination was to send those packets to our bridge interface, 192.168.122.1 – however this is incorrect. We need to remember that our HostWorkStation IS! the bridge. The bridge interface is not a separate device with it’s own routing table, is literally is the linux machine we are running. Therefore we want to route packets to the next device who knows how to get to our docker containers, we want to route packets to DockerHostVM (192.168.122.11)

Let’s do just that

#HostWorkStation
ip route add 172.17.0.0/16 via 192.168.122.11

Let’s test connectivity

ldelossa@ldubuntu:~$ ping 172.17.0.2 
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=63 time=0.815 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=63 time=0.466 ms
64 bytes from 172.17.0.2: icmp_seq=3 ttl=63 time=0.550 ms
64 bytes from 172.17.0.2: icmp_seq=4 ttl=63 time=0.539 ms
64 bytes from 172.17.0.2: icmp_seq=5 ttl=63 time=0.396 ms
64 bytes from 172.17.0.2: icmp_seq=6 ttl=63 time=0.452 ms
64 bytes from 172.17.0.2: icmp_seq=7 ttl=63 time=0.554 ms
^V64 bytes from 172.17.0.2: icmp_seq=8 ttl=63 time=0.568 ms

Very nice!

So the full picture, how does this work?

#from workstation to container
1) We generate a ping from our workstation toward 172.17.0.2
2) Workstation looks at it’s routing table and says “Okay I want to send to 172.17.0.2, no problem I’ll send these packets over to 192.168.122.11”
3) The networking stack then determines where to source this ping from. It determines that it has an interface on the 192.168.122.0/24 network, the bridge, and sends the ping out this interface, over to 192.168.122.11 sourced from 192.168.122.1. (this step is exactly why we do not see 192.168.122.1 as a hop, the networking stack owns the bridge, and can source packets from this bridge directly)
4) Our DockerHostVM (192.168.122.11) gets this packet with the destination of 172.17.0.2
5) Our DockerHostVM does a routing table lookup and finds it’s route for 172.17.0.0/16 going toward docker0 and sends the packet that way.
6) The packet is delivered via the bridge to the correct container

Let’s see this with tcpdump. We will initiate a ping from our HostWorkStation toward 172.17.0.2. We will run tcpdump on both how DockerHostVM and our container

#HostWorkStationVM
ldelossa@ldubuntu:~$ ping 172.17.0.2
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=63 time=0.613 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=63 time=0.665 ms
64 bytes from 172.17.0.2: icmp_seq=3 ttl=63 time=0.477 ms
64 bytes from 172.17.0.2: icmp_seq=4 ttl=63 time=0.650 ms
64 bytes from 172.17.0.2: icmp_seq=5 ttl=63 time=0.535 ms
64 bytes from 172.17.0.2: icmp_seq=6 ttl=63 time=0.583 ms
64 bytes from 172.17.0.2: icmp_seq=7 ttl=63 time=0.632 ms
64 bytes from 172.17.0.2: icmp_seq=8 ttl=63 time=0.448 ms
64 bytes from 172.17.0.2: icmp_seq=9 ttl=63 time=0.611 ms

#DockerHostVM
[root@dockerhost01 ~]# tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on docker0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:43:47.053790 IP 192.168.122.1 > 172.17.0.2: ICMP echo request, id 24412, seq 1, length 64
23:43:47.053956 IP 172.17.0.2 > 192.168.122.1: ICMP echo reply, id 24412, seq 1, length 64
23:43:47.055546 IP 172.17.0.2.55398 > dns02.kvm.lan.domain: 8571+ PTR? 1.122.168.192.in-addr.arpa. (44)
23:43:47.056461 IP dns02.kvm.lan.domain > 172.17.0.2.55398: 8571 NXDomain* 0/1/0 (99)
23:43:47.056987 IP 172.17.0.2.36237 > dns02.kvm.lan.domain: 25760+ PTR? 3.122.168.192.in-addr.arpa. (44)
23:43:47.058016 IP dns02.kvm.lan.domain > 172.17.0.2.36237: 25760* 1/2/2 PTR dns02.kvm.lan. (137)
23:43:48.052682 IP 192.168.122.1 > 172.17.0.2: ICMP echo request, id 24412, seq 2, length 64
23:43:48.052936 IP 172.17.0.2 > 192.168.122.1: ICMP echo reply, id 24412, seq 2, length 64
23:43:49.051689 IP 192.168.122.1 > 172.17.0.2: ICMP echo request, id 24412, seq 3, length 64
23:43:49.051818 IP 172.17.0.2 > 192.168.122.1: ICMP echo reply, id 24412, seq 3, length 64
23:43:50.050764 IP 192.168.122.1 > 172.17.0.2: ICMP echo request, id 24412, seq 4, length 64
23:43:50.050964 IP 172.17.0.2 > 192.168.122.1: ICMP echo reply, id 24412, seq 4, length 64
23:43:51.050628 IP 192.168.122.1 > 172.17.0.2: ICMP echo request, id 24412, seq 5, length 64
23:43:51.050728 IP 172.17.0.2 > 192.168.122.1: ICMP echo reply, id 24412, seq 5, length 64
23:43:52.050675 IP 192.168.122.1 > 172.17.0.2: ICMP echo request, id 24412, seq 6, length 64
23:43:52.050817 IP 172.17.0.2 > 192.168.122.1: ICMP echo reply, id 24412, seq 6, length 64

#Container
[root@dockerhost01 ~]# docker exec -it nipap-psql01 /bin/bash
root@22e95e83a211:/# tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
04:43:47.053812 IP 192.168.122.1 > 22e95e83a211: ICMP echo request, id 24412, seq 1, length 64
04:43:47.053953 IP 22e95e83a211 > 192.168.122.1: ICMP echo reply, id 24412, seq 1, length 64
04:43:47.055538 IP 22e95e83a211.55398 > dns02.kvm.lan.domain: 8571+ PTR? 1.122.168.192.in-addr.arpa. (44)
04:43:47.056467 IP dns02.kvm.lan.domain > 22e95e83a211.55398: 8571 NXDomain* 0/1/0 (99)
04:43:47.056981 IP 22e95e83a211.36237 > dns02.kvm.lan.domain: 25760+ PTR? 3.122.168.192.in-addr.arpa. (44)
04:43:47.058036 IP dns02.kvm.lan.domain > 22e95e83a211.36237: 25760* 1/2/2 PTR dns02.kvm.lan. (137)
04:43:48.052782 IP 192.168.122.1 > 22e95e83a211: ICMP echo request, id 24412, seq 2, length 64
04:43:48.052933 IP 22e95e83a211 > 192.168.122.1: ICMP echo reply, id 24412, seq 2, length 64
04:43:49.051751 IP 192.168.122.1 > 22e95e83a211: ICMP echo request, id 24412, seq 3, length 64
04:43:49.051816 IP 22e95e83a211 > 192.168.122.1: ICMP echo reply, id 24412, seq 3, length 64
04:43:50.050798 IP 192.168.122.1 > 22e95e83a211: ICMP echo request, id 24412, seq 4, length 64
04:43:50.050960 IP 22e95e83a211 > 192.168.122.1: ICMP echo reply, id 24412, seq 4, length 64
04:43:51.050651 IP 192.168.122.1 > 22e95e83a211: ICMP echo request, id 24412, seq 5, length 64
04:43:51.050726 IP 22e95e83a211 > 192.168.122.1: ICMP echo reply, id 24412, seq 5, length 64
04:43:52.050706 IP 192.168.122.1 > 22e95e83a211: ICMP echo request, id 24412, seq 6, length 64
04:43:52.050814 IP 22e95e83a211 > 192.168.122.1: ICMP echo reply, id 24412, seq 6, length 64

We have successfully achieved end-to-end connectivity from our KVM host to our docker containers being hosted on a machine. This now allows us to lab with the more exciting aspects of docker such as, multi-docker host overlay networking, CI deployments, orchestration, and logging just to name a few.

Hope this helped anyone who was looking to perform a similar set-up.

2 weeks of salt – a practical example

I’ve decided to take on a greenfield deployment of saltstack. I’ve used puppet in the past and I don’t really enjoy learning a DSL that’s not applicable in any other area. I’ve also worked with Ansible which is a great tool for quick and dirty configuration management, but it lacks real depth.

SaltStack is an interesting beast. I would consider it on the lesser side of a steep learning curve. Working with YAML is very easy. Understanding Jinja takes a little bit of work but it’s not the hardest thing in the world and can be applied to anywhere you use python and text rendering. The hardest time I had was really determining the structure of how you want your configuration management and modularity to occur. I want this to be the focus on this article for two reasons, 1) so I can get feedback, 2) so you can understand how pillars, states, and jinja2 all work together.

I’m going to use a particular task: Installing erlang from source (because erlang gets no love). In my example my salt-master is ‘salt-mater’ and my minion is ‘client01p’

So first and for most let me show you my file structure using /srv/ as the root

.
├── pillar
│   ├── core
│   │   ├── open_ports.sls
│   │   └── packages.sls
│   ├── erlang
│   │   ├── common.sls
│   │   └── packages.sls
│   ├── haraka
│   │   ├── open_ports.sls
│   │   └── packages.sls
│   ├── nodejs
│   │   └── common.sls
│   ├── redis
│   │   ├── open_ports.sls
│   │   └── packages.sls
│   └── top.sls
└── salt
    ├── core
    │   ├── map.jinja
    │   ├── open_ports.sls
    │   ├── packages.sls
    │   ├── python34.sls
    │   ├── selinux.sls
    │   ├── ssh.sls
    │   ├── stash_keys.sls
    │   └── sysctl.sls
    ├── erlang
    │   ├── install.sls
    │   └── testinstall.sls
    ├── files
    │   ├── core
    │   │   ├── authorized_keys
    │   │   ├── issue
    │   │   ├── sshd_config
    │   │   └── sysctl.conf
    │   ├── haraka
    │   │   └── haraka.service
    │   └── stash
    │       ├── id_rsa_stash_automated
    │       └── id_rsa_stash_automated.pub
    ├── haraka
    │   ├── install.sls
    │   └── map.jinja
    ├── _modules
    │   └── customuser.py
    ├── nodejs
    │   └── install.sls
    └── top.sls

Let’s explain these folders quickly. We are working out of a /srv/ root directory. This is where all configuration around salt’s pillars and states live. We see there’s two folders under this root, /pillar/ and /salt/. The pillar directory holds pillar information, and the salt directory holds state information. I then further divide my pillars and states to logical functions. In this example we are going to only worry about the the sub-folders “core” and “erlang”. Before we dig into these sub-folders let me quickly explain what pillars and state files are.

Pillars

Pillars hold YAML data structures which you can “tag” onto your minion and are assigned to the minion via the top.sls file. Pillars do not actually do ANY work. Pillars are simple a way for us to pair information, and a minion, together. I like to think of it as defining attributes on an object in a programming language. Typically in object oriented program we define a class, instantiate that object, and then use it’s parameters to perform a task. I.e.

>>> class Adder:
...     def __init__(self):
...         self.number = 4
>>> 
>>> 
>>> Add = Adder()
>>> def addit(Adder):
...     print(Add.number + 4)
...     
>>> addit(Add)
8

In the above example I want you to think of the number 4 as a pillar and the function ‘addit’ as a state. We have an object with a number ‘4’(pillar) which our function(state) ‘looks-up’ and applies a ‘state’ too.

States

I use states as my actual unit of work. My state should be ‘fed’ what ever it needs to complete it’s tasks. You can place logic and variables in your states, however best practices would guide you to place this logic else where and keep your state as simple as possible.

With the above example let’s see how we can install a specific set of packages required to build erlang on a minion, based on a pillar and a state file.

Installing and removing packages with pillars and states

First I want to ‘attach’ some kind of data structure to my minion which will facilitate the install and removal of packages on our minion. We are interested in creating an ‘erlang’ minion who’s only function is to host the erlang language. I then create the following pillar file

#/srv/pillar/erlang/packages.sls

packages:
  install:
    {% if grains['virtual'] != none %}
    - open-vm-tools
    {% endif %}
    - lsof
    - tcpdump
    - mtr
    - traceroute
    - telnet
    - bind-utils
    - curl
    - wget
    - ftp
    - tftp
    - samba
    - samba-client
    - ntp
    - git
    - gcc
    - gcc-c++
    - m4
    - ncurses-devel
    - autoconf
    - java-1.8.0-openjdk-devel
    - openssl-devel
    - make
    {% if grains['os_family'] == 'RedHat' %}
    - yum-utils
    {% endif %}
  remove:
    - postfix 
    - NetworkManager 

This is a list of base packages I want on all my machines along with packages that are required to build erlang.

I then target this pillar to my minion client01p in our pillar top.sls file

#/srv/pillar/top.sls

base:
  'client01p':
    - erlang.packages
    - erlang.common

As you see above, we target our pillar by the specifying the sub-folder the pillar belongs in under the pillar/ root, a ‘.’, and then the .sls file name leaving the .sls out. You notice I targed ‘client01p’ to erlang.common also – more on that in part 2.

So now if we were to do a lookup on the pillar items that ‘client01p’ posses we will see our list of packages. Only pay attention to our ‘Packages’ object.

[root@salt-master srv]# salt 'client01p' pillar.items
client01p:
    ----------
    erlang_version:
        17.5
    open_ports:
        ----------
        22:
            tcp
        161:
            udp
    packages:
        ----------
        install:
            - open-vm-tools
            - lsof
            - tcpdump
            - mtr
            - traceroute
            - telnet
            - bind-utils
            - curl
            - wget
            - ftp
            - tftp
            - samba
            - samba-client
            - ntp
            - git
            - gcc
            - gcc-c++
            - m4
            - ncurses-devel
            - autoconf
            - java-1.8.0-openjdk-devel
            - openssl-devel
            - make
            - yum-utils
        remove:
            - postfix
            - NetworkManage

So this is great, we have a minion object client01p, we ‘tagged’ parameters on this object indicating which packages it should install and remove by use of a pillar.sls an top.sls file. So now how do we use this?

We turn to our core.packages state file now. I placed all my really common tasks in a ‘core’ folder. This is how I decided to do it, and by no means indicates that this is how you should do it. Let’s take a look at the core.packages state file.

#/srv/salt/core/packages.sls

install_packages:
  pkg.installed:
    - pkgs:
    {% for pkg in salt['pillar.get']('packages:install') %}
      - {{ pkg }}
    {% endfor %} 


remove_packages:
  pkg.purged:
    - pkgs:
    {% for pkg in salt['pillar.get']('packages:remove') %}
      - {{ pkg }}
    {% endfor %}

So first thing you’re going to ask, what is that crazy syntax in our state file? Welcome to Jinja2. Jinja2 is a rendering engine, which means it does nothing more then determine what the text in a file looks like after being rendered.

Let’s walk through this state file step by step:

First we have an ID for the state. This can be any arbitrary name as long as it’s unique within the state file. Correction: the ID must be unique across all states that are running not just within the state file.

Next we have the state module we want to use. In our case we want to use the state module named pkg.installed. For a full list of state modules available to you reference here:
https://docs.saltstack.com/en/develop/ref/states/all/index.html

The next line is a state module directive. If you reference the documentation at the link above we see the usage definition for this directive:

pkgs (list) --
A list of packages to install from a software repository. All packages listed under pkgs will be installed via a single command

Now our first piece of Jinja2 code. In Jinja when we want to add any logic, flow control, or variables we begin these statements with {% and end with %}. We are initiating a for loop in this example. We are going to iterate over a function within the salt dictionary. This function takes the arguments ‘packages:install’ which will return each item in our Yaml data structure following install: Let me clarify this,

salt['pillar.get']('packages:remove')
^     ^             ^
|     |             | 
|     |             The arguments to this module 
|     |
|     The execution module you're interested in running
|
Dictionary containing all built in execution modules

What we are doing here is running a remote execution model inside our state file to pull the results into our state and use it there. To get an idea of the output this produces you can do the following

[root@salt-master srv]# salt 'client01p' pillar.get packages:install
client01p:
    - open-vm-tools
    - lsof
    - tcpdump
    - mtr
    - traceroute
    - telnet
    - bind-utils
    - curl
    - wget
    - ftp
    - tftp
    - samba
    - samba-client
    - ntp
    - git
    - gcc
    - gcc-c++
    - m4
    - ncurses-devel
    - autoconf
    - java-1.8.0-openjdk-devel
    - openssl-devel
    - make
    - yum-utils

So we now have this list of packages to feed into our for loop. For ever item in this list we are telling Jinja2 to render the line

– {{ pkg }}

The resulting FILE we arrive at would be this (even though this is invisible to you)

#/srv/salt/core/packages.sls

install_packages:
  pkg.installed:
    - pkgs:
      - open-vm-tools
      - lsof
      - tcpdump
      - mtr
      - traceroute
      - telnet
      - bind-utils
      - curl
      - wget
      - ftp
      - tftp
      - samba
      - samba-client
      - ntp
      - git
      - gcc
      - gcc-c++
      - m4
      - ncurses-devel
      - autoconf
      - java-1.8.0-openjdk-devel
      - openssl-devel
      - make
      - yum-utils


remove_packages:
  pkg.purged:
    - pkgs:
    {% for pkg in salt['pillar.get']('packages:remove') %}
      - {{ pkg }}
    {% endfor %}

So to stress this point, all Jinja2 is doing here is rendering a new TEXT file which salt will then use to run our state. These are two distinct and mutually exclusive processes. First Jinja2 is called to render our state file, then salt evaluates the syntax and performs our state. The same exact thing happens for remove_packages and all the above can just be implied.

Ok so if this all makes sense, all we do now is target this state to our minion in the STATE top.sls file

base:
  'client01p':
    - core.packages

We can now run highstate or do a one off execution of our state on the minion

salt 'client01p' state.highstate

or

salt 'client01p' state.sls core.packages

Let’s summarize:

We created a pillar file with a data structure which allowed us to easily query which packages we want to install and remove for a particular minion

We targeted this pillar to the minion via the /srv/pillar/top.sls file

We then created a state file in /srv/salt/core/packages.sls. This state file ‘looksup’ the installed and removed packages from the minion’s pillar and runs a for loop rendering these packages in the correct YAML syntax. After this is rendered salt runs the state and performs the code which actually installs the packages.

Now as you can tell from my overall folder structure, we are creating a solid framework for modularity. I can easily create a new sub-folder for a machine type, i.e. mysql. We can then copy the /srv/pillar/core/packages.sls pillar file into /srv/pillar/mysql/ pillar directory and edit the packages.sls file, adding the new packages we want. Then in our /srv/pillar/top.sls pillar file we can even target ‘*mysql* to apply this pillar to all minions with mysql in the name.

One thing I’d love to see is simple list merging. As things are today, each salt/pillar/*/package.sls pillar file MUST contain ALL packages you want to install on the images. If we tried to do something like the following

#/srv/salt/top.sls

base:
  '*':
    - core.packages
  'client01p':
    - erlang.packages

One would assume that the wild card will install our base packages, and we would only need to place the erlang specific packages in our /srv/pillar/erlang/packages.sls file. This is not the case. they will overwrite each other, leaving the minion only with the packages listed in /srv/pillar/erlang/packages.sls

I hope this helps clear some stuff up. In part two I’ll be moving forward and showing you guys how to use pillars, states, and map.jinja files to download erlang from source and compile.

Stay tuned.

Python Enhanced Generators – A deviation from traditional functions.

Here’s a concept that I’ve struggled with a little bit in Python – Enhanced Generators. They always seemed intriguing but there were several road blocks which stopped me from truly grokking them. One of those road blocks was the fact that we known functions so well, we understand their normal workflow and can toss them around like it’s second nature. To truly understand enhanced generators we need to stop, step back, and change our thought flow when it come to functions.

Enhanced Generators give us the ability to send information into a function, and use this information to manipulate the output given by the function. Keep in mind, when I’m using the term function here – I actually mean generator – and I will explain the difference below.

In order for you to understand enhanced generators we’ll need a quick explanation of generators in general, and to do THAT we need to explain what iterators are very quickly.

Iterators

Iterators are used in python as a general term for any object which implements an __iter__ method. I’m not going to jump into the nitty gritty of implementing the __iter__ method into a class – but for good measure know that this needs to be present in order for an object to be considered an Iterator.

If you’ve used python for some time you’re already familiar with what iterators are. Iterators are any objects that you can run a for loop on. Lists, arrays, strings, etc… And when in doubt, run the built in dir() function on the object in question – if you find __iter__ is listed as one of the members on that object it is an iterator.

So that’s a pretty simple concept. Now,

Generators

Generators are a lot like functions except for these key points:

1. They use a yield statement instead of return
2. They do not directly return results of your code, they return an object which in turn will return the output of your code.

So before anything let’s give you a quick example of a generator

def number_generator(max):
    i = 0
    while i < max:
        yield i
        i += 1

Okay so let’s explain.

I always like to start my explanations with WHAT we are trying to accomplish. Right now I’m making a simple number generator. The first call to the generator should produce the initial value of variable i. All subsequent calls to the generator should produce the last value of variable i plus 1. So let’s explain the code.

We define a generator the same way we define a function using the well know def statement, a name, and the value which we will inject into the generator. We then set variable i to 0 as our initial value. We then enter a while loop saying “while i is less then our max value, 1) yield i 2) add one to the current value of i. So now the big question what is that weird yield statement doing?

I want you to read our generator step by step, and when you hit the yield comment stop, and think about what value is in variable i. Imagine this value being the return value of your function. What happens when you hit a return statement? You get back what ever value is after your return. This is exactly what happens – BUT what also happens is your generator FREEZES keeping all your local variables defined. So now imagine – you ran this code, we get to yield, we see the value of i return (to stdout if that helps you visualize) and then we tuck this generator away – for future use.

So the time comes now and you want to use our generator again. Right now before reading forward – take a guess at what will happen when we call our generator again?

If you guessed, it will pick up from where the yield statement left off – you are correct. Let me clarify

So in our little mental run-through above we left off with thinking of the yield as the return statement, and variable i being returned from our generator. Now we want to call our generator again. Instead of tracing through our generator from the first statement, “i = 0”, begin your mental execution at the “yield i” statement. Keep in mind that we are starting at yield and this generator REMEMBERS where the “state” was after your first call. In other-less-technical-words your generator REMEMBERS that “i = 0”. Now if we pick up at the yield statement, the next executing piece of code is “i += 1” setting our variable “i” to “1”. We then continue execution by checking our wild-loop condition. If “i” is still < “max” then we hit the next “yield” statement which will return i. What value is currently in “i”? It is 1! This process continues on and on until you break the while loop condition. When the wild loop condition is met, the loop is broken, we do not hit a yield, and we get a very specific error named “StopIteration.” This error is present when we have a generator and the code within the generator no longer “hits” a yield statement.

So how do we use this generator? Like I stated above, generators do not directly return results of your code, they return an object which in turn will return the output of your code.

Here’s an example:

 
numgen = number_generator(10)

Running number_generator(10) returns a generator object, we place this object in the variable numgen

Here’s how we call the generator:

>>> next(gen)
0
>>> next(gen)
1
>>> next(gen)
2
>>> next(gen)
3
>>> next(gen)
4
>>> next(gen)
5
>>> next(gen)
6
>>> next(gen)
7
>>> next(gen)
8
>>> next(gen)
9
>>> next(gen)
Traceback (most recent call last):
  File "", line 1, in 
StopIteration

We use the next() function to “call” the generator. Keep in mind every time we call the next() function on the numgen object, we are running the code starting at the yield statement. We also see here the “StopIteration” error when we hit the maximum value of 10 injected into the generator.

Makes sense so far?

Enhanced Generators

So with that under our belt I want to present the main focus of this post – Enhanced generators. This features (if I’m not mistaken) was introduced in Python2.5. What an enhanced generator allows us to do is inject a value into our generator and effect the output of our yield statement.

I’m going to introduce a different task in order to show how enhanced generators work.

I want to create a generator which will predictably output a string. This generator should output the string in which we call the generator with. I also want to provide some added functionality by being able to CHANGE the string that outputs without having to alter my generator. We are building a dynamic generator which we can alter the behavior of by “sending” it data.

def printer(string):
    s = string
    while s is not 'quit':
        injected_string = (yield s)
        if injected_string is not None:
            s = injected_string

Okay so this is going to be a little bit of a brain twister. Let’s take this step by step.

We define a generator in the same way we define a function, and the same way we defined a our generator above. We then set variable “s” to the string passed into our generator.

Now we enter a while loop. What we are looking for to exit our loop is the variable “s” to be equal to the string ‘quit’. While “s” does not equal “quit” we will enter our loop.

Next we have the first piece of code that looks unfamiliar. What we are doing here is providing a variable for us to “inject” data into. If we decide to “inject” data into our generator this data will be held in the “injected_string” variable. If we DO NOT inject data into our generator then (yield s) is evaluated to None and placed into injected_string.

Let’s further define for clarity.

So, in the above function, if we inject a string, our code is going to “yield” the injected string AND place that injected string into the variable “injected_string”. If we DO NOT inject a string, our code is going to yield the current variable “s” and then evaluate to None. This None state is going to be assigned to “injected_string”. This is a little hard to understand since the “(yield s)” statement is doing two things here. If you’re a little confused about this here’s a little experiment. Real quickly try to assign the output of the built in print function to a variable

>>> x = print('hello')
hello
>>> x
>>> 
>>> print(x)
None

What the print function actually does is run code which knows how to send some data to stdout, BUT it RETURNS none. This is the same pattern our “(yield s)” code is using. It will RETURN None, but still YIELD what ever data is in “s”.

So now we should understand that if we DO inject data into our generator, it’s stored in the variable “injected_string”, if we don’t we just yield the ORIGINAL string that is passed into the generator and assign “injected_string” the built in None.

So now let’s examine this piece

def printer(string):
    s = string
    while s is not 'quit':
        injected_string = (yield s)
        if injected_string is not None:
            s = injected_string

So following our thought process let’s look at this logically. We have variable “s” equal to the string passed into the generator. We then enter a while loop which checks to make sure “s” does not equal ‘quit.’ We then give the opportunity to evaluate a passed in piece of data and store that data in “injected_string”. Now we perform a check saying “If the injected_string variable is not None, which it would be if we DIDN’T inject a value into the generator, then set s to “injected_string”.

Let’s stop here and go over the flow. Remember that generators always start and stop (other then the very first call to the generator) at the yield statement in the function. Therefore, if we decided to inject a string into our generator and then call it, the following will happen:

1. the generator with “yield” or RETURN the injected string
2. the generator then stores the injected string in the variable “injected_string”
3. we do a check to see if the variable “injected_string” is not None
4. If the variable “injected_string” is not None, we then assign our “s” variable equal to the “injected_string”

Now since we ultimately changed variable “s” to be our new string, all SUBSEQUENT calls to our generator using the next() function will yield “s” – our new string!

So I’ve pretty much showed you everything except USING the code we are talking about. I will show an example below, we use the built in next() function to iterate through our generator and the generator member function “send” to send data into our generator.

>>> printer1 = printer('hello')
>>> next(printer1)
'hello'
>>> next(printer1)
'hello'
>>> next(printer1)
'hello'
>>> next(printer1)
'hello'
>>> printer1.send('hi!')
'hi!'
>>> next(printer1)
'hi!'
>>> next(printer1)
'hi!'
>>> next(printer1)
'hi!'
>>> next(printer1)
'hi!'

So our code works as expected. On the first call the generator is created with an initial string, this initial string is placed into variable “s”. Variable “s” does not equal ‘quit’ so we enter our while loop. We are presented our first yield statement – which will yield or RETURN “s” and assign injected_string the built in None since we did not inject data into our generator – we just called the next() function on it. Since injected_string is now equal to None, we skip the if statement.

Now to send data into our generator we use the member function “printer1.send(‘hi!’)”. When we use this method we pick up at the yield statement in our code. By using the “send” method we can imagine replacing “(yield s”) with our injected string. Like so:

def printer(string):
    s = string
    while s is not 'quit':
        injected_string = 'hi!'
        if injected_string is not None:
            s = injected_string

Now that we have a value in injected_string we enter our if statement and assign “s” to our injected string. We then evaluate our while loop condition, s is not ‘quit’. Looking back at our original code – we then yield or RETURN “s” – which is our new injected variable – and assign None to “injected_string”

We have ultimately achieved our goal. We have a generator which will generate the original string we called the generator with. Then if we “send” some data into the generator it will yield that data on the send call along with all subsequent calls.

I hope this helps clear up enhanced generators and the power they they provide you. Truly understanding how to use them takes a little deviation from traditional functional thinking.

HA Proxy fail over cluster with heartbeat – A quick WIN.

Starting my career as a network engineer I’m immediately drawn to the very “infrastructurey” components in my current position. I flock to software such as HaProxy, Varnish, Nginx, VyOS, and RabbitMQ to name a few. The design aspects of high availability carry over from hardware to software – and I love to design with these solutions.

I was recently working with a client who’s network path toward their API was needlessly long and took a few too many hops. Instead of re-configuring their current environment I purposed we setup HaProxy on the same LAN at which the load balancing will happen. Once time freed up and they could afford to fix the bad path, they would do so.

As everyone knows by know HaProxy is a beast. It’s written in C and can handle a LOT of traffic – however the former network engineer in me would not be able to sleep unless I had at least one method of HA. Luckily keepalived has been tweaked to work with HaProxy offering a mechanism to check two HaProxy instances. Keepalived uses VRRP which is something familiar to me. VRRP is a protocol which allows two [originally]routers to have one IP. With VRRP both nodes have a “real” ip and mac address. While both nodes are up the primary will own a VIP (virtual ip address) and will respond to ARP’s for that IP address. If the secondary node notices that the primary is gone (via a multicast keep alive control packet, hence the name) the secondary node will send a gratuitous ARP which says “Hey! I own this MAC and IP Address now!”

Okay enough with the background let’s get into the configs:

I have 4 servers in play here – two HaProxy boxes and two web servers:

192.168.1.1 – haproxy01p

192.168.1.2 – haproxy02p

192.168.1.3 – api01p

192.168.1.3 – api02p

I will also be using a VIP:
192.168.1.4 (this is the virtual ip address that will belong to both HaProxy boxes)

First things first – Get HaProxy and KeepaliveD installed. I’m on CentOs and both packages are in the default repository. Feel free to install these on both HaProxy boxes.

yum install haproxy keepalived -y

Let’s configure haproxy01p and 02p – they are going to have the same exact configs

#/etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    
    

defaults 
    log global
    mode http
    option httplog
    option dontlognull
    option forwardfor
    option http-server-close
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend platforminternal_frontend
    bind 192.168.1.4:80 
    option httplog
    reqadd X-Forwarded-Proto:\ https
    default_backend platforminternal_backend

backend platforminternal_backend
    balance roundrobin
    server api01p 192.168.1.2:80 check
    server api02p 192.168.1.3:80 check
      

listen stats :1936
    mode http
    stats enable
    stats uri /
    stats hide-version
    stats auth admin:admin

Note that the frontend is an IP address that isn’t currently on the box. If you were to start up haproxy right now it would fail since the VIP is not configured on this box. We don’t need to manually add another IP to the server – keepalived will handle this.

Now let’s edit keepalived configuration file – this we will actually have two different configurations, changing the priority in haproxy02p in order to indicate it’s the secondary.

# haproxy01p /etc/keepalived/keepalived.conf
bal_defs {
   notification_email {
     alteremail@alertemail.com
   }
}

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    interface ens160 
    state MASTER
    virtual_router_id 10
    priority 101
    virtual_ipaddress {
        192.168.1.4
    }
    track_script {
        chk_haproxy
    }
}

And for haproxy02p

bal_defs {
   notification_email {
     alertemail@alertemail.com
   }
}

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    interface ens160 
    state MASTER
    virtual_router_id 10
    priority 100
    virtual_ipaddress {
        192.168.1.4
    }
    track_script {
        chk_haproxy
    }
}

It goes to note – a higher priority wins – so 101 beats 100 for priority and 01p becomes the master.

With this all setup go ahead and start keepalived and then HaProxy. You should notice an extra IP address on the master node.

Test out your load balancing and your stat’s page. If everything looks good try to do a fail over. I typically do a simple test by starting a continuous ping to the VIP and hard resetting the master node.

So that’s really all there is too it – nice, quick, and easy win for high availability.

UPDATE:

Thanks to MMDeveloper on reddit I was given a pretty awesome tip here it is verbatim

When I’m setting up haproxy pools I sometimes like to monitor more than one service on each of the HAProxy nodes, I usually script keepalived to use a check script instead of a one-liner.. for example:

vrrp_script chk_loadbalancedservices {
    script "/etc/keepalived/check.sh"
    interval 2
    weight 2
}

contents of check.sh

#!/bin/bash

### This will check all processes you wish to monitor
### In the event one of them is 'down', it will trigger
### a failover to the next-in-line HAProxy server

killall -0 haproxy 2> /dev/null
if [ $? -eq 1 ]
then
    exit 1
fi



### Copy/Paste the above block of code for each
### process you want to monitor
### This script should always exit with a code of 0 or 1
### An exit code of 0 means all is good
### An exit code of 1 will trigger a failover


exit 0

What MMDeveloper is doing here is giving us the ability to check multiple services on the haproxy box to give a more granular approach to fail over. The script is generic and you can place which ever process name you’d like. If you have a critical service that runs next to haproxy on the same box, then add another stanza for this service:

killall -0 otherimportantservice 2> /dev/null
if [ $? -eq 1 ]
then
    exit 1
fi

Now if other important service fails – fail over will occur.

Why learning to code marks the true maturity of your career.

I… HATE computer programming, but… we are trying to work things out.

It’s a rocky relationship, reminiscent of that “one college friend.” You know the guy, always around to help you out, give you a hand, help you lift heavy kegs. You love this guy! Then you walk into your apartment and he’s making out with your girlfriend. What the hell?!

Yeah, that’s how programming and I get along. You’re always there for me, can probably solve any issues I encounter, but in the back of my mind there’s always the looming premonition that this code is going to screw me over.

Enough with the cheesy metaphors, let me start by giving you a quick background of my career in IT.

I started in computer networking at a young age. If I remember correctly my first networking job was at 16 setting up home routers, modems, port forwarding, firewalls etc… I stayed in the networking field up until around 20. I then got a job as a systems admin which introduced me to systems engineering. With both those skills sets I them moved up to mixed-rolled positions. Positions where I was responsible for the “full stack” as people like to call it (Networking, Storage, Server, Virtualization, etc..). I enjoyed this space a lot as it taught me how to run an infrastructure from ground up, green field, day 0 implementations.

Then something happened.

I believe it was around 21 where I was faced with issues that I could not solve. These were various things – “watch a directory till a file is created, then send an email”, “watch a network interface until no packets are received, then change a route.” You get the idea.

Now, I feel there are two people in IT. There’s the individuals who will except this as an innate limitation. Individuals who will say “I’m a network engineer, not my job” and “I’m a systems engineer, sounds like a developer’s thing to me.” Then there’s the individuals, like myself, who literally can’t accept that something can’t be done. Individuals who don’t want to rely on others to accomplish tasks necessary within their own domain, be it systems engineers, networking engineers, hell – even storage admins.

It’s at this point where I decided, it’s time – I needed to learn how to code. So with only the first three classes of computer science in high school as my experience (I dropped out) I began to take the steps toward expanding my skill set. I began with powershell and c# being in a windows environment. After about a year I could write intricate services for windows servers and c# applications for querying AD.

Listen, you want your boss to love you and that raise you’re looking for? Tell them you implemented a file watching service which will delete files automatically before the drive fills up and alerts you at 2am in the morning. Even better – Do it for yourself!

So as a natural evolution of my career, I added programming to my skill set. This didn’t happen over night. I’m going to be 27 next month and I’m only starting to get into full application development.

But now lets move our attention to the title of this post. “Why learning to code marks the true maturity of your career.” Let me answer this the best I can.

Learning to code is analogous to you taking your craft, your trade, what you love to do, and moving from an observer to a creator. It’s one thing to learn the ins-and-outs of OS, networking system, or even a large distributed system. It’s a whole other thing to be able to create within this space. To come up with novel solutions to simple and complex problems alike. In my opinion, you haven’t reached the true maturity of your career until you can sit down and engineer custom solutions to environment specific issues that arise.

As of today I work with C#, Powershell, and Python as my primary go-to languages. I find so much confidence in the fact that if a requirement is dropped on my desk I’m 95% certain that I won’t be saying “This can’t be done.” Ask me this a few years back and I can’t promise you you’d get the same answer.

I wanted to write this just to give motivation to any individuals who are in jr. roles right now. There’s no denying the fact that companies are moving more and more toward infrastructure as code and automation. Getting a grasp on the basics of scripting and programming now would be a great benefit to anyone’s career. Take it from me, someone who can barely do math, you can learn to program.

Python Decorators – A practical example

Hey guys,

I work in DevOps and as you can tell from my blog post I’ve been working with monitoring and system metrics lately. I came across a great python package called psutils. You can find it here: http://pythonhosted.org/psutil/

The psutil tools tend to give disk and memory metrics back in bytes. I don’t know about you but I don’t enjoy doing the conversion from bytes to human readable forms in my head. Their documentation points us to using the following function for conversion:

def bytes2human(n):
    # http://code.activestate.com/recipes/578019
    # >>> bytes2human(10000)
    # '9.8K'
    # >>> bytes2human(100001221)
    # '95.4M'
    symbols = ('K', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y')
    prefix = {}
    for i, s in enumerate(symbols):
        prefix[s] = 1 << (i + 1) * 10 for s in reversed(symbols): 
            if n >= prefix[s]:
                value = float(n) / prefix[s]
                return '%.1f%s' % (value, s)
    return "%sB" % n

I like this function a lot – quick and dirty, however I didn’t want to have to pipe the values that psutil outputs to this function. I instead would like a function that I can call and obtain human readable output without jumping into psutil’s source code.

The perfect tool in the python language for this is to create a decorator (without access to the source code)

I’ll be working with the function:
psutil.virtual_memory()

This function returns a named tuple with values, here’s an example:

>>> psutil.virtual_memory()
svmem(total=8048656384, available=2548318208, percent=68.3, used=7201034240, free=847622144, active=4398604288, inactive=2348535808, buffers=84848640, cached=1615847424)

Now what we want to do is extract the keys and the values from this output, convert the values to human readable, zip the keys and values back up into a dictionary and return said dictionary. For this we are going to create a decorator.

What is a decorator ?

1. A decorator takes in a function.
2. It then defines a function which manipulates the output of the original passed in function and returns the manipulated object.
3. After the above function is defined, the enclosing function (the decorator itself) returns the defined function object.

That last part is key here – a decorator TAKES IN a function and RETURNS a function – a decorator does not directly give you the manipulated results.

So let’s see some code. In this example I’ve already declared “bytes2human” function.

def convert_humanreadable(virtualmemory): 
    def wrapper(): 
        tuple = virtualmemory()
        keys = tuple._fields
        values = [ bytes2human(value) for value in tuple ]
        return dict(zip(keys, values)) s
    return wrapper

So let’s compare the above example with my definition of a decorator.

The first thing we do here is declare our decorator function. It’s going to take in a “virtualmemory” object which will be the psutil.virtual_memory function.

Next we immediately define a function which is going to manipulate the output of the passed in function (psutil.virtual_memory). We take the tuple that we get when we run psutil.virtual_memory() and place it into a variable “tuple”. We then extract the keys from the tuple. We then use a list comprehension to take each value in tuple and convert it to a human readable form. We then give the instruction to return a dictionary of the keys and the new human readable values.

We wrap up our decorator by returning the function we defined above. Remember, the decorator itself returns a function, you use this new function to obtain the post-manipulated human-readable values.

So how do we use it?

virtual_memory_hr = convert_humanreadable(psutil.virtual_memory)
virtual_memory_hr() 
{'available': '2.6G', 'buffers': '82.8M', 'total': '7.5G', 'cached': '1.5G', 'inactive': '2.1G', 'active': '4.0G', 'free': '1.0G', 'percent': '65.1B', 'used': '6.5G'}

There you have it, now I have a function I can run to output everything in a human readable format.

Now if I felt so inclined, I could download the source code for psutil, define my decorator function along with the bytes2human function in the source and decorate each function. I’ll give a small example below (source at: https://github.com/giampaolo/psutil/blob/master/psutil/_pslinux.py#L169)

@convert_humanreadable
def virtual_memory():
    total, free, buffers, shared, _, _ = cext.linux_sysinfo()
    cached = active = inactive = None
    with open('/proc/meminfo', 'rb') as f:
        for line in f:
            if line.startswith(b"Cached:"):
                cached = int(line.split()[1]) * 1024
            elif line.startswith(b"Active:"):
                active = int(line.split()[1]) * 1024
            elif line.startswith(b"Inactive:"):
                inactive = int(line.split()[1]) * 1024
            if (cached is not None and
                    active is not None and
                    inactive is not None):
                break
        else:
            # we might get here when dealing with exotic Linux flavors, see:
            # https://github.com/giampaolo/psutil/issues/313
            msg = "'cached', 'active' and 'inactive' memory stats couldn't " \
                  "be determined and were set to 0"
            warnings.warn(msg, RuntimeWarning)
            cached = active = inactive = 0
    avail = free + buffers + cached
    used = total - free
    percent = usage_percent((total - avail), total, _round=1)
    return svmem(total, avail, percent, used, free,
                 active, inactive, buffers, cached)

The @convert_humanreadable is basically syntaxual sugar for:

virtual_memory_hr = convert_humanreadable(psutil.virtual_memory)
virtual_memory_hr()

The difference is the @decorator syntax re-maps the output of our decorator to the original (decorated) function name. Which would look like this:

virtual_memory = convert_humanreadable(virtual_memory)
virtual_memory()

Python takes the original function, places that in the decorator, and then maps the original function name to the function the decorator returns. This… is a little bit of a brain twister, but ultimately allows us to continue to call psutil.virtual_memory() in our programs and not the decorator itself.

Hope this gives a good example of when to use decorators and how they can elegantly solve specific problems.

Icinga 2 – Lessening the steep learning curve

Icinga 2 has sparked a lot of interest in the DevOps and opensource communities. It’s a great monitoring solution which provides a fabulous web GUI. This is one of the most complete packages for monitoring I’ve seen in some time , however the learning curve is steep. The documentation on the site is thorough, but in my opinion lacks at creating a cohesive “how-to” for professionals of all experience levels. My goal for this post is to lessen the learning curve and give a  user friendly introduction to the basics of Icinga 2.

This post will focus on the the features and configuration of Icinga 2 monitoring. I will not be going over how to install all the components which make up Icinga2. Instead my lab will consist of:

• icinga/icinga2 docker image
  • https://hub.docker.com/r/icinga/icinga2/
• KVM Hosted CentOS7 VM

If you aren’t familiar with docker, I highly suggest you become. It’s a great way to get complex setups up and running within seconds. I will be doing my own post about docker basics following this one but here are some basics to get you started.

Getting the docker image up and running:

Icinga2 did a great job in this image and made it pretty damn easy to get up and running. You can run the following code:

docker run -it --name icinga01 -p 3080:80 icinga/icinga2 

This will launch a init script whichyou can find here on git (if interested): https://github.com/Icinga/docker-icinga2/blob/master/content/opt/icinga2/initdocker

Once the init script is done Icinga2 will be running within the terminal you ran the above command. Ctrl-C will stop the image.

Stopping/Removing Icinga2 image:

If you loose tty control of the icinga2 image and want to stop it, or you want to start another icinga2 image with the same name of icinga01 you must use the docker client to stop and remove the image. Stopping simply ceases the image running. Removing removes the image from docker’s active inventory, allowing you to create another image with the name of icinga01

docker stop icinga01
docker rm icinga01

Here’s a little tip  when working with a docker image. I use Bash’s built in variables to aid me in typing less. It’s very often that when getting things going you want to start, stop, and remove an image until you get something right. I do the following

start='docker run -it --name icinga01 icinga/icinga2'
stop='docker stop icinga01
remove='docker rm icinga01'

Now when I can simply just use my variables on the commandline to start, stop, and remove the image I’m working with.

Getting into your icinga2 image

Docker is meant to encapsulate your applications. Best practices say that you should only run one process in a docker container. However I choose functionality over best practices in this example. We will launch an interactive-tty bash session. This will give us a shell inside our Icinga2 instance and allow us to write configuration files, restart the Icinga2 process, and many other tasks you’re accustomed. In a new terminal type the following docker command

docker exec -it icinga01 '/bin/bash'

What we are doing here is launching a second process(/bin/bash) in our icinga01 container.

A different and more “Docker” approved way is to look at the docker file, find the volumes that are exposed and then use the docker inspect command to find where on the host file system the exposed container volumes are located. I use this command:

docker inspect icinga01 | grep -A 10 'Volumes'

What you’ll see is something like the following:

"Volumes": {
 "/etc/icinga2": "/var/lib/docker/vfs/dir/f7d3baec7251b682411985719fb495e98750440b8452c6d1af83770bfb223aa3",
 "/etc/icingaweb2": "/var/lib/docker/vfs/dir/f4175c7a5fec51b1618eac9a8b20f7096dae5ae16cf5e62597534290612f38ca",
 "/usr/share/icingaweb2": "/var/lib/docker/vfs/dir/2ecfca1002fcde46343cf82b16933c5016082f055057802d5abbbd740d73d96d",
 "/var/lib/icinga2": "/var/lib/docker/vfs/dir/9db43889fc7f38cb619ce3f187ca705d799a77985e262fbc1a2bbbc9bc249867"
 },

What this is indicating is that the container’s directory ‘/etc/icinga2’ is located or “mounted” on my host filesystem at “/var/lib/docker/vfs/dir/f7d3baec7251b682411985719fb495e98750440b8452c6d1af83770bfb223aa3” 

Any more detail then this is outside the scope of this guide.

For our test purposes I will run a bash shell inside the container for all tasks.

Connecting to Icinga2

If you followed the instructions provided by the docker hub page you should be able to connect to icingaweb2 at: http://localhost:3080/icingaweb2

Take a look around the interface. You’ll see a default host object called docker-icinga2. They did a great job GUI and I’m pretty impressed by the multi-tab nature of it.

Conf files

At this time I’d like you to open a new terminal and run:

docker exec -it icinga01 /bin/bash

Now you are inside your icinga01 docker image.

The root of icinga2’s configuration is at “/etc/icinga2/”. Files that I won’t dig to much into but are import are:

icinga2.conf – main configuration file for icinga2 server.

constants.conf – a location for you to put variable constants. These are variables that can be accessed by any scripts we write within Icinga2 configuration directories.

We will be spending most of our time is the conf.d folder.

The conf.d folder and scripts

The scripts in this folder give an example of configuration. Right now the hosts.conf script is providing you with the docker-icinga2 Host Object you see in icingaweb2. The services that are being checked for the docker-icinga2 are defined in the services.conf file. Icinga2 will only check files within the conf.d folder if they end with “.conf”. Keep this in mind when naming any files you wish the Icinga2 daemon to parse.

I’m actually going to blow these two files away for the purposes of this guide and to provide you with a fresh example of setting up a host and checking services. I’d like you to keep the following two links open – so we can reference the examples given to us within those files (also, give it a solid read)

https://github.com/Icinga/icinga2/blob/master/doc/4-configuring-icinga-2.md#hosts-conf

https://github.com/Icinga/icinga2/blob/master/doc/4-configuring-icinga-2.md#services-conf

Let’s remove those files now and update icinga2

rm /etc/icinga2/conf.d/hosts.conf
rm /etc/icinga2/conf.d/srevices.conf
pgrep icinga2 | xargs kill -SIGHUP

Refresh your icingaweb2 instance and we should now have a completely blank slate to work with.

Folder Structure and your first managed Host

Icinga2 allows you to have any folder structure you like within the conf.d. I take liberty of this and create a hosts directory and a services directory.

mkdir /etc/icinga2/conf.d/hosts/
mkdir /etc/icinga2/conf.d/services/

Now I must make you aware that there’s two mentalities for defining your hosts and services (that I’ve found thus far).

The Flat Design – You have one config file which has both your host objects and your service objects defined in one file.

Apply Rules Design – You have conf file for host object declaration. In this file you place custom attributes which are just “tags” on your host. You then create service conf files which will apply themselves if that “tag” is present on the host.

Note* Icinga2 is, if nothing else, extremely flexible. You can most likely make your own design rules, please share in comments if you have!

Flat Design Configuration

Let’s start with a very simple Flat config file (taken from icinga2’s documentation):

I have a host named icinganode01 with IP: 192.168.122.90/24. I’m going to create a host object for this node and then create a “ping” check service. Fill in the “address” with your VM you want to monitor.

#/etc/icinga2/conf.d/hosts/icinganode01.conf
    object Host "icinganode01" {
       address = "192.168.122.90"
       check_command = "hostalive"
   }
   object Service "ping4" {
       host_name = "icinganode01"
       check_command = "ping4"
   }

Let’s quickly update Icinga2 service with:

pgrep icinga2 | xargs kill -SIGHUP

Refresh your Icinga2web page and you should see you have a new node and 1 service being checked. Poke around in the interface and play around with it for a little.

So let’s explain this flat file above.

Inside the icinganode01.conf file we declared a host object. Each object will have certain attributes which are required. In our case the Host object requires that we specify a check_command attribute. How did I determine this? Via icinga2’s documentation: http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/object-types#objecttype-host

Now on the second block of code we declare a service. Let’s look at the required attributes for this Service object: http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/object-types#objecttype-service

Our two required arguments are host_name and check_command which we supplied.

Keep in mind – Icinga2 doesn’t care about our folder structure or where we declare objects. I could have declared 4 more host objects in my icinga01.conf file and they will be added to icinga2, but that would be pretty ugly and non-intuitive. In order to keep things neat and also to work nicely with a config management software (salt, puppet, chef etc..) I prefer to keep each node in their own file.

Apply Rules configuration

There’s another method for declaring hosts and services. This involves us declaring a host object in one file and an “apply” Service object in another file. The idea is that when we declare our host we will add an attribute to the host object. We then create a service apply file which “looks” for this attribute on a host. If the attribute is present on the host the service will be applied. Let’s show an example.

First off let’s delete the previous icinganode01.conf in the hosts folder we created above.

Now let’s create these files:

#/etc/icinga2/conf.d/hosts/icinganode01.conf

object Host "icinganode01"{
        address = "192.168.122.90"
        check_command = "hostalive"
        vars.os = "Linux"
     }
#/etc/icinga2/conf.d/services/apply_ping4.conf

    apply Service "ping4" {
        import "generic-service"
        check_command = "ping4"
        assign where host.vars.os == "Linux"
    }

pgrep icinga2 | xargs kill -SIGHUP

What you should see now in icinagweb2 is your node and the ping4 service. We have effectively decoupled the service assignment process from the Host object declaration.

Just to clarify, lets look at the above example.

We declared a host object called icinganode01, just like before. What’s different is that on the host object we declared an attribute called “vars.os”. We assigned the string “Linux” to the vars.os attribute.

We then declare our service. We use a different keyword here, the “apply” keyword. This tells Icinga2 that this is going to be used as an apply rule. As we know if we used the “object Service” declaration we’d have to specify a “host_name” attribute and that would break our decoupling. (I’m assuming this is why icinga2 uses a separate keyword here)

The import statement loads in attributes which are located in the built-in template that icinga2 provides. These imported parameters deal with threshold, timeout,etc.. for more details check the documentation.

check_command is defining which check we would like to run (more on this in part2, the suspense is killing you isn’t it?)

Then we reach the “assign where” directive. This is where we link the service to the host object based on the host object’s attributes. We are saying “Assign this service where you see a host that as vars.os attribute with the value being Linux” As you can see we are giving a fully qualified attribute name here “host.vars.os”. Since we declared “vars.os = “Linux” inside a HOST object, we need to locate that attribute via “host.vars.os” when we are searching outside of the HOST object.

Expanding on apply rules

If you expand upon this idea we can immediately see how quickly setting up a basic set of monitors can be for each new machine that enters your network. We can have a “apply_default_services.conf” service file. Inside this file we can have multiple apply Service directives looking for host.vars.os == “Linux”. Every time you define a new host object and define the attribute “vars.os = ‘Linux'” that set of “default” services will be applied. Furthermore, you can still add specific “object Service” commands in the host.conf file for specific services outside of the default.

In the following posts I plan on covering Remote Monitoring, CheckCommand and Custom Checks, and digging deeper into the domain specific language Icinga2 offers (attributes, macros, data structures, etc..)

I myself have a few questions which I’d like to open up to the community. I’m not seeing the full use of Templates within the DSL – anyone can to share some real world examples? One other thing that’s a little puzzling for me is why does the Host object require a check_command attribute? I feel this is a bit confusing when using apply rules.