HA Proxy fail over cluster with heartbeat – A quick WIN.

Starting my career as a network engineer I’m immediately drawn to the very “infrastructurey” components in my current position. I flock to software such as HaProxy, Varnish, Nginx, VyOS, and RabbitMQ to name a few. The design aspects of high availability carry over from hardware to software – and I love to design with these solutions.

I was recently working with a client who’s network path toward their API was needlessly long and took a few too many hops. Instead of re-configuring their current environment I purposed we setup HaProxy on the same LAN at which the load balancing will happen. Once time freed up and they could afford to fix the bad path, they would do so.

As everyone knows by know HaProxy is a beast. It’s written in C and can handle a LOT of traffic – however the former network engineer in me would not be able to sleep unless I had at least one method of HA. Luckily keepalived has been tweaked to work with HaProxy offering a mechanism to check two HaProxy instances. Keepalived uses VRRP which is something familiar to me. VRRP is a protocol which allows two [originally]routers to have one IP. With VRRP both nodes have a “real” ip and mac address. While both nodes are up the primary will own a VIP (virtual ip address) and will respond to ARP’s for that IP address. If the secondary node notices that the primary is gone (via a multicast keep alive control packet, hence the name) the secondary node will send a gratuitous ARP which says “Hey! I own this MAC and IP Address now!”

Okay enough with the background let’s get into the configs:

I have 4 servers in play here – two HaProxy boxes and two web servers:

192.168.1.1 – haproxy01p

192.168.1.2 – haproxy02p

192.168.1.3 – api01p

192.168.1.3 – api02p

I will also be using a VIP:
192.168.1.4 (this is the virtual ip address that will belong to both HaProxy boxes)

First things first – Get HaProxy and KeepaliveD installed. I’m on CentOs and both packages are in the default repository. Feel free to install these on both HaProxy boxes.

yum install haproxy keepalived -y

Let’s configure haproxy01p and 02p – they are going to have the same exact configs

#/etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    
    

defaults 
    log global
    mode http
    option httplog
    option dontlognull
    option forwardfor
    option http-server-close
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend platforminternal_frontend
    bind 192.168.1.4:80 
    option httplog
    reqadd X-Forwarded-Proto:\ https
    default_backend platforminternal_backend

backend platforminternal_backend
    balance roundrobin
    server api01p 192.168.1.2:80 check
    server api02p 192.168.1.3:80 check
      

listen stats :1936
    mode http
    stats enable
    stats uri /
    stats hide-version
    stats auth admin:admin

Note that the frontend is an IP address that isn’t currently on the box. If you were to start up haproxy right now it would fail since the VIP is not configured on this box. We don’t need to manually add another IP to the server – keepalived will handle this.

Now let’s edit keepalived configuration file – this we will actually have two different configurations, changing the priority in haproxy02p in order to indicate it’s the secondary.

# haproxy01p /etc/keepalived/keepalived.conf
bal_defs {
   notification_email {
     alteremail@alertemail.com
   }
}

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    interface ens160 
    state MASTER
    virtual_router_id 10
    priority 101
    virtual_ipaddress {
        192.168.1.4
    }
    track_script {
        chk_haproxy
    }
}

And for haproxy02p

bal_defs {
   notification_email {
     alertemail@alertemail.com
   }
}

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    interface ens160 
    state MASTER
    virtual_router_id 10
    priority 100
    virtual_ipaddress {
        192.168.1.4
    }
    track_script {
        chk_haproxy
    }
}

It goes to note – a higher priority wins – so 101 beats 100 for priority and 01p becomes the master.

With this all setup go ahead and start keepalived and then HaProxy. You should notice an extra IP address on the master node.

Test out your load balancing and your stat’s page. If everything looks good try to do a fail over. I typically do a simple test by starting a continuous ping to the VIP and hard resetting the master node.

So that’s really all there is too it – nice, quick, and easy win for high availability.

UPDATE:

Thanks to MMDeveloper on reddit I was given a pretty awesome tip here it is verbatim

When I’m setting up haproxy pools I sometimes like to monitor more than one service on each of the HAProxy nodes, I usually script keepalived to use a check script instead of a one-liner.. for example:

vrrp_script chk_loadbalancedservices {
    script "/etc/keepalived/check.sh"
    interval 2
    weight 2
}

contents of check.sh

#!/bin/bash

### This will check all processes you wish to monitor
### In the event one of them is 'down', it will trigger
### a failover to the next-in-line HAProxy server

killall -0 haproxy 2> /dev/null
if [ $? -eq 1 ]
then
    exit 1
fi



### Copy/Paste the above block of code for each
### process you want to monitor
### This script should always exit with a code of 0 or 1
### An exit code of 0 means all is good
### An exit code of 1 will trigger a failover


exit 0

What MMDeveloper is doing here is giving us the ability to check multiple services on the haproxy box to give a more granular approach to fail over. The script is generic and you can place which ever process name you’d like. If you have a critical service that runs next to haproxy on the same box, then add another stanza for this service:

killall -0 otherimportantservice 2> /dev/null
if [ $? -eq 1 ]
then
    exit 1
fi

Now if other important service fails – fail over will occur.