Dynamic haproxy config in SaltStack

If you are anything like me, you like to over utilize haproxy. Salt stack has a nice feature for files that allow you to add to them in different state files.

Say you want to have a stock haproxy config but sick and tired of having to maintain 5 different versions of a haproxy config.

So our basic formula layout looks like this

– packages/haproxy.sls
– web/init.sls

The haproxy.sls stores our basic haproxy install, template and makes sure the service is running

The web/init.sls is what we might apply to our app servers that want access to mysql and kafka.

So pretty much you have have as many of the file.accumulated states as you wish as long as they all have the same name unless you want to have multiple loops in your template. As long as you include the packages/haproxy.sls file in any other state file this will work.

guestfish problems with virt-filesystems

I was trying to use guestfish to increase a qcow2 partition without booting live and fdisking and all that mess. So I tried to run it and was getting

# virt-filesystems --long --parts --blkdevs -h -a disk.qcow2
libguestfs: error: /usr/bin/supermin-helper exited with error status 1.  
To see full error messages you may need to enable debugging.  
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs  

Scratching my head a bit and figured out you need to update the guestfs appliance packages


Now the command works!

# virt-filesystems --long --parts --blkdevs -h -a disk.qcow2
Name       Type       MBR  Size  Parent  
/dev/sda1  partition  83   3.8G  /dev/sda
/dev/sda2  partition  82   976M  /dev/sda
/dev/sda   device     -    5.0G  -

Openvswitch / KVM / Libvirt / Ubuntu / VLANs the right way

There are a lot of old blog posts out there to getting KVM guests to use different vlans via openvswitch. There are a lot that tell you to create fake bridges or create the ports via ovs-vsctrl and add tell libvirt to use that created interface or portgroup. Then there are almost no blogs that really say, when you setup openvswitch, this is how you make the interface settings stick. The correct way to do it is this basic flow 1) Create a bridge via ovs-vsctrl 2) Add your working interface to the bridge via ovs-vsctrl 3) Set your ip info on the new bridge 4) Create a libvirt network 5) Select the port group you want to use from your new network on the guest xml via libvirt 6) When the guest starts if the interface for the vlan isn’t created it will auto create it in openvswitch for you. So this works with Ubuntu 14.04 This also assumes bonding is setup via LACP on the host. It works the same if you just have a single interface like eth0. Just remove all the bond options. So my starting ifconfig for my bond0 device looks something like

bond0     Link encap:Ethernet  HWaddr 00:25:90:ed:dc:f0  
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::f1:41ff:fe72:a331/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:713943 errors:0 dropped:0 overruns:0 frame:0
          TX packets:390750 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:49037015 (49.0 MB)  TX bytes:674651803 (674.6 MB)

So the first thing we want to do is install openvswitch-switch apt-get install openvswitch-switch Now we need to create a bridge in openvswitch ovs-vsctrl add-br br0 Now we need to add our working interface to the bridge. THIS WILL CAUSE YOUR CONNECTION TO DROP. Do not run this command if you don’t have remote KVM access or on the console.ovs-vsctrl add-port br0 bond0 Now that we have a bridge setup we need to give it IP information

ifconfig bond0 0  
ifconfig br0 netmask  
route add default gw  

So now your bridge interface is up and it uses bond0 still. We gave it the same IP information. Now lets setup your the following file so the system reboots correctly

# The loopback network interface
auto lo  
iface lo inet loopback

auto p1p1  
iface p1p1 inet manual  
  bond-master bond0

auto p1p2  
iface p1p2 inet manual  
  bond-master bond0

auto bond0  
allow-br0 bond0  
iface bond0 inet manual  
  bond-mode 4
  bond-miimon 100
  bond-lcap-rate 1
  xmit_hash_policy layer3+4
  bond-slaves none
  ovs_bridge br0
  ovs_type OVSPort
  pre-up ifconfig $IFACE up
  post-down ifconfig $IFACE down

auto br0  
allow-ovs br0  
iface br0 inet static  
  ovs_type OVSBridge
  ovs_ports br0
  pre-up ifconfig $IFACE up
  post-down ifconfig $IFACE down

The big things to add/change are as follows

  • *allow-br0 bond0  *This tells ovs to use bond0
  • Make sure your bond0 interface is set to manual
  • Also add the pre-up/post-up lines and address line to make sure the interface comes up ok
  • ovs_bridge br0 tells the system bond0 is apart of the ovs bridge br0
  • ovs_type OVSPort tells the system that this is a port in ovs
  • allow-ovs br0 tells the system this is for ovs
  • ovs_type OVSBridge tells the system this is a bridge
  • ovs_ports br0

Now that’s all set you can run reboot and the bridge should come up just fine Now lets create a network. Here is my sample network file. It creates a network with an un-tagged port and 2 other ports that get tagged with vlans 2 and 3

 <forward mode='bridge'/>
 <bridge name='br0'/>
 <virtualport type='openvswitch'/>
 <portgroup name='vlan-01' default='yes'>
 <portgroup name='vlan-02'>
     <tag id='2'/>
 <portgroup name='vlan-03'>
     <tag id='3'/>

So you’ll want to change the name of the network group and also the vlan info. My first vlan is un-tagged. and the next two are tagged. So create a file called vlans.xml and put that in now we can load it in libvirt

virsh net-define ./vlans.xml  
virsh net-start vlans  
virsh net-autostart vlans  

Once that is all setup you can define an interface like

<interface type='network'>  
 <source network='vlans' portgroup='vlan-02'/>

So my example if I show my running set looks like

root@vmnode2:~# ovs-vsctl show  
    Bridge "br0"
        Port "vnet1"
            tag: 3
            Interface "vnet1"
        Port "br0"
            Interface "br0"
                type: internal
        Port "bond0"
            Interface "bond0"
        Port "vnet0"
            tag: 2
            Interface "vnet0"
    ovs_version: "2.0.1"

This way we don’t have to tell the guests to tag their traffic going out and we just have openvswitch tag the traffic. One gotcha might be your hardware switch has to know about the vlan ids even if you trunk the port the KVM host is connected to. In cisco that is like

vlan 2  
name WebVlan  

Simple as that.

Get Mandos working in Ubuntu

I’ve been doing a lot of playing around with full dis encryption. Now there’s one big problem when you do full disk encryption is when the server reboots you are left at a prompt to enter your password to mount the drive. This is solved by a tool call mandos. This is a client/server tool that the mandos client is loaded into the initrd image on the server and on boot will query the server and if the server will send back the encryption key to the client to use.

So the issue is the packages just don’t work in ubuntu 12.04 and even 14.04. I have a patch you can apply to your source if you want to rebuild the packaged versions to make debs of your own.

Below is the patch. This works for 14.04 but is basically the same for 12.04. I think the initrd script is slightly different but you can get the gist of it.

--- mandos-1.6.0.orig/initramfs-tools-hook
+++ mandos-1.6.0/initramfs-tools-hook
@@ -148,11 +148,7 @@ for hook in /etc/mandos/network-hooks.d/

 # GPGME needs /usr/bin/gpg
-if [ ! -e "${DESTDIR}/usr/bin/gpg" \
-    -a -n "`ls \"${DESTDIR}\"/usr/lib/libgpgme.so* \
-        2>/dev/null`" ]; then
-    copy_exec /usr/bin/gpg
+copy_exec /usr/bin/gpg

 # Config files
 for file in /etc/mandos/plugin-runner.conf; do
--- mandos-1.6.0.orig/mandos-keygen
+++ mandos-1.6.0/mandos-keygen
@@ -231,8 +231,12 @@ if [ "$mode" = keygen ]; then

     # Generate a new key in the key rings
     gpg --quiet --batch --no-tty --no-options --enable-dsa2 \
-    --homedir "$RINGDIR" --trust-model always \
-    --gen-key "$BATCHFILE"
+        --homedir "$RINGDIR" \
+        --import-ownertrust 

If anyone wants working packages for this let me know and I can post them for 12.04 and 14.04.

Verify user's password on the command line

If there’s any chance you need to verify a user’s password on the command line and you are root you can use openssl with the info from /etc/shadow.

So first we want to grab the entry from /etc/shadow

cat /etc/shadow | grep mike

That will give us something that looks like


So the items we want are the $6 and the $tCFXiZHH. The $6 is important because that tells us the password is using sha512 for encryption. And the $tCFXiZHH is the salt.

So now we can run

mkpasswd -m sha-512 somePasswordHere tCFXiZHH

The output should match up with what’s above and if it is.. you have a valid password.