Policy-based routing on Linux

A customer system needed to be upgraded and for several reasons it was supposed to be moved from Windows server to Linux (more on that another time). The server was supposed to be moved from one IP range to another as we are moving from our old Provider Aggregatable (PA) IP addresses to our "new" Provider Indepedent (PI) addresses.

I set the system up and it was supposed to be a hard cutover on one day. It quickly turned out that this was not feasible (again, for several reasons, e.g. the amount of data to copy over was too big). So, finally, it was decided that the cutover was to be smooth: the majority of data was rsync'ed over from Windows in the days before the cutover date, then the rest on the cutover day. The SSL certificate was to be copied over on day X and the new server running Linux should take over the IP address of the old (Windows) server. As the system was now multihomed we needed to cope with asymmetric routing. First off, we thought it should be possible to hide all incoming traffic to the old IP addresses behind the internal IP address of the firewall - but it turned out our product does not allow for that.



The solution to this is policy-based routing: if the packet goes out on Interface_new, a different routing decision needs to be taken than when it would go out Interface_old. Fortunately, Linux does allow for this with the iproute2 package: you can have several routing tables glued together with a routing policy, i.e. a set of rules that controls the selection of the routing table. If a rule matches and a route is selected from a routing table the packet gets routed according to this route. If there is no matching route the rule traversal continues.

In our setup this means, that all packets from IP_PI-Space should get routed to the PI-Space gateway whereas all packets from IP_PA-Space should get routed to the PA-Space gateway. Currently, the default gateway is the PA-Space one, so we don't have to do much.

In Debian/Ubuntu syntax, the network is then setup through /etc/network/interfaces with a stanza like the following:

auto eth0

iface eth0 inet static

        address   A.B.C.D

        netmask  255.255.255.240

        network   A.B.C.128

        broadcast A.B.C.143

        dns-nameservers some.dns.serv.er

        gateway A.B.C.129

And for the Interface in the PI address space, the stanza looks like this:

auto eth1

iface eth1 inet static

        address   V.W.X.Y

        netmask  255.255.255.240

        network   V.W.X.128

        broadcast V.W.X.143

        dns-nameservers some.dns.serv.er

        post-up ip rule add from V.W.X.Y table PI-Space

        post-up ip route add default via V.W.X.129 table PI-Space

The two post-up lines are the magick in here: the first adds a rule to the routing policy that, for all traffic originating from the PI space interface, a lookup should be performed in routing table PI-Space. Then, we add a second default route to that very same routing table. Now, whenever a packet goes out Interface eth1, the kernel checks if there is a matching route in routing table PI-Space. As we have a default route, this will always match and the packet gets routed to the gateway in our PI space.
Obviously, all other traffic originates on eth0 so the "normal" routing table will be used, thus this traffic will go out via the gateway residing in the PA address space.

loose ends

Actually, the kernel does not check a routing table named "PI-Space". It will use a numerical identifier that is mapped in /etc/iproute2/rt_tables like this:

#

# reserved values

#

255     local

254     main

253     default

0       unspec

#

# local

#

#1      inr.ruhep

100     PI-Space