IPv6 subnetting on FreeBSD with VNET jails

Microsoft are doing some really wonky stuff not delivering messages to my MX. I’ve only been able to work around that by changing the IP’s of my incoming mail server. Now routable IPv4 space doesn’t exactly grow on trees for hobbyists on a tight budget. My box at Hetzner has a single address and I got a single one at home as well. That’s it. Fortunately I got a lot more IPv6 at home, but to my surprise that was by far the hardest part to configure. I nailed it though, so this article is something of a note-to-future-self, hoping it may also help others who have issues supernetting IPv6 to a FreeBSD host with a bridge interface and epairs handling jails.

My router at home runs OPNSense and I got a /48 from my upstream provider. I created a VLAN on my network named ‘Hosting’ where I placed a hastily scrounged Raspberry Pi 4B to serve as an MX. The network now roughly looks like this.

The Raspberry Pi has a genet0 interface that’s on untagged VLAN 200 configured with 10.200.0.2/24 for its IPv4 address and 2a02:a45f:d306:200:2::2/64 for IPv6. This is the host interface that’s really only exposed for management purposes.

Inside the Pi’s OS I have bridge0 with 10.200.1.1/24 and 2a02:a45f:d306:200::2::1/64 acting as a gateway for the jails that will host the actual workloads. For now there’s only a single jail, but I may add a few more for testing purposes later.

The jail for my new MX is called mx2 and has epair network interfaces. These are a FreeBSD-ism that creates two network interfaces: epair0a and epair0b that behave as though a wire is between them. You put a into your host’s L2 network and b goes into the jail itself. In practice that means that epair0a is a member of bridge0.

In order for OPNSense to understand where to push the traffic for a /56 we can’t use the simple ’track interface’ construct from the web GUI. Instead, I set up a static IPv6 address of 2a02:a45f:d306:200:2::1 on the OPNSense side and crated a ‘Gateway’ (in System -> Gateways) for my Raspberry Pi with 2a02:a45f:d306:200:2::2 as the address and Hosting as the interface. This gives OPNSense a remote gateway to work with.

You also need a static route in OPNSense (System -> Routes) to get L3 to play nice. You tell it to throw all of 2a02:a45f:d306:200::/56 to the gateway you just defined and traffic should start to flow from the rest of your network to at least the addresses of genet0 and bridge0.

After setting ipv6_gateway_enable="YES" in /etc/rc.conf we should get IPv6 forwarding up and things should now be able to reach my MX jail, right? Wrong! And this one cost me quite a bit of hair.

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#2                        URS             lo0
default                           2a02:a45f:d306:200::1         UGS          genet0
::1                               link#2                        UHS             lo0
::ffff:0.0.0.0/96                 link#2                        URS             lo0
2a02:a45f:d306:200::/64           link#1                        U            genet0
2a02:a45f:d306:200::2             link#2                        UHS             lo0
2a02:a45f:d306:200:2::1           link#2                        UHS             lo0
fe80::%lo0/10                     link#2                        URS             lo0
fe80::%genet0/64                  link#1                        U            genet0
fe80::e65f:1ff:fe7a:7fc8%lo0      link#2                        UHS             lo0
fe80::%lo0/64                     link#2                        U               lo0
fe80::1%lo0                       link#2                        UHS             lo0
ff02::/16                         link#2                        URS             lo0

This is the routing table of the Pi’s host OS. Things choke on 2a02:a45f:d306:200:2::1 going through lo0. I’d have liked for that to be a network route for the entire /64 that would go through bridge0 instead. When I try to add such a route, FreeBSD complains about already having that in the table. Ugh.

Adding a route per host for each jail got me up and running for now.

route add -6 -host 2a02:a45f:d306:200:2::2 -iface bridge0

This tells the OS where my MX jail lives and traffic starts flowing! Now I have yet to figure out if replacing the route for 2a02:a45f:d306:200:2::1 outright would fix anything, but that’s for another time. For now I got my stuff working and incoming mail is also fixed.