Blog, Rants & An Eclectic Collection of Stuff

If your ISP supports IPv6 and assigns the IP address of your eth0 interface via DHCPv6 you may also be able to obtain one or more additional IPv6 prefixes. These prefixes can be used for other interfaces eth1 or eth2 but also for any virtual interfaces such as docker0, virbr0 and podman0 used for containers or VM’s running on your system.

In NetworkManager, find the connections you are interested in:

Ensure that the interface to your ISP is set to use dhcpv6:

If not, modify it to use dhcp

Now you need to tell NetworkManager which interface to assign the delegated prefix from your ISP. Sadly NetworkManager will only ask for one prefix to be delegated. As a result, only one device can be told use it. Here, the podman cni bridge “cni-podman0” is setup to have the delegated prefix assigned to it.

The magic in Linux which allows this to work are the sysctl values related to IPv6 SLAAC. By default, IPv6 is intended to auto-configure itself on all interfaces:

Now add containers created via podman that run on the “cni-podman0” bridge will obtain IPv6 prefxies which your ISP will recognize.

In this post I’ll show how you can use podman to provide native IPv6 connectivity to containers. This allows direct access to and from the Internet without the need for such gimmicks as NAT and port forwarding.

Install podman

By default, podman will place a CNI configuration file in

At the time of this writing the file 87-podman-bridge.conflist does not provide IPv6 information. We will fix this now.

Create a new file:

which we will modify to use IPv6 so that changes made as a result of these instructions do not touch the podman installed files. We will be keeping the installed podman files as they are.

The CNI networks to use are supplied on the command line, allowing you to choose which network configuration to use. Having an additional configuration file will expose you to this feature. By using a separate configuration file, you can start over if you need to. Once you are comfortable with your changes, you can setup your config files however you like.

Add this file to /etc/cni/net.d/

The IPv6 subnet to use (“subnet”:”2600:3c03:e000:0391::/64″ n the json above) is what your service provider assigns to you. Your provider already knows how to get packets from the network to you and how to get packets from you out to their destination on the network. I’ll talk more about this subnet in section “About that IPv6 Subnet” later.

To use this configuration, just supply the name of the CNI network (not filename) on the podman command. In this example the CNI network name is

podman6, which corresponds to the second like of json (“name”: “podmanv6”) in 88-podmanv6-bridge.conflist file above.

To use this network with podman run:

Here an httpd container is started using the image dperson/nginx and is given the name “http-X”. The name “http-X” is given to the container and it runs on the CNI network named “podman6” which we configured above.

You can find the IPv6 address assigned to the container via:

In this example the address assigned via CNI host-local plugin is

Now from another place on the IPv6 network point your browser to this IPv6 address. Using cURL:

You are connecting from your browser directly to the container via IPv6. There is no need to NAT, or to map port 80 on the host to port 80 on the container.

About That IPv6 Subnet

A subnet, to simplify, gives you a range of IP addresses, typically shown as a prefix and a mask (i.e. prefix/mask.) For example, the IPv4 prefix 10.88.0.0/16 lets you use 64k address all starting with 10.88. A prefix is how I refer to the subnet as a whole, and how the network routes packets to you. The network only cares about the prefix when deciding how to get the packet to its destination. Once at the destination, how that prefix is used is up to you. More on this shortly.

Using IPv4 one is typically allocated just one routable IP address. You probably have one assigned via e.g. DHCP from your ISP for your home or one that is assigned to your Virtual Machine (VM) when using a cloud provider. Even then it’s often the case that the IPv4 address assigned to your VM is routable, but you don’t see the routable address inside your VM. A private, non-routable, RFC1918 IPv4 address (e.g. 192.168.0.1) is used. The cloud provider will use NAT to translate the routable address to an RFC1918 non-routable address inside your VM. Last I checked, the big cloud providers such as AWS, Azure, Google and IBM did this. Due to this limitation, I stopped using them. Another limitation I found was a lack of IPv6 support of any kind, or just providing one IPv6 address.

Smaller providers such as Linode and Digital Ocean (and I’m sure others) do provide your VM with a routable IPv4 address. Because of the ability to get a routable IPv4 address, I use them exclusively. For IPv6, they also provide one routable IPv6 address as well. What’s even better, they will provide multiple IPv6 addresses (i.e. a subnet) of various sizes. With Linode, you can get an entire /64. The ability to have an IPv6 subnet is key to this entire article. Getting an IPv6 subnet is the norm, not the exception. See RFC 7934 for more information.

With only one IPv6 address, hacks like NAT and/or “port forwarding” that, sadly, are common when IPv4 addressing is used would need to also be used with IPv6. These games were played due to the limited number of IPv4 addresses available. IPv6 was invented (over 20 years ago!) to solve this addressing problem. Given the number of IPv6 address available, there isn’t any reason not to leverage them. Some ISP’s will assign an IPv6 Address to a subscriber for the link which connects the subscriber to the ISP. The ISP typically will also supply a “delegated prefix“, of /48, /56 or /64 for use on networks internal to the subscriber via DHCPv6, or some other means. e.g. Linode provides my IPv6 subnets via their web interface.

Another common option for obtaining an IPv6 subnet is to use the Hurricane Electric (HE) tunnel broker. HE will tunnel IPv6 inside IPv4 so that a site that cannot obtain IPv6 from their provider can still access IPv6 networks by tunneling IPv6 packets inside IPv4 packets. HE will provide you with an IPv6 subnet for use at your site and a tunnel endpoint. See Hurricane Electric for specifics.

In the podman IPv6 example above, I used the IPv6 subnet 2600:3c03:e000:0391::/64 provided to me. The network provider will get any packets with a destination IPv6 address matching the first 64 bits of 2600:3c03:e000:0391 (in hexadecimal) to podman, most likely via the interface which connects to your provider, what I’ll refer to as an “uplink”. In some cases yet another IPv6 address is assigned to the uplink, in other cases, just one subnet is provided. In the podman example above, a seperate IPv6 address (not shown) exists on the uplink. I simply had to put the provided subnet in the CNI configuration. The provider takes case of IP routine etc. Things are more interesting when only one subnet is supplied for your entire system, regardless of how many interfaces, physical (eth0, eth1 etc.) or virtual (e.g. podman0, podman1, docker0 etc.)

When only a single subnet is at your disposal all you need to do is subnet the subnet. In the IPv4 case, say your node is given 10.89.0.0/16, and the uplink (eth0) is assigned the address 192.168.0.1. Your provider knows how to get traffic to and from any packet with 192.168.x.x as the prefix. (Note: If this is your home IPv4 router connected to your ISP, you would be using NAT to translate 192.168.x.x address to the routable IPv4 address on the uplink to your ISP.) Once a packet arrives on your node, you can do what you want with the packet. You could assign a network for podman to use 192.168.1.0/24, another, 192.168.2.0/24 for local LAN devices connected via eth1 etc.

With IPv6 the same thing is possible, and there is no need for NAT! To simplify, if given 2001:DB8:1::/48 (HE will provide you a /48!), you can split this up into 64k different /64 subnets, 2001:DB8:1:0::/64 through 2001:DB8:1:FFFF::/64, more than enough for your home, enterprise or VM to use for podman networks.

As the line blurs between nodes that used to just do things such as client and server applications and nodes that would just do networking functions such as Layer 3 routers and Layer 2 switches I find that I have to remind myself, and others about a few things. These are things I know, but need to be reminded of in the heat of the moment. When you are working on a router or switch, being “multi-homed”, that is having multiple interfaces in the node is a given. On a “host” such as node running a web server, one typically receives HTTP requests and reply to them. Typically the reply will be sent via the interface the request arrived on. But this isn’t always the case, in fact, it is typically not the case as I’ll describe here.

As the networking world is more and more virtualized, that is networking functions are being implemented on hosts that have traditionally be servers, it’s important to be reminded of how the “networking” world works.

These concepts exist anywhere TCP/IP is used. But I describe them here using Unix in general, but Linux specifically. Below I’ll describe what I believe are common misconceptions about IP addresses and network interfaces. The main topics covered are:

• IPv4 and IPv6 are UNI-directional not bi-directional
• IP addresses are NOT configured on interfaces but on the node
• RFC 1918 IPv4 addresses are indeed routable

Misconceptions about IP Addresses and Interfaces

On a node with one interface, say eth0 packet flow is obvious. Any packets sent out will egress eth0 and anything to in will ingress eth0. The same if wlan0 is used (and eth0 is now disabled.) The wlan0 interface is used for all traffic. But what happens if both eth0 and wlan0 were enabled at the same time? Or more likely, you are on a Linux server running Apache with more than one interface to the Internet? How about a Bare Metal node that is “dual-homed”? Things are now not as simple as before.

In this post I’ll describe a few misconceptions I’ve noticed over the years related to how IP addresses and interfaces relate to each other.

IPv4 and IPv6 are UNI-directional

The IP protocols (both IPv4 and IPv6) are uni-directional. There is no guarantee that a link used to send a packet to nodeZ will be the same link used when nodeZ sends a packet back to you. When more than one interface exists in a node you can (and often do) send packets on ethX and receive packets on ethY, even for the same TCP connection. This is a feature, not a bug. Being uni-directional allows the network to scale globally (i.e. The Internet.) In this context, interface is logical. You may have just one physical NIC, but that NIC may represent more than one “interface” (e.g. Linux netdev device). For example, multiple VLANs on the same NIC, or the NIC is using macvlan, SRIOV etc.

On Linux you may have already experienced this. The sysctl “net.ipv4.conf.all.rp_filter” used to have a binary value. This binary value needed to change to account for the possibility that a packet isn’t always received from a remote node using the link used by the node to sent to it. When there are more than one interfaces, packets received by the local node do not always arrive via the interface used to send to the destination the packet arrived on.

When a node (nodeA) is connected to the outside world via multiple links, a given node can only control how a packet will egress the node. The local routing table used will indicate which interface, ethX or ethY to use to send the packet to the next node in the path used to reach the packets destination (say nodeZ). This decision is made on each node in the IP network until the packet reaches the node that containers the packets destination.

The same is true in the opposite direction. When the destination of the packet described above, nodeZ wants to send back to nodeA the process repeats, in the reverse direction. Packets in the path may not even traverse the same nodes let alone the same links from nodeZ back to nodeA as those used for nodeA to send to nodeZ.

If you drive in a large city like I do, with one way streets, the concept will look familiar. How you drive (or Uber) from your house to the pub is probably not the same route used to get back home.

I’ll show a simplified example of this on Linux using podman next. Start two containers using podman. Each container will have multiple interfaces, eth0 and eth1. Each eth0 connects to net10 (10.200.10.0/24) and eth1 connects to net11 (10.200.11.0/24).

First let’s setup nodeA.  We will start a container with two interfaces in it.  The interface eth0 will connect to net10, which uses the 10.200.10.0/24 subnet.  The eth1 interface will connect to net11, which uses the 10.200.11.0/24 subnet. 

Now we setup node nodeZ:

Ensure that we can ping from nodeA to nodeZ:

So far so good, nothing new here. Now for the not so typical stuff. What happens when you tell ping to use the address of a different interface (that of eth1 using -I 10.200.11.5) to reach an address (10.200.10.4) which is “configured for” eth0?

On nodeA we do:

We see ping was successful. To see what is going on, let’s look at what was put “on the wire” and “which” wire was used, when. In different windows tcpdump is run to capture packets on on each interface. First eth0:

Notice that the ICMP Echo Request is seen, but not the ICMP Echo Reply. The Echo Request is sent on eth0, but the Echo Reply will arrive via eth1:

On node nodeZ we also run tcpdump on both eth0 and eth1. You will see that the ICMP-Echo arrives on the eth0 interface, and the ICMP-Echo-Reply is sent on the other, eth1 interface. This is the interface pointed to via the route table (shown above) on nodeZ for any packets it sends.

In the first window tcpdump is run on eth0:

The source address is what was “configured” on eth1 on nodeA, but received on eth0. Since the destination of the packet is 10.200.10.4, the route table on nodeA is used to figure out what interfce, eth0, to use to send the packet. The source address isn’t used.

At the same time, in another window, tcpdump is also being run on eth1:

As you see, nodeA sent ping on eth0. Using -I tells ping to use the address 10.200.11.5 “configured” for eth1. It was done just for this explanation. In the real world, the route tables used between source and destination are NOT typically symmetric as in this simple example. That’s the point of this post.

I had to learn this the hard way when I installed my first IPsec offload board in a node. This device dealt with all IPsec encryption. Packets were sent to the device from Unix server and were put on the wire encrypted using IPsec . However packets from the remote end sent back arrived at a different interface, thus bypassing decryption. We had a problem. If you ever wonder why Juniper routers have a dedicated IPsec blade in their chassis, accessible via any port in the chassis, it is to solve this problem.

You might have noticed I place “configured” in double quotes when talking about addresses and interfaces. As I’ll explain shortly, interfaces don’t really have an address. This is another misconception. The interfaces themselves do not have addresses, the nodes does. That’s the topic of the next section.

IP addresses are NOT configured on interfaces

You already know this intuitively, but are so used to the simple case, it is understandable when you forget. Syntax like this example configuration can also mislead you:

One can get the impression from this example configuration syntax that the link has the address, but that’s not accurate. When you only have one interface on your laptop it’s understandable to think this is the case. Likewise if your cloud VM just has an eth0 interface. You are actually doing is configuring an interface for the node itself. In the case of interfaces which connect to broadcast media (aka Ethernet), this familiar command is doing multiple things at once. You are adding an interface to the node itself as mentioned, but also a static route to the subnet reachable via the interface. The assumption is you will want to route to anything on the subnet via this interface. When the interface that has an address assigned to it using ip addr add isn’t on broadcast media (i.e. point to point link), you will not have a route added for you.

If you think about the description of IP protocols being UNI-directional as described above, it starts to become obvious that you are in fact just adding an address to the node. This is exactly how nodeA was able to use any address configured on nodeA (10.200.11.5) to ping nodeZ address 10.200.10.4. The source address didn’t matter; the destination address is used to lookup in the routing table which interface is to be used as the next hop node in the path to that destination. As expected, on nodeA to reach anything on 10.200.10.0/24 eth0 was used. Upon receipt, nodeZ needed to respond to 10.200.11.5. It’s route table said to send the packet via eth1. When nodeA receives a packet on eth1 destined for ANY address local to the node, the packet is accepted. The interface the packet arrived on doesn’t matter. NodeA was able to process a packet destined to it that arrived on an interface configured for a different subnet just fine.

The takeaway here is that ALL IP addresses configured on the node can be reached via any interface. Thik of it this way. If your house has a front door and a back door, regardless of which door I use, I reach you. On broadcast (e.g. Ethernet) networks, if a node receives an ARP for 10.200.11.4 via eth0, IT WILL respond with Ethernet address of eth0 and vice versa for any address that is local to the node.

Those who care about security leverage this. IP addresses are placed on loopback interface only or just on any virtual interface used by the node such as local bridges (br0, virbr0, podman0, docker0, etc.) A “link local” (LL) address is used for all others. With IPv6 link local addresses are common, but they exist for IPv4 too. Using LL addresses in IPv4 has an added benefit of saving (or reclaiming) Global IPv4 addresses..

By only using addresses where they are needed, the attack surface becomes smaller. As shown earlier a node accepts packets for any IP address local to the node, regardless of what interface the packet arrived on. If you have 4, 8, 12, 24 ports etc. (physical or virtual) on your node, and each had a public IP address just for the port to talk to other nodes adjacent to the ports, any of those addresses could be used to hack into the node. Using LL addresses you don’t don’t need firewall rules for “transit” interface address used only between nodes. Nothing except the other end of the link can address your end of the link.

Which brings up another common security (and scaling) aspect. When connecting nodes (which is what networking does), it is better to use /31 prefix (IPv4) and only IPv6 Link Local Addresses. Do this even if you are using broadcast (i.e. Ethernet) media. (If LL is a problem for your site use /127 prefixes.) Now each end of the prefix (i.e. link) is either the .0 or .1 address. For example every ToR node uses .0 and nodes which connect to it (e.g. Unix servers) use .1 There is now no overhead for ARP or Neighbor Discovery.

Absent manually using something like the example syntax above you need a routing protocol, Ansible or orchestration mechanism etc. to configure routes to make use of these interfaces. Modern Data Centers and Enterprises have been doing this for decades. When using routing protocols, it turns out that BGP better at this than OSPF/IS-ISRFC 1918 Addresses are not routable addresses

RFC 1918 IPv4 addresses are not routable

I often hear are that RFC1918 addresses, aka “private IPv4 addresses” are not routable. This is completely false. These are addresses from 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16 address ranges you have probably seen used by e.g. your home router(s). These addresses are not routable ON THE INTERNET. That is no ISP will accept a route advertisement, from you or anyone else including another ISP, with one of these address prefixes. They are routeable on your home network, or on your internal corporate network. I’m able to communicate with colleagues at work who all over the planet, all using an address from the 10.0.0.0/8 prefix. I can guarantee you that we are not all on the same Layer 2 network. Packet can and are able to be routed from one floor to another, between building, between campuses and countries, provided the traffic is intended to stay within the organization.

IPv6 uses ULA (Unique Local Address) for a similar purpose. These addresses are routable and exist for any communication known to be local to the “domain” (e.g. your home network, your company etc.) For example, DNS query for corporate servers would return a ULA to reach that server from inside the company.

If you need to leave your “domain”, your application either picks a globally routable address to use (when possible.) Failing that, NAT can be used, but a/ only at the border to the public network (e.g. your home router connection to your ISP, corporate egress to the Internet etc.) and b/ never for IPv6.