Furor Teutonicus
Export date: Tue Sep 21 10:38:37 2021 / +0000 GMT

Enabling host-guest networking with KVM, Macvlan and Macvtap


The perfect setup, nearly

(Updated on 2020-07-06.)

You installed your Linux server and naturally selected KVM (Kernel Virtual Machine) as hypervisor. Using virt-manager, you also created one or more guest VMs (Virtual Machines).

You want fast networking. So you use the paravirtualized virtio drivers for the guests.

You also want no difference between virtual and non-virtual machines. All should be able to talk over the same LAN, use the same subnet, contact the same DHCP server and talk with each other. So you use the Macvtap 1 driver. Macvtap makes use of Macvlan, also written as MAC VLAN. MAC VLAN 2 allows you to have multiple Ethernet MAC (Media Access Control) addresses on one NIC (Network Interface Card). Network traffic will go directly to and from the physical line to the guest VM. If you enable bridge mode, then all kind-of-virtual NICs attached to the same host (or physical NIC, I'm not sure) can see each other.

It's just so much easier than having to create and manage traditional brctr bridges. And probably it performs better, too.

The problem: the host cannot talk with the guests

The guests can talk to each other. But the host is excluded from the social event. Look at the picture below. Guest 1 and guest 2 are connected using a red line; they are also connected with the eth0 physical NIC of the host. Packets delivered to eth0 will be sent to the network immediately. The hypervisor cannot intercept them.


Solution: create a macvlan interface on the host

If you create a macvlan interface on the host, and use that one instead of eth0, than the host can communicate with the guests. Some people don't like this solution because of bad integration with the NetworkManager, but I like it because I don't have to modify the guests. And I'm using only one host machine, so I can handle that with ease.

I have tested this solution myself on two different computers, both running Scientific Linux 3 6.4 (a RHEL 4 derivative). I also use this script on my Oracle LInux 8.2 server, another RHEL clone. So beware, YMMV 5.

What I did: I wrote a simple shell script that takes care of the creation of and routing to a macvlan interface on the host. So on the host, you have to run this script on startup, e.g. by adding the full path to the script in /etc/rc.local. Or you could load it using Systemd (example given below the script).

Here is the script:

# Let host and guests talk to each other over macvlan.
# Configures a macvlan interface on the hypervisor.
# Run this on the hypervisor (e.g. in /etc/rc.local)
# Made for IPv4; need modification for IPv6.
# Meant for a simple network setup with only eth0,
# and a static (manual) ip config.
# Evert Mouw, 2013. Slightly modified in 2020.


# ------------
# test if interface already exists
# ------------
if ip link show | grep "$MACVLN@$HWLINK" > /dev/null
echo "Link $MACVLN already exists on $HWLINK."

# ------------
# wait for network availability
# ------------

while ! ping -q -c 1 $TESTHOST > /dev/null
echo "$0: Cannot ping $TESTHOST, waiting another 5 seconds."
sleep 5

# ------------
# get network config
# ------------

IP=$(ip address show dev $HWLINK | grep "inet " | awk '{print $2}')
NETWORK=$(ip -o route | grep $HWLINK | grep -v default | awk '{print $1}')
GATEWAY=$(ip -o route | grep default | awk '{print $3}')

# ------------
# setting up $MACVLN interface
# ------------

ip link add $MACVLN link $HWLINK type macvlan mode bridge
ip address add $IP dev $MACVLN
ip link set dev $MACVLN up

# ------------
# routing table
# ------------

# empty routes
ip route flush dev $HWLINK
ip route flush dev $MACVLN

# add routes
ip route add $NETWORK dev $MACVLN metric 0

# add the default gateway
ip route add default via $GATEWAY

Beware: If the underlying eth{n} link is down, then also the macvlan will go to the "down" state. That means that the hardware ethernet link must be up, otherwise macvlan/macvtap based VMs will not be able to communicate with each other, or with the host. Also, NetworkManager can play nasty on your customized routing table when the link comes up again.

Here is an example Systemd unit file:
# /etc/systemd/system/macvlan.service

Description=MACVLAN bridge to share a physical network interface with KVM/QEMU hosts.



Another, pure Systemd / Networkd approach can be found here 6.

The resulting routing table will look like this:
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface U 0 0 0 macvlan0 UG 0 0 0 macvlan0

Guest configuration

The guest must be configured to use macvtap in bridge mode. Typically, in the configuration XML (/etc/libvirt/qemu) you will find:
<interface type='direct'>
<source dev='eth0' mode='bridge'/>

Remember that the guest will then use the DHCP server of the physical LAN. No need any more for the dnsmasq part on the hypervisor. If all your guests use this trick, then you can do:

rm /etc/libvirt/qemu/networks/autostart/*

That removes the bridge interfaces you see when you run ifconfig. If you cannot wait until the next reboot, also do for each network:

virsh net-destroy _network-name_


Acknowledgements / sources

  • Superuser 9 "Guest and host cannot see each other using linux-kvm and macvtap"

  • KVM mailing list, Arnd Bergmann 10 in "Re: Does macvtap support host to guest communication?"

  • libvirt.org 11 "Guest can reach outside network, but can't reach host (macvtap)"

  • crashmag 12 "Linux KVM host to guest connectivity" (link to original article is broken)

  • Gentoo forums 13 "one way communication with kvm+macvlan"

  • Jim's Depository 14 "Notes on Linux's macvlan module"

  • Kernelnewbies 15 MacVTap

  • pocketnix 16 "Linux Networking: MAC VLANs and Virtual Ethernets"

  • Oracle 17 "Configuring Operating System Containers"

  • Red Hat 18 "Interface Configuration Files"

  • Major Hayden 19 "systemd-networkd and macvlan interfaces"

Excerpt: How to use macvlan on the KVM hypervisor with macvtap on KVM guests so that the host and the guests can ping each other and behave like they are all on the same subnet (LAN).
  1. http://virt.kernelnewbies.org/MacVTap
  2. http://www.pocketnix.org/posts/Linux%20Networking: %20MAC%20VLANs%20and%20Virtual%20Ethernets
  3. http://en.wikipedia.org/wiki/Scientific_Linux
  4. http://fedoraproject.org/wiki/Red_Hat_Enterprise_L inux
  5. http://www.urbandictionary.com/define.php?term=YMM V
  6. https://major.io/2015/10/26/systemd-networkd-and-m acvlan-interfaces/
  7. http://www.furorteutonicus.eu/wp-content/uploads/2 013/08/create_macvlan_bridge.sh_.zip
  8. http://www.furorteutonicus.eu/wp-content/uploads/2 013/08/macvtap_macvlan.dia_.zip
  9. http://superuser.com/questions/349253/guest-and-ho st-cannot-see-each-other-using-linux-kvm-and-macvt ap
  10. http://www.spinics.net/lists/kvm/msg52732.html
  11. http://wiki.libvirt.org/page/Guest_can_reach_outsi de_network, _but_can%27t_reach_host_%28macvtap%29
  12. http://crashmag.net
  13. https://forums.gentoo.org/viewtopic-t-961174.html? sid=257bb22f6505b9b3f5471e62ceb32dc4
  14. http://jim.studt.net/depository/index.php/notes-on -linux-s-macvlan-module
  15. http://virt.kernelnewbies.org/MacVTap
  16. http://www.pocketnix.org/posts/Linux%20Networking: %20MAC%20VLANs%20and%20Virtual%20Ethernets
  17. http://docs.oracle.com/cd/E37670_01/E37355/html/ol _config_os_containers.html
  18. https://access.redhat.com/site/documentation/en-US /Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ s1-networkscripts-interfaces.html
  19. https://major.io/2015/10/26/systemd-networkd-and-m acvlan-interfaces/
Post date: 2013-08-04 21:51:24
Post date GMT: 2013-08-04 20:51:24

Post modified date: 2020-07-06 12:31:03
Post modified date GMT: 2020-07-06 10:31:03

Export date: Tue Sep 21 10:38:37 2021 / +0000 GMT
This page was exported from Furor Teutonicus [ https://www.furorteutonicus.eu ]
Export of Post and Page has been powered by [ Universal Post Manager ] plugin from www.ProfProjects.com