Tapping into Qemu
Qemu - the virtualization software and system emulator by Fabrice Bellard and the fine folks at www.qemu.org is capable of emulating a wide variety of systems. When you pair it up with FreeBSD, and the if_bridge(4) and tap(4) interfaces you can create a complete virtualized network inside your host machine.
This techbits article will show you how to set up and run the virtual machines in the above image. They were developed with Qemu version 7.2.0 along with a number of custom scripts that allow easy bridge and tap setup and virtual machine setup.
If you want, you can download the lab layout and scripts here and follow along.
The host machine file directory layout for the exercises is:
/home/foo /LAB /SCRIPTS _create2bsds.sh (script to create two Qemu disks images) left.sh (script for running 'left' VM host') right.sh (script for running 'right' VM host') mkbr.sh (script to create bridge and tap devices) /BMP FreeBSD_splash_640x480.bmp (splash screen) /ISO fbsd.iso (latest FreeBSD install iso) /VM left.img (Qemu disk image for 'left' hosts) right.img (Qemu disk image for 'right' host)
You will need:
- A laptop or desktop machine, preferably running a 64 bit AMD or Intel CPU with at least 8 GB of memory. More memory will give you better performance when running the host and four or more QEMU virtual machines simultaneously. This will be the host machine, noted as such throughout this article. A large, fast disk (SSD preferred) with at least 256GB is recommended.
- A decent sized monitor to be able to view several (possibly up to 8) terminal windows at once. A screen capable of 1920 x 1080 works nicely.
- A decent Internet connection, preferably wired (gigabit speed recommended)
- A recently downloaded ISO of FreeBSD to install on the QEMU virtual machines. It is, of course, best if you can use the same version on the host and all virtual machines.
- The host machine will need a graphical environment such as X Windows and a decent window manager such as XFCE or similar.
Step by step instructions follow.
If you already have FreeBSD installed on your host machine, skip to the next section.
Install FreeBSD on the host machine on your local network, and update it to the latest patch level with
freebsd-update(8) fetch install.
Note that the default user should be a member of the wheel group.
Install a graphical desktop and window manager on the host machine. The following packages should be sufficient. Installation, even over a fast Internet connection will take some time.
- xorg – Graphical environment, will bring in a lot of dependencies
- xfce4 (or use a different window manager. Use the latest version.)
- qemu – software for virtualization. With Qemu, you get very granular control over all aspects of the virtual machine. The KVM Qemu version is not required (or desired).
- nmap – required for including the ncat(1) program, the version used in this book.
- sudo – required to run virtual machines as root (configure sudo as needed)
- shell_of_your_choice – or just use the Bourne shell if you are old-school.
$ sudo pkg install xorg xfce qemu nmap sudo
Once the host software is installed, you should confgure the graphical environment and window manager to your liking, and double check all network settings.
One note about the xfce4 window manager - its xfce-terminal graphical terminal has features that make using it for these exercises very handy - scriptable tabs and resizing, command execution, and window placement. Some examples are given below.
Installation of the virtual machines and the required packages are described below.
Virtual Machine Install
You’ll need to move the downloaded .iso to the ~/LAB/ISO directory. Also,
the scripts below assume the name to be
fbsd.iso so either rename the
downloaded file, or better, make a link:
$ ln -s FreeBSD-13.1-RELEASE-amd64-dvd1.iso fbsd.iso $ ls -al total 3755526 drwxr-xr-x 2 jpb jpb 4 Mar 26 17:20 . drwxr-xr-x 5 jpb jpb 5 Mar 26 17:33 .. -rw-r--r-- 1 jpb jpb 4621281280 May 16 2022 FreeBSD-13.1-RELEASE-amd64-dvd1.iso lrwxr-xr-x 1 jpb jpb 35 May 16 2022 fbsd.iso -> FreeBSD-13.1-RELEASE-amd64-dvd1.iso
The next step is to set up the two virtual machines. This involves creating a Qemu file-based disk image for each and installing FreeBSD on the images from the same ISO you used for the host install. Follow these steps for installing the virtual machines:
- Change directory to the location for virtual machines ( ~/LAB/VM). Run the qemu-img command to create two images – one for the “left” VM and the other for the “right” VM:
$ qemu-img create -f raw -o size=15G left.img $ qemu-img create -f raw -o size=15G right.img
Note that Qemu uses a technique called “copy on write” for managing disk space. This technique does not require a complete 15G disk, just a stub version that is managed directly by the VM. The disk grows dynamically as the VM writes to it.
The naming conventions above are used throughout this article, but you can name them whatever you want.
Virtual Machine Network
Before you start the Qemu software, you will need to set up one software if_bridge(4) devices and two tap(4) devices. The tap devices are used by Qemu as part of the network em(4) devices.
A script, mkbr.sh, is the easiest way to accomplish correctly setting up the bridge and tap devices. You can specify bridge and tap devices, the only requirement being that the bridge come before the tap devices that are connected to it. Here’s an example:
$ sudo /bin/sh mkbr.sh reset bridge0 tap0 tap1 _hostintf_
Substitute your host ethernet interface (igb0, em0, etc.) for “hostintf” in the above command. This script will create bridge0 and attach two taps and the ethernet interface. Run ifconfig -a to see all the devices in their new configuration. It also sets the required sysctls to work with tap(4) devices.
Install the Virtual Machines
Note that this install creates a FreeBSD system with a single interface - em0. The interface is bound to a tap(4) device defined above.
We will assume that the directory layout is as shown above and the FreeBSD iso file is in /home/foo/LAB/ISO/fbsd.iso.
The below script will start Qemu set to boot the fbsd.iso image and configure the virtual machine with 4GB memory, a vga card, hard disk, and one ethernet port.
Put the below text into a script file (e.g. left.sh) execute this script under sudo(8). Note the use of relative file paths.
#!/bin/sh # Basic Qemu vm startup script for booting FreeBSD /usr/local/bin/qemu-system-x86_64 -monitor none \ -cpu qemu64 \ -vga cirrus \ -m 4096 \ -cdrom ../ISO/fbsd.iso \ -boot order=cd,menu=on,splash=BMP/FreeBSD_splash_640x480.bmp,splash-time=3000 \ -drive if=none,id=drive0,cache=none,aio=threads,format=raw,file=../VM/left.img \ -device virtio-blk,drive=drive0 \ -netdev tap,id=nd0,ifname=tap0,script=no,downscript=no,br=bridge0 \ -device e1000,netdev=nd0,mac=02:20:6c:65:66:74 \ -name \"left\"
Qemu will start up a virtual machine and boot the FreeBSD iso.
TIP: If you click into the Qemu console window, it will “grab” the mouse. To get your mouse back, type “Ctl-Alt-G”.
During the installation there are several points to note:
- Select to use UFS as the filesystem. ZFS does not perform well with small memory sizes.
- Do not select to use ntpd or ntpdate.
- For em0, use 203.0.113.10 , netmask 255.255.255.0, with default gateway 203.0.113.100.
- Do not fill in any DNS nameservers or search path.
- When adding the default user, ensure they are a member of the wheel group.
Once the installation completes, the virtual machine should reboot into the newly installed FreeBSD image. Several additional steps are required to configure the virtual machine:
Login as root. To update the system, you will first have to override the network settings to get a working DHCP address for the VM. (Note that these instructions only work on a wired system and will not work on a wireless only system. Ensure that the host ethernet interface is connected to the same bridge as em0.)
The examples here are shown for the left VM. When making changes to the right VM, use 203.0.113.50 .
# ifconfig em0 203.0.113.10 delete (this only changes the settings for this session) # dhclient em0 (this will load a valid IP on the local LAN) # freebsd-update fetch install (download all OS patches since the release) # reboot (reboot into latest patch level) Note: you may need to run the `left.sh` or `right.sh` script again to restart the Qemu virtual machine.
Login as root again and perform the same network override as above. You can now add any packages you want for your system:
# ifconfig em0 203.0.113.10 delete # dhclinet em0 # pkg install cmdwatch nmap lynx nginx bash # reboot
That’s all you need for the left VM.
Put the below text into a script file (e.g. right.sh) and execute this script under sudo(8) to create the right VM:
#!/bin/sh # Basic Qemu vm startup script for booting FreeBSD /usr/local/bin/qemu-system-x86_64 -monitor none \ -cpu qemu64 \ -vga cirrus \ -m 4096 \ -cdrom ../ISO/fbsd.iso \ -boot order=cd,menu=on,splash=BMP/FreeBSD_splash_640x480.bmp,splash-time=3000 \ -drive if=none,id=drive0,cache=none,aio=threads,format=raw,file=../VM/right.img \ -device virtio-blk,drive=drive0 \ -netdev tap,id=nd0,ifname=tap1,script=no,downscript=no,br=bridge0 \ -device e1000,netdev=nd0,mac=02:72:69:67:68:74 \ -name \"right\"
Once the second VM is installed, update it as shown above.
You may find it desireable to disable sendmail. Put the following statements at the end of /etc/rc.conf:
sendmail_enable="NONE" sendmail_submit_enable="NO" sendmail_outbound_enable="NO" sendmail_msp_queue_enable="NO"
On the next reboot, sendmail and its associated components will not start.
Adding the Serial Port Interface to the VMs
You may have noticed that we have not added the serial interfaces yet.
Each Qemu virtual machine (VM) running FreeBSD needs ‘console=“comconsole” '
/boot/loader.conf to allow use of the Qemu serial port. The serial
port is redirected to a TCP port on the host system at VM startup and
the Qemu monitor waits to activate the guest VM until a telnet(1) connection
occurs on that port. Once the FreeBSD system starts booting and recognizes
the console directive in
/boot/loader.conf it starts up a serial console.
The Qemu monitor detects this and directs the necessary character I/O on
that serial port to the TCP port on the host.
It is important to note that the this serial redirect over TCP takes place outside the virtual machine. There is no interaction with any network on the virtual machine and thus it is not subject to any firewall rules. Think of it just like a dumb terminal sitting on an RS-232 port on a real machine.
Also required are changes to the startup scripts for
Consider copying them as
and making the below changes.
#!/bin/sh # Basic Qemu vm startup script for booting FreeBSD echo echo "NOTE!!! telnet server running!" echo "To start QEMU, start another session and telnet to localhost 4410 " echo /usr/local/bin/qemu-system-x86_64 -monitor none \ -serial telnet:localhost:4410,server=on,wait=on \ -display gtk \ -cpu qemu64 \ -vga cirrus \ -m 4096 \ -cdrom ../ISO/fbsd.iso \ -boot order=cd,menu=on,splash=BMP/FreeBSD_splash_640x480.bmp,splash-time=3000 \ -drive if=none,id=drive0,cache=none,aio=threads,format=raw,file=../VM/left.img \ -device virtio-blk,drive=drive0 \ -netdev tap,id=nd0,ifname=tap0,script=no,downscript=no,br=bridge0 \ -device e1000,netdev=nd0,mac=02:20:6c:65:66:74 \ -name \"left\"
#!/bin/sh # Basic Qemu vm startup script for booting FreeBSD echo echo "NOTE!!! telnet server running!" echo "To start QEMU, start another session and telnet to localhost 4450 " echo /usr/local/bin/qemu-system-x86_64 -monitor none \ -serial telnet:localhost:4450,server=on,wait=on \ -display gtk \ -cpu qemu64 \ -vga cirrus \ -m 4096 \ -cdrom ../ISO/fbsd.iso \ -boot order=cd,menu=on,splash=BMP/FreeBSD_splash_640x480.bmp,splash-time=3000 \ -drive if=none,id=drive0,cache=none,aio=threads,format=raw,file=../VM/right.img \ -device virtio-blk,drive=drive0 \ -netdev tap,id=nd0,ifname=tap1,script=no,downscript=no,br=bridge0 \ -device e1000,netdev=nd0,mac=02:72:69:67:68:74 \ -name \"right\"
Running these scripts is a bit different. Here’s an example running
[jpb@jpbsd:~/LAB/SCRIPTS]$ sudo /bin/sh right+serial.sh NOTE!!! telnet server running! To start QEMU, start another session and telnet to localhost 4450 qemu-system-x86_64: -serial telnet:localhost:4450,server=on,wait=on: info: QEMU waiting for connection on: disconnected:telnet:::1:4450,server=on
Start up another terminal session and telnet to port 4450:
[jpb@jpbsd:~]$ telnet localhost 4450
The new terminal session starts up the Qemu VGA console and starts up the boot process in the session as if on a real serial port.
Eventually you end up with a console session, and a serial session as shown in Figure 3:
You now have two working virtual machines with a console and a serial port session for each one as shown in Figure 4:
Notes on Using the Serial Console
In the VMs, use “vidcontrol black lightwhite” to reverse the video if you want. It has no effect on the serial console. Also, on the serial console if you resize the window, execute the resizewin(1) command to update the terminal size.
Disabling syslog messages to the console in the virtual machines
You may find it desirable (even necessary) to stop syslog message from being sent to the console (either the Qemu console, or the serial port).
To configure syslog to stop logging to the console perform thes steps as root:
# # Configure a file to receive console messages: # # touch /var/log/console.log # # chmod 0600 /var/log/console.log #
Then, modify the line in
/etc/syslog.conf to appear (instead of
And, if necessary:
# service syslogd restart
All messages previously bound for the console, will be directed to