Testing Endian Aware Code

posted by Stephan Brumme

Turning Around

CPUs store numbers in two different ways: little or big endian. The majority of CPUs are designed to access memory in Little Endian way which seems to have the bytes mixed the wrong way. The memory layout of the hexadecimal number 0x12345678 looks like this:
byte 0 byte 1 byte 2 byte 3 Typical CPUs
Little Endian 0x78 0x56 0x34 0x12 Intel/AMD x86, x64
Big Endian 0x12 0x34 0x56 0x78 PowerPC, ARM*
Note: ARM chips often support little endian, too.

Virtual Machines / QEMU

I don't have a big endian computer at home (my ARM chips are little endian by default) but some of my source code required testing that everything works as expected on big endian machines, too. That's when I learnt about QEMU, a great open-source processor emulator. It's available on Unix and Windows.

This posting explains how to setup QEMU and run Debian Wheezy on an emulated big endian PowerPC.

Installing QEMU

On Linux distributions you can easily install QEMU with your favorite package manager.
# Fedora, Red Hat yum install qemu # Debian, Ubuntu apt-get install qemu
As of today, there is no pre-compiled Windows version available for download but on the QEMU homepage they provide links to some unofficial binaries.
Eric Lassauge's QEMU 2.2 binaries worked out-of-the-box without installation - just unzip the archive. The download is quite huge (more than 100MB) and hosted on a pretty slow web server, so please be patient.

Alternatively, you can download a 21MB zip archive from my fast server, which contains only the PowerPC emulator:
Download  qemu-2.2-powerpc-on-windows.zip
Latest release: January 29, 2015, size: 21.5 MBytes

CRC32: b4ec8ff4
MD5: e892e6655e11cb68c263fc640e42ce1c
SHA1: b7c175bcdf6ec3f849ec8edbd713204d23868377

Debian Wheezy on PowerPC

I found Debian Wheezy for PowerPC on the official Debian servers in QEMU's own QCOW2 file format.
Using the QEMU PowerPC emulator (on Windows), the image can be run using this command line:
# look for BIOS in the "bios" subdirectory # virtual machine gets 1GByte memory # output windows is 1024x768 pixels, 256 colors qemu-system-ppcw -hda debian_wheezy_powerpc_standard.qcow2 -L bios -m 1G -g 1024x768x8
This Debian Wheezy image is more or less the most basic Debian installation. It doesn't even come with GCC or SSH.
Network access works instantly and so I started apt-get / aptitude right away: You can SSH on port 2222 into the virtual machine if you change the QEMU start-up parameters to:
# map port 2222 to port 22 (i.e., connect via ssh to localhost:2222 # which is mapped by QEMU to the virtual machine's port 22) qemu-system-ppcw -hda debian-wheezy-powerpc.qcow2 -L bios -m 1G -g 1024x768x8 -redir tcp:2222::22
My QCOW2 image - which I refreshed in January 2015 - can be downloaded here:
Download  debian-wheezy-powerpc.qcow2
Latest release: January 31, 2015, size: 217.8 MBytes

CRC32: ca6a5482
MD5: 1217e548176807a9e035a41d320dc0f7
SHA1: 2ea7c41e7062d090ce7427c9451395003d59488c

The logins remain unchanged:
account: root => password: root account: user => password: user
As expected, this little program prints "big endian":
hide endian.c // ////////////////////////////////////////////////////////// // endian.c // Copyright (c) 2013-2015 Stephan Brumme. All rights reserved. // see http://create.stephan-brumme.com/disclaimer.html // #include <stdio.h> int main(int argc, char* argv[]) { // on little endian systems, twoBytes is stored as 0x01, 0x00 and // on big endian systems, twoBytes is stored as 0x00, 0x01 short twoBytes = 0x0001; // get first byte of twoBytes char oneByte = *(char*) &twoBytes; if (oneByte == 1) puts("little endian"); else puts("big endian"); return 0; }

Improving QCOW2 compression

The QCOW2 file grew each time I updated the Debian image. After an hour or so it was bigger than 1 GByte. At first I got rid of all locales (localepurge) and fonts. Unused packages like BIND9 were removed, too. Then I cleaned up the apt-get cache. Surprisingly, the QCOW2 file became even bigger.

QCOW2 is a compressed file format. However, all changes and all new files are written uncompressed. Therefore I converted my Debian image from QCOW2 to QCOW2 (yes, same input and output file format !).
# flags: -c enable compression # -p show progress # -O qcow2 enforce QCOW2 output file format qemu-img convert -p -c -O qcow2 debian-wheezy-powerpc.qcow2 shrinked.qcow2 # overwrite image with compressed clone mv shrinked.qcow2 debian-wheezy-powerpc.qcow2
A posting on serverfault.com came to the rescue:
On most file systems, deleted files are not deleted - only their metadata is flagged as "deleted".
The apparently deleted file contents is almost always not as good compressible as "empty" space consisting of zeros. If I create a very large file full of zeros then at some point the deleted file's contents will be overwritten by this new dummy file. So I ran:
# file consisting of zeros => until we run out of space # use block size of 16 MBytes to speed up dd dd if=/dev/zero of=zerofill bs=16M
It took dd about 15 minutes until it ran out of disk space and stopped. Then I typed in:
rm zerofill shutdown now -h
After calling qemu-img (see above) to convert from QCOW2 to QCOW2 again the result exceeded my wildest expectations: the final Debian Wheezy image is about 25% smaller (217MB vs. 266MB) than the original Debian Wheezy and pretty much up-to-date.

zerofree is a dedicated tool which fills all unused sectors with zeros. It should be faster than the aforementioned approach because the underlying QCOW2 file doesn't grow during the process. While it does write only a few bytes (zeroing those dangling unused sectors), it still needs to read all sectors.

In the end, it's only slightly faster. A major drawback is that zerofree only works on readonly-mounted partitions.
Therefore we have to terminate all programs that have open files. In my basic image we just have to stop the networking daemon:
service networking sto mount -o ro,remount / zerofree -v /dev/sda3
Obviously, you still have to run qemu-img convert on our host system to actually shrink the image.