Testing Endian Aware Code
posted by Stephan Brumme
Turning Around
CPUs store numbers in two different ways: little or big endian. The majority of CPUs are designed to access memory in Little Endian way which seems to have the bytes mixed the wrong way. The memory layout of the hexadecimal number 0x12345678 looks like this:byte 0 | byte 1 | byte 2 | byte 3 | Typical CPUs | |
---|---|---|---|---|---|
Little Endian | 0x78 | 0x56 | 0x34 | 0x12 | Intel/AMD x86, x64 |
Big Endian | 0x12 | 0x34 | 0x56 | 0x78 | PowerPC, ARM* |
Virtual Machines / QEMU
I don't have a big endian computer at home (my ARM chips are little endian by default) but some of my source code required testing that everything works as expected on big endian machines, too. That's when I learnt about QEMU, a great open-source processor emulator. It's available on Unix and Windows.This posting explains how to setup QEMU and run Debian Wheezy on an emulated big endian PowerPC.
Installing QEMU
On Linux distributions you can easily install QEMU with your favorite package manager.
# Fedora, Red Hat
yum install qemu
# Debian, Ubuntu
apt-get install qemu
Eric Lassauge's QEMU 2.2 binaries worked out-of-the-box without installation - just unzip the archive. The download is quite huge (more than 100MB) and hosted on a pretty slow web server, so please be patient.
Alternatively, you can download a 21MB zip archive from my fast server, which contains only the PowerPC emulator: Latest release: January 29, 2015, size: 21.5 MBytes
CRC32:
b4ec8ff4
MD5:
e892e6655e11cb68c263fc640e42ce1c
SHA1:
b7c175bcdf6ec3f849ec8edbd713204d23868377
SHA256:
184c0cc9de6e93fcc99b3d919ed7e0da86bcaf66c097844ac30e9bd35660c821
If you encounter any bugs/problems or have ideas for improving future versions, please write me an email: create@stephan-brumme.com
Debian Wheezy on PowerPC
I found Debian Wheezy for PowerPC on the official Debian servers in QEMU's own QCOW2 file format.Using the QEMU PowerPC emulator (on Windows), the image can be run using this command line:
# look for BIOS in the "bios" subdirectory
# virtual machine gets 1GByte memory
# output windows is 1024x768 pixels, 256 colors
qemu-system-ppcw -hda debian_wheezy_powerpc_standard.qcow2 -L bios -m 1G -g 1024x768x8
Network access works instantly and so I started
apt-get
/ aptitude
right away:
- Perl 5.14 and Python 2.7 were pre-installed
- GCC was updated to 4.9 (from Debian testing repositories)
- I switched to Linux kernel 3.16 (from Debian testing, too)
# map port 2222 to port 22 (i.e., connect via ssh to localhost:2222
# which is mapped by QEMU to the virtual machine's port 22)
qemu-system-ppcw -hda debian-wheezy-powerpc.qcow2 -L bios -m 1G -g 1024x768x8 -redir tcp:2222::22
CRC32:
ca6a5482
MD5:
1217e548176807a9e035a41d320dc0f7
SHA1:
2ea7c41e7062d090ce7427c9451395003d59488c
SHA256:
717ff5aa37f137fa759acebd2bfa03f583dc70bfcd8c991f48d0830ef7317112
If you encounter any bugs/problems or have ideas for improving future versions, please write me an email: create@stephan-brumme.com The logins remain unchanged:
account: root => password: root
account: user => password: user
big endian
":
hide
endian.c
// //////////////////////////////////////////////////////////
// endian.c
// Copyright (c) 2013-2015 Stephan Brumme. All rights reserved.
// see http://create.stephan-brumme.com/disclaimer.html
//
#include <stdio.h>
int main(int argc, char* argv[])
{
// on little endian systems, twoBytes is stored as 0x01, 0x00 and
// on big endian systems, twoBytes is stored as 0x00, 0x01
short twoBytes = 0x0001;
// get first byte of twoBytes
char oneByte = *(char*) &twoBytes;
if (oneByte == 1)
puts("little endian");
else
puts("big endian");
return 0;
}
Improving QCOW2 compression
The QCOW2 file grew each time I updated the Debian image. After an hour or so it was bigger than 1 GByte. At first I got rid of all locales (localepurge
) and fonts. Unused packages like BIND9 were removed, too.
Then I cleaned up the apt-get
cache.
Surprisingly, the QCOW2 file became even bigger.QCOW2 is a compressed file format. However, all changes and all new files are written uncompressed. Therefore I converted my Debian image from QCOW2 to QCOW2 (yes, same input and output file format !).
# flags: -c enable compression
# -p show progress
# -O qcow2 enforce QCOW2 output file format
qemu-img convert -p -c -O qcow2 debian-wheezy-powerpc.qcow2 shrinked.qcow2
# overwrite image with compressed clone
mv shrinked.qcow2 debian-wheezy-powerpc.qcow2
On most file systems, deleted files are not deleted - only their metadata is flagged as "deleted".
The apparently deleted file contents is almost always not as good compressible as "empty" space consisting of zeros. If I create a very large file full of zeros then at some point the deleted file's contents will be overwritten by this new dummy file. So I ran:
# file consisting of zeros => until we run out of space
# use block size of 16 MBytes to speed up dd
dd if=/dev/zero of=zerofill bs=16M
dd
about 15 minutes until it ran out of disk space and stopped. Then I typed in:
rm zerofill
shutdown now -h
qemu-img
(see above) to convert from QCOW2 to QCOW2 again the result exceeded my wildest expectations:
the final Debian Wheezy image is about 25% smaller (217MB vs. 266MB) than the original Debian Wheezy and pretty much up-to-date.zerofree
is a dedicated tool which fills all unused sectors with zeros.
It should be faster than the aforementioned approach because the underlying QCOW2 file doesn't grow during the process.
While it does write only a few bytes (zeroing those dangling unused sectors), it still needs to read all sectors.In the end, it's only slightly faster. A major drawback is that
zerofree
only works on readonly-mounted partitions.Therefore we have to terminate all programs that have open files. In my basic image we just have to stop the networking daemon:
service networking sto
mount -o ro,remount /
zerofree -v /dev/sda3
qemu-img convert
on our host system to actually shrink the image.