Saturday, January 24, 2015

Managing Odroid and "mlinuxguy" changes in git

Here are my notes getting a Debian Linux kernel package built for Odroid C1. I prefer to manage the Linux kernel source under its natural habitat (git) so let's clone it. I'm using the googlesource.com mirror but feel free to substitute another one.
git clone https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable.git linux
cd linux
Now let's add hardkernel's fork as a remote repository. I'm also adding the repository owned by mlinuxguy who has been working on improving the ethernet driver. The benefit of adding these repositories to the same clone is that they can share objects, and I can easily git-diff the changes.
git remote add hardkernel https://github.com/hardkernel/linux.git
git remote add mlinuxguy https://github.com/mlinuxguy/odroid-c1-network-driver.git
git fetch --all
Unfortunately, mlinuxguy only included the files he changed under drivers/amlogic/ethernet but not with the full path, so we need to fix up his commits before we can merge them. We'll create a new local branch and then do a filter-branch to rewrite the commits.
git checkout remotes/mlinuxguy/master -b mlinuxguy-full
git filter-branch --prune-empty --tree-filter '
    mkdir -p drivers/amlogic/ethernet/phy &&
    git ls-tree --name-only $GIT_COMMIT |
    xargs -I FILE mv FILE drivers/amlogic/ethernet/FILE'
Now let's create a branch that merges hardkernel's odroidc-3.10.y branch with the changes from mlinuxguy.
git checkout remotes/hardkernel/odroidc-3.10.y -b odroidc-3.10.y-mlinuxguy
git merge -s recursive -X theirs mlinuxguy-full
Notice: the ethernet's MAC address changed some commits after 764e57, so you will need to reset /etc/udev/rules.d/70-persistent-net.rules; otherwise you might not have network access.
And now we have a kernel tree ready to be built. If you have not done so, install the prerequisite build packages.
sudo apt-get install build-essential kernel-package lzop u-boot-tools initramfs-tools
The Debian make-kpkg program allows you to configure a custom kernel using an existing .config, which we generate using Odroid C1's default.
make odroidc_defconfig
The following command will build linux-image-* and linux-headers-* in the parent directory. It takes about 50 minutes on the Odroid C1 which is not bad (the kernel itself took 20 minutes, the rest are modules and packaging). It saves me the hassle of messing with cross-compilation toolchain. It also means that I get to use Debian's gcc, not some binary from an untrusted third-party.
nice time make-kpkg -j5 --rootcmd fakeroot --initrd --append-to-version=-odroidc+mlinuxguy kernel_image kernel_headers
Note that the dtbs (device tree blob) is not packaged, so if you want to update it, you could make dtbs from the Linux source (the path is shown in the make output) and copy it to /boot manually. You should also be able to just use the old one.

You can then install the Debian packages using the dpkg -i command which installs the vmlinuz-* under /boot. The initramfs-tools hook will create the initrd.img-* file under /boot as well. Note however that vmlinuz and initrd.img has to be converted to uImage and uInitrd for U-Boot. Rather than making the uImage under the linux source tree, we'll generate it from /boot. I'm using the same parameters that Odroid C1 kernel uses as revealed by running mkimage -l on an existing uImage and uInitrd.
cd /boot
mkimage -A arm -O linux -T kernel -C none -a 00208000 -e 00208000 \
    -n "Linux-3.10.44" -d /boot/vmlinuz-3.10.44-odroidc+mlinuxguy \
    /boot/uImage-3.10.44-odroidc+mlinuxguy
mkimage -A arm -O linux -T ramdisk -C none \
    -n "uInitrd 3.10.44" -d /boot/initrd.img-3.10.44-odroidc+mlinuxguy \
    /boot/uInitrd-3.10.44-odroidc+mlinuxguy
(TODO: I could not get any compression working under U-Boot possibly due to version mismatch, but uncompressed works.) (TODO: you could automate this with a kernel install hook, like how update-initramfs works.)

If something fails to boot, you could boot into the original kernel in U-Boot like this:
fatload mmc 0:1 0x21000000 uImage
fatload mmc 0:1 0x21800000 meson8b_odroidc.dtb
fatload mmc 0:1 0x22000000 uInitrd
setenv bootargs "console=ttyS0,115200n8 root=/dev/mmcblk0p2 rootwait ro no_console_suspend vdaccfg=0xa000 logo=osd1,loaded,0x7900000,720p,full dmfc=3 cvbsmode=576cvbs hdmimode=720p m_bpp=32 vout=hdmi disableuhs"
bootm 0x21000000 0x22000000 0x21800000
After the build, to return the source tree back to prestine state (no littered build artifacts), run:
git clean -f -x -d

Tuesday, January 13, 2015

Installing Debian on Odroid C1

After my adventure of installing Debian on Pogo, I decided to ditch the Pogo and go for Odroid C1 which has gigabit ethernet and a quad core armv7 running at 1.5GHz. My objective again is to install Debian from scratch using the installer over serial console, without using a flash image that somebody else prepared.
Warning: The ethernet driver for Odroid C1 is buggy and may have auto-negotiation problems which prevent the Gigabit ethernet from achieving line speed.
Here is my shopping list:
The USB UART Module Kit contains three pieces: the USB cable, a small board with FTDI chip, and a Molex cable. I mostly bought the kit just for the Molex cable because Odroid C1 serial console has a Molex socket. I kind of wished that Hardkernel could just sell the Molex cable separately so I could use the USB-TTL adapter I already had for Raspberry Pi.

The Molex cable could be mated to a commodity USB-TTL adapter by inserting the short-end of the break-away headers to the Molex connector, and the long end to the TTL.

Beware: I found out that you could (accidentally) power the Odroid C1 through the serial port if you connect the red wire, but the Ethernet chip would not receive enough power to work reliably. In my case, packets are transmitted but no packets are received. If you ping host X from the Odroid and run tcpdump on host X, you'd see the ICMP echo and reply, but Odroid C1 never sees the reply.

You will have to disconnect the serial console's power even if you plug in the DC power, or the Odroid will continue to draw power from the serial port.

First I need to flash the eMMC from Odroid's image to ensure the proper bootloader and the partition layout. It would be nice in the future if they could provide a tool to format the eMMC.
  • From: http://odroid.com/dokuwiki/doku.php?id=en:c1_release_linux_ubuntu
  • unxz --keep ubuntu-14.04.1lts-lubuntu-odroid-c1-20150102.img.xz
  • On Mac OS X:
    • Use "diskutil list" to find the eMMC device.
    • Use "diskutil umount" to unmount whichever volume is currently mounted, prior to flashing with the dd command.
    • Use "diskutil eject" to eject the device after the flashing is done.
  • dd if=ubuntu-14.04.1lts-lubuntu-odroid-c1-20150102.img of=/dev/rdisk2 bs=1M
This will result in an eMMC with two partitions. Mac OS X or Windows will see only the boot (FAT32) partition. While you're at it, edit boot.ini and change bootargs from:
setenv bootargs "console=ttyS0,115200n8 root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait ro no_console_suspend vdaccfg=0xa000 logo=osd1,loaded,0x7900000,720p,full dmfc=3 cvbsmode=576cvbs hdmimode=${m} m_bpp=${m_bpp} vout=${vout_mode} ${disableuhs}"
To:
setenv bootargs "console=ttyS0,115200n8 root=/dev/mmcblk0p2 rootwait ro no_console_suspend vdaccfg=0xa000 logo=osd1,loaded,0x7900000,720p,full dmfc=3 cvbsmode=576cvbs hdmimode=${m} m_bpp=${m_bpp} vout=${vout_mode} ${disableuhs}"
That's because we will reformat the filesystem which results in a different UUID. It is also fixable later.

We will now install Debian using Odroid's uImage with the uInitrd of the Debian network installer for efikamx, which you could download from here. I don't know if other installers under armhf work or not; maybe I just got lucky. You can either put the uInitrd under a different name (say uInitrd-deb) into the boot partition (the FAT32 partition that has boot.ini) or TFTP it in later. But we will first boot into Odroid's Xubuntu to resize the root filesystem since Debian installer can't repartition it.

Don't plug in the Ethernet cable the first time, until you change the password for 'root' and 'odroid' over the serial console. Also, while you're at it, change /etc/ssh/sshd_config to disallow root login.
# passwd -d root
# passwd odroid
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
# sed -i 's/\(PermitRootLogin\) .*/\1 no/' /etc/ssh/sshd_config
# service ssh reload
This ensures that nobody could login as root and replace our kernel before we're ready to install Debian. We will be keeping the same kernel for now. Run odroid-utility.sh now to resize the root filesystem to fill the eMMC.

Reboot the Odroid, and press the enter key immediately to abort U-Boot. You will see the "odroidc#" prompt. Now it is safe to plug in the Ethernet cable.
QA5:A;SVN:B72;POC:17F;STS:0;BOOT:0;INIT:0;READ:41;READ:41;READ:0;CHECK:0;PASS:0;
-----------------------------------------------------------------------
* Welcome to Hardkernel's ODROID-C... (Built at 19:33:00 Dec  8 2014) *
-----------------------------------------------------------------------
PU : AMLogic S805
MEM : 1024MB (DDR3@792MHz)
BID : HKC13C0001
S/N : HKC1CC0349B9915D
0x0000009f
Loading U-boot...success.


U-boot-00000-g9f5aee4(odroidc@odroidc-v2011.03) (Jan 02 2015 - 17:56:06)

DRAM:  1 GiB
relocation Offset is: 2ff0c000
MMC:   eMMC: 0, SDCARD: 1
IR init is done!
vpu clk_level = 3
set vpu clk: 182150000Hz, readback: 182150000Hz(0x701)
mode = 6  vic = 4
set HDMI vic: 4
mode is: 6
viu chan = 1
config HPLL
config HPLL done
reconfig packet setting done
MMC read: dev # 0, block # 33984, count 12288 ... 12288 blocks read: OK
There is no valid bmp file at the given address
============================================================
Vendor: Man 450100 Snr 01281340 Rev: 4.7 Prod: SDW32
            Type: Removable Hard Disk
            Capacity: 29820.0 MB = 29.1 GB (61071360 x 512)
------------------------------------------------------------
Partition     Start Sector     Num Sectors     Type
    1                 3072          263168       6
    2               266240         9363456      83
============================================================
Net:   Meson_Ethernet
init suspend firmware done. (ret:0)
Hit Enter key to stop autoboot -- :  1 tstc enter

exit abortboot: 1
odroidc#
Here we will load the original uImage and boot into uInitrd-deb of the Debian installer. You could just TFTP in the Debian installer uInitrd using the tftpload command (not shown).
odroidc#fatload mmc 0:1 0x21000000 uImage
reading uImage

5434708 bytes read
odroidc#fatload mmc 0:1 0x22000000 uInitrd-deb
reading uInitrd-deb

5473116 bytes read
odroidc#fatload mmc 0:1 0x21800000 meson8b_odroidc.dtb
reading meson8b_odroidc.dtb

17748 bytes read
odroidc#setenv bootargs "console=ttyS0,115200n8 rootwait ro no_console_suspend vdaccfg=0xa000 logo=osd1,loaded,0x7900000,720p,full dmfc=3 cvbsmode=576cvbs hdmimode=720p m_bpp=32 vout=hdmi disableuhs"
odroidc#bootm 0x21000000 0x22000000 0x21800000
The bootargs contain some magic incantation that tells the board how to activate the frame buffer. Without them you'd get a kernel panic. The bootargs and other load commands all come from /boot/boot.ini on the eMMC card. After you boot, you should be able to continue with the usual Debian Installer. Make sure that you use /dev/mmcblk0p2 as your root.

If you forgot to fix boot.ini to use root=/dev/mmcblk0p2 instead of the UUID earlier, initramfs will drop to an "(initramfs)" prompt after failing to find root. The following commands fix it by assigning the new root filesystem the same UUID as before:
mount /dev/mmcblk0p2 /root
mount /dev /root/dev
chroot /root
tune2fs -U e139ce78-9841-40fe-8823-96a304a09859 /dev/mmcblk0p2
exit
reboot
The resulting system won't have any kernel modules, but the builtin kernel is functional enough without additional modules. You will be able to install the modules over the network after the system boots normally. See my next post on how to compile a kernel for Odroid C1.

Wednesday, January 7, 2015

Load Balancing Methods and Trade-Offs

Load balancing is the act of distributing service requests from clients to multiple servers that can perform a service. There are many excellent articles about load balancing, particularly the mechanisms used for the web, but few understand the trade-offs. The mechanisms can be roughly classified as client-side or server-side load balancing.
  • In client-side load balancing, the client somehow obtains a list of possible server it may use, and it implements the smart to decide how to distribute its own requests among the list of servers it has. Round-robin DNS is such an example, where the client receives a list of servers over DNS query. Very little additional infrastructure is needed other than configuration changes.
  • In server-side load balancing, the client reaches a server-side load balancer that distributes requests to backend servers. For what the client cares, the load balancer is simply one gigantic server. Some examples are IP Virtual Server and HTTP Reverse Proxy. They require heavier infrastructure investment.
In production, a mix of client and server side load balancing are often used. It's not hard to see why, if we consider the impact of load balancing at the client and at the server. Here we consider just latency and throughput.

Client-side load balancing tends to suffer worse latency than server-side load balancing. The reason is because the client often doesn't have visibility on server availability or load. If a server is overloaded or is offline, then the client may have to wait for a timeout before it tries another server. Propagating server load information to the client does not improve latency because the propagation delay only adds to the overall service time. Server availability is also highly dynamic, so a client cannot reuse this information for an extended period of time. On the other hand, a server-side load balancer is much closer to the backend servers, which means it could obtain server availability much faster. Also, the balancer can amortize the cost of querying server availability over many requests. Server-side load balancing, therefore, incurs no latency penalty to the client.

Client-side load balancing tends to enjoy greater throughput than server-side load balancing. That's because the network path for each client-server path could potentially take different routes. Of course, if a client tries to contact many servers, it would saturate its own uplink first, but if a server's uplink is saturated, we can easily add more servers and more uplinks. On the other hand, a server-side load balancer becomes a unique destination for many clients, so its uplink can be easily saturated. If it is the case that requests are smaller than the responses (which is often the case but not always), then Direct Routing can alleviate some of the load-balancer throughput bottleneck by having the backends return the response directly to the client, but it doesn't scale as easily as client-side load balancing.

The solution often used in production is to deploy both client and server side load balancing. A cluster in server-side load balancing makes a destination highly available, which guarantees the best service time for a given client. But client-side load balancing can send a client to multiple such clusters for better throughput.