Cross-compiling openldap on Debian Wheezy

Notes on problems cross-compiling openldap on Debian Wheezy and how I overcome some of the problems. I was trying to build smbkrb5pwd into openldap.

• Problem: setting up build prerequisites:
• Use emdebian gcc-4.4-arm-linux-gnueabi toolchain.
• Use xapt -a armel -m <builddep> and xdeb.
• Problem: configure couldn't find gnutls or some other dependencies.
• Solution: use xapt -a armel -m <package> to install them (-m is for multiarch, which only works on Wheezy). Then xdeb -aarmel openldap should build it from local source.
• Problem: libtool link complained about "liblutil.a: could not read symbols: Archive has no index; run ranlib to add one"
• Solution: change configure.in and replace AC_CHECK_PROGS with AC_CHECK_TOOLS.  Rerun autoreconf.
• Problem: unresolved external symbol lutil_memcmp ...
• Solution: the library order in several Makefile.in and build/top.mk are messed up. Reorder so that $(LDAP_LIBLUTIL_A) comes after $(LDAP_LIBLDAP_R_LA) or $(LDAP_LIBLDAP_LA).
• Problem: /usr/arm-linux-gnueabi/lib/libkrb5.so: No such file or directory
• Solution: the libkrb5.la specified the wrong directory ... dpkg-cross has it installed under /usr/arm-linux-gnueabi/lib/heimdal instead. Either symlink the .so or modify the .la.
• Problem: contrib/.../Makefile sets CC=gcc.
• Solution: in debian/rules, override in the command line like this: $(MAKE) -C ... CC=$(DEB_HOST_MULTIARCH)-gcc
• Problem: dpkg-cross on libkrb5-dev removes /usr/arm-linux-gnueabi/lib/mit-krb5/...
• I give up. There's gotta be a gooder way to do this...

It turns out that building smbkrb5pwd into openldap is a bad idea. It requires krb5 as a dependency, but krb5 requires libldap-dev provided by openldap, which means it a recursive dependency. Rather than putting smbkrb5pwd into the contrib package, the right way seems to be building it standalone.

Cap'n proto inspired encoding

I guess the fact that Kenton Varda, a principle author of Protocol Buffer and the guy who built the ultimate LAN party house, came up with Cap'n Proto specifically to address the problem of serialization and deserialization overhead, is worthy of an investigation. The idea is to have a wire format that can be dumped to some memory location and used directly without decoding. But the encoding of Cap'n Proto is too complex for me to understand, so I decided to come up with my own.

First I like the idea that each field within a struct is 64-bit word aligned. Large objects like nested structs, strings, and arrays are referenced by an offset. In order to support offsetting and chunking, the wire protocol divides data to segments. When you construct a struct, you allocate memory from the segment. This is where I think it got too complicated.

Rather, let me propose a simpler wire format. A struct consists of uniquely numbered fields. Each field is exactly 64-bit. Plain old integral data will be stored as is. They can be packed, e.g. four 16-bit integers or two 32-bit integers in a field. The 64-bit field size allows storing a pointer on a 64-bit machine. This pointer is a machine and implementation specific object pointer, and is meaningless when serialized. Here is where my proposal differs from Cap'n Proto.

Immediately after each struct are a series of "fix-ups" or "field population instructions" that are basically a sequence of (field number, data type, data length, data) tuples. The field number specifies the field of the struct to be populated. Data type specifies how to populate this field with data. Data length specifies the number of bytes, followed by the action data.

For example, to serialize a struct like this:

struct S {
  // x, y, u, v are packed into field 0.
  // packed subfields must be naturally aligned to
  // their respective size.
  0: int32 x;
     int16 y;
     uint8 u, v;

  1: list<string> xs;
};

Then we would first dump field 0 (which has x, y, u, v all packed together) and field 1 of S, even though field 1's value would be meaningless. However, for each string in xs, we'll output a fix-up that says "add this data as a string to field 1." The decoder of fix-up instructions can reuse the same read buffer to construct string pointer references, so the string content need not be copied.

When this wire format is decoded, the decoder allocates memory for S and dumps all the field values there. It would consult the definition of S and realizes that it has to initialize field 1 by allocating a list object to hold the strings. Then it would process through the fix-ups and add the strings as required.

This will work for nested structs as well. What I haven't quite figured out is dictionary. The encoding would not be very efficient if we encode the fix-up for the key, then the fix-up for the value. In fact, since all the keys are of the same type, and all the values are of the other type, it is more efficient to encode them as two parallel lists.

With this encoding scheme, I don't need to deal with that inter-segment pointer non-sense.

Earlier on I wrote about an improved binary JSON serialization format. I think there is still a use case for that. Basically if a programming language does not use raw struct layout in memory, then it won't be able to take advantage of the Cap'n Proto styled encoding, and having a compact wire format is probably better. Only C/C++ can really take advantage of Cap'n Proto styled encoding. Also, this raw encoding requires a schema in order to know how to interpret data. With binary JSON, the data is self-tagging and self-delimiting. You can manipulate data even if you don't have their schema.

Exponential back-off algorithm for IP packet fragmentation

On reading about Path MTU discovery, I found this article Never Ending Story of IP Fragmentation. It explains the problem with bad IP fragmentation when there are multiple encapsulations. Long story short, network links can only carry packets of a certain size called "maximum transmission unit" or MTU. Large packets are split into fragments to fit to the MTU. But MTU is reduced when tunneling IP traffic, e.g. GRE (used by VPN like PPTP and IPsec) and PPPoE. If we naively split a packet by next-hop MTU, then we might get this situation:

• Original packet has 1516 bytes to traverse over an Ethernet link with MTU of 1500. It is split into two packets, fragment 1 of 1500 bytes and fragment 2 of 36 bytes (the excess 16 bytes plus additional 20 bytes IP header).
• Fragment 1 traverses through a PPPoE tunnel with MTU of 1492 bytes. It is split into two more packets, fragment 1.1 of 1492 bytes and fragment 1.2 of 36 bytes (8 bytes excess + 28 bytes overhead). However, fragment 2 traverses unaffected but now becomes 44 bytes.
• Fragment 1.1 traverses through a IPsec tunnel with MTU of 1400 bytes. It is split into two more fragments, fragment 1.1.1 of 1400 bytes and fragment 1.1.2 of 152 bytes (the excess 92 bytes plus 60 bytes encapsulation overhead). The other fragment 1.2 of 36 bytes traverses unaffected but becomes 96 bytes after encapsulation; fragment 2 also traverses unaffected but also becomes 96 bytes.
• Fragment 1.1.1 traverses through another tunnel...

If there are N tunnel encapsulations, for each packet we might get N fragments. This is a blow-up of N factor in the total number of packets sent. This incurs cost both in packet switching and assembly.

The industry standard practice is "Path MTU Discovery" or PMTUD. It sends a series of "do not fragment" packets down the pipe until it gets rejected, and the hop that rejects it advertises the MTU. The originator then reduces MTU and tries again. The sole purpose is that the originator of the packet can figure out how much data to send in order to fit the smallest MTU of all the hops until it reaches the destination, at the cost of large connection startup time. Each MTU discovery incurs a partial round-trip cost, so the total round trip time is \( O(n^2) \) where n is the number of hops. This is especially costly for servers that deal with a large number of clients. The server would have to probe and remember each client's path MTU.

What I'm proposing is to get rid of PMTUD altogether but apply the principle of exponential back-off in fragmentation.

Every time a packet has to be fragmented, divide the packet into two more-or-less equally sized halves. Or keep halving the packet \(k\) times until the \( \frac1{2_k} \) sized fragment fits to the next link's MTU. The rational is that we're going to be splitting the packet anyway, and halving gives extra headroom for the next hops MTU which might be smaller.

In the worst case where each hop's MTU is exponentially decreasing, you might still get \(2^N\) times the number of fragments, but in practice MTU is linearly decreasing. The reason for the linear decrease is due to the extra encapsulation header being stacked on linearly.

Revisiting the above scenario, the original packets would be divided into two of 758 bytes, both traverse through all tunnel layers without further fragmentation, and without knowing the path MTU. Alternatively, after a costly PMTUD, the host might divide the original packets to two of 1400 and 116 bytes respectively, achieving the same effect without PMTUD but with the exponential back-off fragmentation.

Pogoplug V4 spec for future reference

NAND partitions:

/ # cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00200000 00020000 "u-boot"
mtd1: 00300000 00020000 "uImage"
mtd2: 00300000 00020000 "uImage2"
mtd3: 00800000 00020000 "failsafe"
mtd4: 07000000 00020000 "root"

PCI and USB devices:

/ # lspci
00:01.0 Class 0c03: 1b73:1009
/ # lsusb
Bus 001 Device 001: ID 1d6b:0002
Bus 002 Device 001: ID 1d6b:0002
Bus 001 Device 002: ID 0781:5571

CPU info:

/ # cat /proc/cpuinfo 
Processor       : Feroceon 88FR131 rev 1 (v5l)
BogoMIPS        : 799.53
Features        : swp half thumb fastmult edsp 
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant     : 0x2
CPU part        : 0x131
CPU revision    : 1

Hardware        : Feroceon-KW
Revision        : 0000
Serial          : 0000000000000000

dmesg output:

~ # dmesg
console=ttyS0,115200 root=ubi0:rootfs ubi.mtd=4,2048 rootfstype=ubifs
<4>[    0.000000] PID hash table entries: 512 (order: 9, 2048 bytes)
<6>[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
<6>[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
<6>[    0.000000] Memory: 128MB = 128MB total
<5>[    0.000000] Memory: 118356KB available (3852K code, 261K data, 124K init, 0K highmem)
<6>[    0.000000] Hierarchical RCU implementation.
<6>[    0.000000] NR_IRQS:128
<4>[    0.000000] Console: colour dummy device 80x30
<6>[    0.000000] Calibrating delay loop... 799.53 BogoMIPS (lpj=3997696)
<4>[    0.230000] Mount-cache hash table entries: 512
<6>[    0.230000] CPU: Testing write buffer coherency: ok
<6>[    0.230000] NET: Registered protocol family 16
<6>[    0.230000] Feroceon L2: Enabling L2
<6>[    0.230000] Feroceon L2: Cache support initialised.
<4>[    0.230000] 
<4>[    0.230000] CPU Interface
<4>[    0.230000] -------------
<4>[    0.230000] SDRAM_CS0 ....base 00000000, size 128MB 
<4>[    0.230000] SDRAM_CS1 ....disable
<4>[    0.230000] SDRAM_CS2 ....disable
<4>[    0.230000] SDRAM_CS3 ....disable
<4>[    0.230000] PEX0_MEM ....base e0000000, size 128MB 
<4>[    0.230000] PEX0_IO ....base f2000000, size   1MB 
<4>[    0.230000] PEX1_MEM ....no such
<4>[    0.230000] PEX1_IO ....no such
<4>[    0.230000] INTER_REGS ....base f1000000, size   1MB 
<4>[    0.230000] NFLASH_CS ....base fa000000, size   2MB 
<4>[    0.230000] SPI_CS ....base f4000000, size  16MB 
<4>[    0.230000] BOOT_ROM_CS ....no such
<4>[    0.230000] DEV_BOOTCS ....no such
<4>[    0.230000] CRYPT_ENG ....base f0000000, size   2MB 
<4>[    0.230000] 
<4>[    0.230000]   Marvell Development Board (LSP Version KW_LSP_5.1.3_patch18)-- RD-88F6192A-NAS  Soc: 88F6192 A1 LE
<4>[    0.230000] 
<4>[    0.230000]  Detected Tclk 166666667 and SysClk 200000000 
<4>[    0.230000] Marvell USB EHCI Host controller #0: c403e740
<4>[    0.730000] PEX0 interface detected Link X1
<7>[    0.730000] pci 0000:00:01.0: reg 10 64bit mmio: [0x40000000-0x4000ffff]
<7>[    0.730000] pci 0000:00:01.0: reg 18 64bit mmio: [0x40010000-0x40010fff]
<7>[    0.730000] pci 0000:00:01.0: reg 20 64bit mmio: [0x40011000-0x40011fff]
<7>[    0.740000] pci 0000:00:01.0: supports D1
<6>[    0.740000] pci 0000:00:01.0: PME# supported from D0 D1 D3hot
<6>[    0.740000] pci 0000:00:01.0: PME# disabled
<6>[    0.740000] PCI: bus0: Fast back to back transfers disabled
<4>[    0.740000] mvPexLocalBusNumSet: ERR. Invalid PEX interface 1
<4>[    0.750000] bio: create slab <bio-0> at 0
<5>[    0.750000] SCSI subsystem initialized
<6>[    0.750000] usbcore: registered new interface driver usbfs
<6>[    0.750000] usbcore: registered new interface driver hub
<6>[    0.750000] usbcore: registered new device driver usb
<6>[    0.750000] NET: Registered protocol family 2
<6>[    0.750000] IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
<6>[    0.750000] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
<6>[    0.750000] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
<6>[    0.750000] TCP: Hash tables configured (established 4096 bind 4096)
<6>[    0.750000] TCP reno registered
<6>[    0.750000] NET: Registered protocol family 1
<6>[    0.750000] cpufreq: Init kirkwood cpufreq driver
<7>[    0.750000] cpufreq: High frequency: 800000KHz - Low frequency: 200000KHz
<7>[    0.750000] cpufreq: Setting CPU Frequency to 800000 KHz
<7>[    0.750000] cpufreq: Setting PowerSaveState to off
<6>[    0.760000] XOR registered 4 channels
<6>[    0.760000] XOR 2nd invalidate WA enabled
<4>[    0.760000] cesadev_init(c000d7fc)
<4>[    0.760000] mvCesaInit: sessions=640, queue=64, pSram=f0000000
<6>[    0.760000] squashfs: version 4.0 (2009/01/31) Phillip Lougher
<6>[    0.770000] msgmni has been set to 231
<6>[    0.770000] alg: No test for cipher_null (cipher_null-generic)
<6>[    0.770000] alg: No test for ecb(cipher_null) (ecb-cipher_null)
<6>[    0.770000] alg: No test for digest_null (digest_null-generic)
<6>[    0.770000] alg: No test for compress_null (compress_null-generic)
<6>[    0.780000] alg: No test for stdrng (krng)
<6>[    0.780000] alg: No test for hmac(digest_null) (hmac(digest_null-generic))
<6>[    0.790000] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
<6>[    0.790000] io scheduler noop registered
<6>[    0.790000] io scheduler anticipatory registered (default)
<4>[    0.790000] Initializing ths8200_init
<4>[    0.790000] Initializing dove_adi9889_init
<6>[    0.810000] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
<6>[    0.810000] serial8250.0: ttyS0 at MMIO 0xf1012000 (irq = 33) is a 16550A
<6>[    0.810000] console [ttyS0] enabled
<4>[    0.820000] Integrated Sata device found
<4>[    0.830000] IRQ 21/mvSata: IRQF_DISABLED is not guaranteed on shared IRQs
<6>[    0.850000] scsi0 : Marvell SCSI to SATA adapter
<6>[    0.860000] scsi1 : Marvell SCSI to SATA adapter
<4>[    0.870000] Loading Marvell Ethernet Driver:
<4>[    0.870000]   o Cached descriptors in DRAM
<4>[    0.880000]   o DRAM SW cache-coherency
<4>[    0.880000]   o 1 Giga ports supported
<4>[    0.880000]   o Single RX Queue support - ETH_DEF_RXQ=0
<4>[    0.890000]   o Single TX Queue support - ETH_DEF_TXQ=0
<4>[    0.890000]   o TCP segmentation offload (TSO) supported
<4>[    0.900000]   o Large Receive offload (LRO) supported
<4>[    0.900000]   o Receive checksum offload supported
<4>[    0.910000]   o Transmit checksum offload supported
<4>[    0.910000]   o Network Fast Processing (Routing) supported - (Disabled)
<4>[    0.920000]   o Driver ERROR statistics enabled
<4>[    0.930000]   o Proc tool API enabled
<4>[    0.930000]   o SKB Reuse supported - (Disabled)
<4>[    0.930000]   o SKB Recycle supported - (Disabled)
<4>[    0.940000]   o Rx descripors: q0=128
<4>[    0.940000]   o Tx descripors: q0=532
<4>[    0.950000]   o Loading network interface(s):
<4>[    0.950000]      o register under mv88fx_eth platform
<4>[    0.960000]      o eth0, ifindex = 2, GbE port = 0
<4>[    0.960000] 
<4>[    0.960000] mvFpRuleDb (c45b2000): 1024 entries, 4096 bytes
<4>[    0.970000] Counter=0, opIdx=6, overhead=16
<4>[    0.970000] Counter=1, opIdx=2, overhead=0
<4>[    0.980000] Counter=2, opIdx=1, overhead=18
<4>[    0.980000] Counter=3, opIdx=2, overhead=0
<6>[    0.990000] tun: Universal TUN/TAP device driver, 1.6
<6>[    0.990000] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
<6>[    1.000000] NAND device: Manufacturer ID: 0xad, Chip ID: 0xf1 (Hynix NAND 128MiB 3,3V 8-bit)
<6>[    1.010000] Scanning device for bad blocks
<4>[    1.060000] Using static partition definition
<5>[    1.060000] Creating 5 MTD partitions on "nand_mtd":
<5>[    1.070000] 0x000000000000-0x000000200000 : "u-boot"
<5>[    1.070000] 0x000000200000-0x000000500000 : "uImage"
<5>[    1.080000] 0x000000500000-0x000000800000 : "uImage2"
<5>[    1.080000] 0x000000800000-0x000001000000 : "failsafe"
<5>[    1.090000] 0x000001000000-0x000008000000 : "root"
<5>[    1.100000] UBI: attaching mtd4 to ubi0
<5>[    1.100000] UBI: physical eraseblock size:   131072 bytes (128 KiB)
<5>[    1.110000] UBI: logical eraseblock size:    126976 bytes
<5>[    1.110000] UBI: smallest flash I/O unit:    2048
<5>[    1.120000] UBI: sub-page size:              512
<5>[    1.120000] UBI: VID header offset:          2048 (aligned 2048)
<5>[    1.130000] UBI: data offset:                4096
<5>[    1.350000] UBI: attached mtd4 to ubi0
<5>[    1.350000] UBI: MTD device name:            "root"
<5>[    1.360000] UBI: MTD device size:            112 MiB
<5>[    1.360000] UBI: number of good PEBs:        896
<5>[    1.370000] UBI: number of bad PEBs:         0
<5>[    1.370000] UBI: max. allowed volumes:       128
<5>[    1.380000] UBI: wear-leveling threshold:    4096
<5>[    1.380000] UBI: number of internal volumes: 1
<5>[    1.390000] UBI: number of user volumes:     1
<5>[    1.390000] UBI: available PEBs:             0
<5>[    1.400000] UBI: total number of reserved PEBs: 896
<5>[    1.400000] UBI: number of PEBs reserved for bad PEB handling: 8
<5>[    1.410000] UBI: max/mean erase counter: 1/0
<5>[    1.410000] UBI: image sequence number: 0
<6>[    1.420000] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
<6>[    1.420000] ehci_marvell ehci_marvell.70059: Marvell Orion EHCI
<6>[    1.430000] ehci_marvell ehci_marvell.70059: new USB bus registered, assigned bus number 1
<5>[    1.440000] UBI: background thread "ubi_bgt0d" started, PID 452
<6>[    1.470000] ehci_marvell ehci_marvell.70059: irq 19, io base 0xf1050100
<6>[    1.490000] ehci_marvell ehci_marvell.70059: USB 2.0 started, EHCI 1.00
<6>[    1.490000] usb usb1: configuration #1 chosen from 1 choice
<6>[    1.500000] hub 1-0:1.0: USB hub found
<6>[    1.500000] hub 1-0:1.0: 1 port detected
<6>[    1.510000] xhci_hcd 0000:00:01.0: xHCI Host Controller
<6>[    1.510000] xhci_hcd 0000:00:01.0: new USB bus registered, assigned bus number 2
<6>[    1.520000] xhci_hcd 0000:00:01.0: irq 9, io mem 0xe0000000
<4>[    1.530000] usb usb2: config 1 interface 0 altsetting 0 endpoint 0x81 has no SuperSpeed companion descriptor
<6>[    1.540000] usb usb2: configuration #1 chosen from 1 choice
<7>[    1.540000] xHCI xhci_add_endpoint called for root hub
<7>[    1.540000] xHCI xhci_check_bandwidth called for root hub
<6>[    1.540000] hub 2-0:1.0: USB hub found
<6>[    1.550000] hub 2-0:1.0: 4 ports detected
<6>[    1.550000] Initializing USB Mass Storage driver...
<6>[    1.550000] usbcore: registered new interface driver usb-storage
<6>[    1.560000] USB Mass Storage support registered.
<6>[    1.570000] usbcore: registered new interface driver ums-datafab
<6>[    1.570000] usbcore: registered new interface driver ums-freecom
<6>[    1.580000] usbcore: registered new interface driver ums-jumpshot
<6>[    1.580000] usbcore: registered new interface driver ums-sddr09
<6>[    1.590000] usbcore: registered new interface driver ums-sddr55
<6>[    1.600000] usbcore: registered new interface driver ums-usbat
<6>[    1.600000] mice: PS/2 mouse device common for all mice
<6>[    1.610000] i2c /dev entries driver
<7>[    1.610000] cpufreq: Setting CPU Frequency to 800000 KHz
<7>[    1.610000] cpufreq: Setting PowerSaveState to off
<6>[    1.620000] sdhci: Secure Digital Host Controller Interface driver
<6>[    1.620000] sdhci: Copyright(c) Pierre Ossman
<5>[    1.630000] mmc0: mvsdio driver initialized, using GPIO 27 for card detection
<6>[    1.640000] usbcore: registered new interface driver usbhid
<6>[    1.640000] usbhid: v2.6:USB HID core driver
<6>[    1.650000] TCP cubic registered
<6>[    1.650000] NET: Registered protocol family 17
<6>[    1.660000] RPC: Registered udp transport module.
<6>[    1.660000] RPC: Registered tcp transport module.
<4>[    1.670000] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
<5>[    1.740000] UBIFS: recovery needed
<5>[    1.820000] UBIFS: recovery completed
<5>[    1.830000] UBIFS: mounted UBI device 0, volume 0, name "rootfs"
<5>[    1.830000] UBIFS: file system size:   110850048 bytes (108252 KiB, 105 MiB, 873 LEBs)
<5>[    1.840000] UBIFS: journal size:       9023488 bytes (8812 KiB, 8 MiB, 72 LEBs)
<5>[    1.850000] UBIFS: media format:       w4/r0 (latest is w4/r0)
<5>[    1.850000] UBIFS: default compressor: lzo
<5>[    1.860000] UBIFS: reserved for root:  0 bytes (0 KiB)
<4>[    1.860000] VFS: Mounted root (ubifs filesystem) on device 0:11.
<6>[    1.870000] Freeing init memory: 124K
<6>[    1.880000] usb 1-1: new high speed USB device using ehci_marvell and address 2
<6>[    2.050000] usb 1-1: configuration #1 chosen from 1 choice
<6>[    2.060000] scsi2 : SCSI emulation for USB Mass Storage devices
<7>[    2.070000] usb-storage: device found at 2
<7>[    2.070000] usb-storage: waiting for device to settle before scanning
<5>[    4.290000] eth0: link down
<5>[    4.290000] eth0: started
<5>[    5.840000] eth0: link up, full duplex, speed 1 Gbps
<5>[    7.070000] scsi 2:0:0:0: Direct-Access     SanDisk  Cruzer Fit       1.26 PQ: 0 ANSI: 5
<5>[    7.090000] sd 2:0:0:0: [sda] 31266816 512-byte logical blocks: (16.0 GB/14.9 GiB)
<5>[    7.100000] sd 2:0:0:0: [sda] Write Protect is off
<7>[    7.100000] sd 2:0:0:0: [sda] Mode Sense: 43 00 00 00
<3>[    7.100000] sd 2:0:0:0: [sda] Assuming drive cache: write through
<3>[    7.110000] sd 2:0:0:0: [sda] Assuming drive cache: write through
<6>[    7.120000]  sda: sda1
<5>[    7.120000] sd 2:0:0:0: Attached scsi generic sg0 type 0
<7>[    7.140000] usb-storage: device scan complete
<3>[    7.150000] sd 2:0:0:0: [sda] Assuming drive cache: write through
<5>[    7.160000] sd 2:0:0:0: [sda] Attached SCSI removable disk
<4>[    7.860000] IRQ 93 uses trigger mode 0; requested 3
<6>[    7.870000] input: gpio-keys as /devices/platform/gpio-keys.0/input/input0
<6>[    7.880000] xce_ebtn: Pogoplug series V4 eject button initialized.
<4>[    8.030000] ufsd: module license 'Commercial product' taints kernel.
<4>[    8.030000] Disabling lock debugging due to kernel taint
<5>[    8.070000] ufsd: driver (8.6 (U86_S[2012-02-28-18:39:23]), LBD=ON, delalloc, ioctl) loaded at bf00c000
<5>[    8.070000] NTFS support included
<5>[    8.070000] Hfs+/HfsX support included
<5>[    8.070000] For 'CloudEngines_PogoPlug_2011-08-03'
<4>[    8.400000] rtusb init rt2870 --->
<6>[    8.410000] usbcore: registered new interface driver rt2870
<4>[    8.440000] Cloud Engines XCE Init [Version: 3.9.0.4]
<6>[    8.440000] XCE: CPU MEMORY MAP:
<6>[    8.440000] XCE:   -- 0x00001000 - 0xbeffffff (3055 MB)  User Space Mappings
<6>[    8.450000] XCE:   -- 0xbf000000 - 0xbfffffff (  16 MB)  Kernel module space
<6>[    8.460000] XCE:   -- 0xc0000000 - 0xc7ffffff ( 128 MB)  Kernel direct-mapped ram
<6>[    8.470000] XCE:   -- 0xc8800000 - 0xe7ffffff ( 504 MB)  Kernel vmalloc space
<6>[    8.470000] XCE:   -- 0xe8000000 - 0xfeffffff ( 367 MB)  Kernel platform space
<6>[    8.480000] XCE: CPU FEATURES:
<6>[    8.480000] XCE:   -- I Cache:         enabled
<6>[    8.490000] XCE:   -- D Cache:         enabled
<6>[    8.490000] XCE:   -- Branch Predict:  disabled
<6>[    8.500000] XCE:   -- MMU:             enabled
<6>[    8.500000] XCE:   -- Alignment Abort: enabled
<6>[    8.510000] XCE: BLPARAMS:   -- Loading properties [c4d49efc].
<6>[    8.520000] XCE: BLPARAMS:   -- MTD @ [c45c1c00].
<6>[    8.520000] XCE: BLPARAMS: Locating parameter block...
<6>[    8.530000] XCE: BLPARAMS: reading 2048 bytes @ a0000
<6>[    8.530000] XCE: Loaded Property Size: 2048
<6>[    8.540000] XCE:    - 'cesvcid' -> '9C5FE2YZ96Z72G5C8AWKUNJHWW'
<6>[    8.540000] XCE:    - 'ceboardver' -> 'PPV4A3'
<6>[    8.550000] XCE:   -- ICache Prefetch: enabled
<6>[    8.550000] XCE:   -- DCache Prefetch: enabled
<6>[    8.560000] XCE:   -- L2 Cache:        enabled
<6>[    8.560000] XCE:   -- L2 Prefetch:     disabled
<6>[    8.570000] XCE:   -- L2 Writethrough: disabled
<6>[    8.570000] XCE:   -- Write Allocate:  disabled
<6>[    8.580000] XCE:   -- Streaming:       disabled
<6>[    8.580000] XCE: Current GPIO State:
<6>[    8.580000] XCE:  GPIO L OUT:    0x01f18400
<6>[    8.590000] XCE:  GPIO L OE:     0xfe004800
<6>[    8.590000] XCE:  GPIO L BLINK:  0x00000000
<6>[    8.600000] XCE:  GPIO L POL:    0x28000000
<6>[    8.600000] XCE:  GPIO L IN:     0x11f00000
<6>[    8.600000] XCE:  GPIO H OUT:    0x00000008
<6>[    8.610000] XCE:  GPIO H OE:     0x00000005
<6>[    8.610000] XCE:  GPIO H BLINK:  0x00000000
<6>[    8.620000] XCE:  GPIO H POL:    0x00000000
<6>[    8.620000] XCE:  GPIO H IN:     0x00000008
<6>[    8.720000] XCE: BLPARAMS:   -- Loading properties [c4babecc].
<6>[    8.720000] XCE: BLPARAMS:   -- MTD @ [c45c1c00].
<6>[    8.730000] XCE: BLPARAMS: Locating parameter block...
<6>[    8.730000] XCE: BLPARAMS: reading 2048 bytes @ a0000
<6>[    8.740000] XCE: BLPARAMS: reading 2048 bytes @ a0800
<6>[    8.750000] XCE: BLPARAMS: reading 2048 bytes @ a1000
<6>[    8.750000] XCE: BLPARAMS: reading 2048 bytes @ a1800
<6>[   14.370000] XCE: XCE: LED -> CONNECTED
<6>[   14.400000] kjournald starting.  Commit interval 5 seconds
<6>[   14.400000] EXT3 FS on sda1, internal journal
<6>[   14.410000] EXT3-fs: mounted filesystem with writeback data mode.
<6>[   14.730000] usb 1-1: reset high speed USB device using ehci_marvell and address 2

Installing Debian on Pogoplug Series V4

Here I'm basing the tools and instruction on Arch Linux. I'm using a PogoPlug Series V4 with a SamDisk Cruzer Fit 16GB flash drive which fits snuggly on the top USB 2.0 port, even with the lid closed.

I also followed the instructions by Moustafa Hassan to identify and solder the serial console pins, but chose a different connection method. I bought female-female jumper wires, extra long breakaway headers to couple the jumper wires, and USB to TTL serial cable which works out of box on Linux. Although it is possible to ssh into PogoPlug V4, having serial console access is a safety net in case I brick the Pogo.

• Select black, green, and white wires. This will be used to make the external connection.
• Pick a bundle of three other wires. I chose grey, purple and blue because that happen to come off next. Do not disband them. Cut these in two halves together, strip one half and save the other half for future use. Solder grey, purple, blue to GND, RX, TX respectively. You might want to glue them to the board since the solder joints come off easily.
• Unfortunately you can't solder breakaway header to the board. The holes are not 1mm spaced, and the holes are too small.
• Use the extra long breakaway to connect grey to black, purple to green, and blue to white. This connection will be inside the Pogo.
• Carefully extend the black (GND), green (RX), white (TX) wires through the vent of the top cover, cap them with another set of breakaway header. These colors will be the same as the USB-TTL serial cable. Red is not used.

Hint: on Debian, adding your non-root user to the "dialout" group (logout, log back in) will allow you use the command "screen /dev/ttyUSB0 115200" without sudo. To find out exactly which group you need to add to on your distro, ls -l /dev/ttyUSB0 and look at the group of the file.

Installing Debian is significantly more work than Arch Linux, and generally requires another Linux machine to do the first stage bootstrapping.

I was going to use Multistrap to prepare a root filesystem but I changed that plan. Since I already have serial console and can TFTP boot anything I want, I can now TFTP boot and use the interactive Debian Installer. After interactively installing, the plan is to manually configure the bootloader using Arch Linux's U-Boot which has USB support.

To boot the Debian Installer, I need to do this one time setup at the U-Boot prompt to make mainline kernel boot on PogoPlug which is an unrecognized device. I found this information at the Plug Computer Forum.

setenv mainlineLinux yes
setenv arcNumber 2097
saveenv
reset

Correction: When using arcNumber 2097 (SheevaPlug), USB3 is not working because PCIe is not initialized, but GigE is working. When using arcNumber 1681 (RD-88F6192-NAS), USB3 works but not GigE. For proper Pogoplug V4 support, one has to build a Debian kernel with the archlinuxarm.patch.

Unfortunately, setting mainlineLinux=yes will prevent the Pogo's built-in Linux kernel from booting, so you'll have to set mainlineLinux=no if you want it back.

And then, following Debian installer for plug computer instructions, after downloading the SheevaPlug installer files to TFTP as debian/uImage and debian/uInitrd, I could boot d-i over TFTP like this:

setenv bootargs console=ttyS0,115200n8 base-installer/initramfs-tools/driver-policy=most
tftp 0x00800000 debian/uImage
tftp 0x01100000 debian/uInitrd
bootm 0x00800000 0x01100000

I installed Debian. The installer finished without requring swap space which is impressive considering that there is only 128MB of main memory. However, to get Debian to boot, I had to boot into PogoPlug Linux first and run ppv4-install.sh. First boot into U-boot:

setenv mainlinuxLinux no
saveenv
reset

Boot into PogoPlug Linux, download ppv4-install.sh and run it. But I need to make more adjustments before Debian can boot. Boot into U-Boot again and run:

setenv mainlinuxLinux yes
setenv alarm_usb 'ext2load usb 0:1 0x800000 /boot/uImage; ext2load usb 0:1 0x1100000 /boot/uInitrd; run alarm_which; run alarm_args; bootm 0x800000 0x1100000'
saveenv
reset

The reason is that ArchLinux boot script only loads the kernel but not the initramfs. If you boot from SATA/IDE, then make the analogous change to alarm_ide also. Debian should now boot normally.

Testing U-Boot over TFTP

Before I flash my PogoPlug v4 with a custom compiled u-boot, I thought it is prudent to be able to test it and make sure it works before I permanently brick the device. Although according to this circuit board breakdown, there are suspected I/O connection to a 5-pin J15 to the Hynix NAND flash chip, it is not clear if I'd be able to flash a bricked device. Better not take the risk.

First setup a tftpd-hpa server. This is just as easy as apt-get install tftpd-hpa and then copy the file to the serving directory, which is /srv/tftp by default. Make sure everyone can read the file copied there. You don't need to configure DHCP to hand out the tftp server's IP address just for testing.

This is what I did to compile my U-Boot. This is based on notes from Always Innovating but using a different configuration.

• Install emdebian cross compilation toolchain which installs /usr/bin/arm-linux-gnueabi-{gcc,as,ld,...}.
• git clone git://git.denx.de/u-boot.git
• cd u-boot && git checkout v2013.04  # use a later release if appropriate.
• To enable ext4, you might want to apply this patch.
• make CROSS_COMPILE=arm-linux-gnueabi- sheevaplug_config
• make CROSS_COMPILE=arm-linux-gnueabi- all

This creates a u-boot.bin in the u-boot source directory. Before I continue, I just want to say that sheevaplug_config doesn't work on pogoplug v4. It's worth finding out. But before that let's do a comparison with what works: the Pogoplug V4 opensourced U-Boot at 1.1.4.

tar jxf pogomv-u-boot.tar.bz2
cd pogomv-u-boot
make CROSS_COMPILE=arm-linux-gnueabi- pogoplug2_config
make CROSS_COMPILE=arm-linux-gnueabi- all
cp u-boot-pogoplug2.bin /srv/tftp

Over serial console, power on the device and hit enter as soon as you see:

U-Boot 1.1.4 (Oct  1 2011 - 12:06:06) Cloud Engines 1.1.2 (3.4.27) PHYADDR=0

U-Boot code: 00600000 -> 0067FFF0  BSS: -> 006918B4
Soc: 88F6192 A1 (DDR2)
CPU running @ 800Mhz L2 running @ 400Mhz
SysClock = 200Mhz , TClock = 166Mhz
DRAM CAS Latency = 3 tRP = 3 tRAS = 8 tRCD=3
DRAM CS[0] base 0x00000000   size 128MB
DRAM Total size 128MB  16bit width
Addresses 8M - 0M are saved for the U-Boot usage.
Mem malloc Initialization (8M - 7M): Done
NAND:128 MB
Flash:  0 kB
CPU : Marvell Feroceon (Rev 1)
CLOUD ENGINES BOARD: PPV4A3
Streaming disabled
Write allocate disabled
USB 0: host mode
PEX 0: PCI Express Root Complex Interface
PEX interface detected Link X1
Net:   egiga0 [PRIME]
Hit any key to stop autoboot:  0 Do it now!

This enters the device's factory default U-Boot prompt (not the one you just compiled).

Let's get the device an IP address. You'll need to hit Ctrl-C to abort loading because my DHCP doesn't give out the correct TFTP server's address. It's using 10.0.5.1 (gateway) as the TFTP but my TFTP is at 10.0.5.33.

CE>> dhcp
BOOTP broadcast 1
*** Unhandled DHCP Option in OFFER/ACK: 28
*** Unhandled DHCP Option in OFFER/ACK: 28
DHCP client bound to address 10.0.5.206
*** Warning: no boot file name; using '0A0005CE.img'
Using egiga0 device
TFTP from server 10.0.5.1; our IP address is 10.0.5.206
Filename '0A0005CE.img'.
Load address: 0x2000000
Loading: * # hit Ctrl-C here
Abort

At this point your device will forget its IP address, but you can set it back using setenv. Essentially we're using the DHCP command just to get an IP address. Here I'm also setting the correct TFTP server's IP address I want to be using.

CE>> setenv ipaddr 10.0.5.206
CE>> setenv serverip 10.0.5.33

And now we can TFTP. We need to specify the filename and the memory location it will be written to once downloaded. The convention is to always chainload to 0x800000. The chainloaded U-Boot will relocate itself back to 0x67FFF0 which is the default CONFIG_SYS_TEXT_BASE.

CE>> tftp 0x800000 u-boot-pogoplug2.bin
Using egiga0 device
TFTP from server 10.0.5.33; our IP address is 10.0.5.206
Filename 'u-boot-pogoplug2.bin'.
Load address: 0x800000
Loading: #################################################################
         ############################
done 
Bytes transferred = 474148 (73c24 hex)

When we start executing the chainloaded U-Boot, it will appear as if the device has reset itself, but it's really our newly compiled binary (see the U-Boot compilation date).

CE>> go 0x800000
## Starting application at 0x00800000 ...


U-Boot 1.1.4 (May 11 2013 - 18:53:25) Cloud Engines 1.1.2 (3.4.27) PHYADDR=0

U-Boot code: 00600000 -> 0067FFF0  BSS: -> 006918AC

Soc: 88F6192 A1 (DDR2)
CPU running @ 800Mhz L2 running @ 400Mhz
SysClock = 200Mhz , TClock = 166Mhz 

DRAM CAS Latency = 3 tRP = 3 tRAS = 8 tRCD=3
DRAM CS[0] base 0x00000000   size 128MB
DRAM Total size 128MB  16bit width
Addresses 8M - 0M are saved for the U-Boot usage.
Mem malloc Initialization (8M - 7M): Done
NAND:128 MB
Flash:  0 kB

CPU : Marvell Feroceon (Rev 1)
CLOUD ENGINES BOARD: PPV4A3

Streaming disabled
Write allocate disabled


USB 0: host mode
PEX 0: PCI Express Root Complex Interface
PEX interface detected Link X1
Net:   egiga0 [PRIME]
Hit any key to stop autoboot:  0 do it now!

Don't forget to hit any key to stop at this point! And then after setenv serverip and ipaddr again. We can now tftp load the u-boot.bin we compiled for sheevaplug_config.

CE>> setenv serverip 10.0.5.33
CE>> setenv ipaddr 10.0.5.206
CE>> tftp 0x800000 sheevaplug.bin
Using egiga0 device
TFTP from server 10.0.5.33; our IP address is 10.0.5.206
Filename 'sheevaplug.bin'.
Load address: 0x800000
Loading: #################################################################
         #############
done
Bytes transferred = 397856 (61220 hex)
CE>> go 0x800000
## Starting application at 0x00800000 ...
data abort
pc : [<008000e8>]          lr : [<00800064>]
sp : 005ffe10  ip : 006554f9     fp : 00000000
r10: 00000000  r9 : 00673674     r8 : 005fffcc
r7 : 00000000  r6 : 00800000     r5 : 00721c24  r4 : 00000002
r3 : f1012000  r2 : 00000020     r1 : e1a03803  r0 : 00625131
Flags: nzcv  IRQs off  FIQs off  Mode SVC_32
Resetting CPU ...

This proves that Pogoplug V4 doesn't like sheevaplug's configuration. I've tested Davy Gravy's patch for pogo_v4_config but couldn't get it to work. I did manage to get the u-boot from ArchLinux's PogoPlug v4 u-boot to work.

Upon closer inspection, I noticed a few things about ArchLinux's U-boot.

• It's probably based on Pogoplug V4's u-boot at 1.1.4, except with USB enabled somehow.
• The ppv4-install.sh script writes ArchLinux u-boot to /dev/mtd0 for 4 blocks at 1MB offset, avoid overwriting the default u-boot.
• The boot script distinguishes factory u-boot from ArchLinux by probing for USB support (this is what "if usb start" is for). The default u-boot has no USB.

I also looked at porting PogoPlug v4 U-boot to a more recent version of U-boot but I gave up. The diff from stock U-boot 1.1.4 to PogoPlug v4 U-boot 1.1.4 reveals that it added a bunch of general purpose Marvell device support. It even included and compiled several files from the Linux kernel such as mv_cesa.c which has no business in U-boot. It's pretty messy.

I also tried the Marvell custodian repo of U-boot without luck. I think ArchLinux's approach is probably as good as it gets ... take the PogoPlug v4 U-boot 1.1.4 and enable USB.