Extended Datagram Protocol

I need a datagram protocol as a basis for transporting RPC over a network. I'd like to call it Extended Datagram Protocol or XDP.

Here are the requirements:

• Message size will be larger than MTU up to ~128KB. Most networks support Ethernet MTU of 1500 bytes, which means large packets need to be fragmented. IP fragmentation handles packet loss poorly. If a single fragment is lost, the whole packet will be dropped.
• The fragments are sent over UDP. XDP will handle packet assembly with forward error correction. Each fragment will have at most 1280 bytes of payload. It is a nice even number and leaves handsome room for the headers.
• The RPC layer will allow messages to arrive out of order, so the datagram protocol does not need to worry about packet ordering.
• The datagram protocol does not need to implement message acknowledgement. It may be handled by the RPC layer, or even the application layer.
• Encryption will be done at the RPC layer.
• Session management will be done at the RPC layer.

I don't have a concrete design yet, but I think it is worth mentioning the rationale behind these requirements. There are a number of proposals that extend UDP such as Reliable User Datagram Protocol and Stream Control Transmission Protocol that don't fit the bill. In-order delivery of packets actually becomes a head-of-line blocker, such is what QUIC tries to avoid.

QUIC (Quick UDP Internet Connections) features packet level error correction, but it is trying to do a lot more such as flow control and encryption. I think flow control should be handled by a layer above (e.g. RPC layer) which can do flow control not only based on network condition but also server load. I don't mind that it handles encryption. However, crypto negotiation packets can easily be larger than the MTU, so I think it would be easier to implement encryption with proper large message support provided by a simpler datagram protocol. I'm not trying to argue that QUIC should be designed differently, only that I think my requirements are different and not well-addressed.

An important design philosophy I have is that we need to share protocol implementation as much as possible, perhaps even across layers. TLS basically invented its own RPC protocol, and it uses X.509 which uses the ASN.1 message serialization scheme that differs from TLS. If I were to layer my own RPC on top of TLS there would be three protocol implementations.

The reason Extended Datagram Protocol does not handle anything besides message fragmentation is that everything else could really be handled by the RPC layer that has a strongly-typed and unified message serialization/deserialization scheme. The only difficulty that message serialization has trouble dealing with is the framing. If we were to send a message over a stream based protocol like TCP, we would first send the message length, then the message body. The sender could do this in one socket write, but the receiver cannot do this in a single read. The receiver would have to first read the message length, then the message itself. The receiver could speculatively read extra bytes in one socket read, but it would need a way to pass the excess data to the next receiver, often complicating the design of the protocol stack.

Let Organizations be Smart

Why is my smart home so fucking dumb?

There are generally two philosophies to design a complex system. One is to centralize all control to a brain that aggregates data from all sensors and decides what to do with its agents. Another is to distribute the intelligence to its agents.

A smart home is an example of a complex system. Let's say one of the design goal is to have motion activated lights that turn on with moving objects in the room and turn off after some inactivity. A centralized design would send motion data triggers to the brain; the brain would figure out which light to turn on, and send the command to turn on the light. The brain would keep a timer and turn off the light as needed. A distributed design would build the motion sensor and the timer right into the light switch in the same room that is hard wired to the light bulb.

One might argue that the distributed design is inflexible. What if you want the motion sensor to do something else other than turning on the light bulb in the same room? But the problem with centralized design is that it is a single point of failure. Imagine when you are upgrading the software in the brain, or if you somehow bricked the brain, then you can't control any light in any room in the home. How inconvenient would that be?

If you take inspiration from a living organism, the responsibility of control is split depending on the function. A creature is operated by the brain for objective, strategy, and coordination (e.g. hungry, find food, eat), but functions like reflexes that need critical response time and heart beats that need critical reliability are regulated locally. Not all actions need involvement from the brain, only the actions that need to be coordinated.

A home that needs to be smart could take this hybrid design as well. The light switch could have a builtin motion sensor and timer, but it could also be controlled by a brain to override its function.

This similar hybrid design can also be applied to corporate decision making. You always hire people who can think independently and figure out on their own how to get things done, but you lead them with objective, strategy, and coordination.

Crypto Ecosystem

The crypto stack is complicated. There are many APIs and implementations of the same API. There are different ways the components could be configured to interact with each other. Here is my attempt at visualizing the various relationships of the crypto ecosystem.

Here is the original drawing.

The depicted components:

• Applications: Chrome, Firefox, GnuPG, Keychain Access, OpenSSH.
• APIs and protocols: CCID, PC/SC, PKCS#11.
• Smart card drivers: CryptoTokenKit, OpenSC, *.tokend.
• Hardware: PKCS#15 card, OpenPGP card.

Yubikey NEO-n

Just got my Yubikey NEO-n today and here is how I got it to work.

• Downloaded Yubikey Manager (optionally, also Yubikey Personalization Tools).
• I changed the connection mode to OTP+U2F+CCID.
• Yubikey Personalization Tools only work if OTP mode is enabled. If not:
• Both ykpersonalize, ykinfo report "no yubikey present"
• Yubikey Manager can only enumerate CCID apps if:
• No other programs are using PC/SC (e.g. gpg-agent).
• ifdhandler has to be loaded (which may not be the case if the workaround below is applied for gpg-agent getting stuck).
• The key already has OpenPGP app installed among others.
• Followed the instruction in My Perfect GnuPG/SSH agent setup.
• For Mac OS X, I downloaded GPGTools which came with the gpg command line and the gpg-agent.
• If the 'generate' command doesn't ask for "Please specify how long the key should be valid" it might be that gpg-agent got stuck. Run killall -KILL gpg-agent and try the gpg command again.
• Mac OS X Yosemite ships with a really buggy pcsc-lite implementation. Here is a workaround so that gpg-agent would not get stuck again.
• sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ifdreader.plist
• I added -w to make the unload persist across reboots.
• After this, scdaemon will use its own CCID driver to talk to Yubikey. Both OTP and U2F will still work since neither are interfaced through PC/SC.
• Yubikey Manager will suffer reduced functionality since it no longer has CCID access:
• It can no longer enumerate Available apps.
• If OTP and U2F are both disabled, it will report "No device found" even if CCID is enabled.
• Use launchctl load -w to re-enable.

Currently I'm using Yubikey NEO-n with OpenPGP key for SSH login. Although the OpenPGP key is protected by a PIN, I only need to enter it once, and the key remains unlocked until I remove the NEO-n from USB. The problem is that while the key is unlocked, any malicious program I run could then gain access to my gpg-agent and impersonate me, which is not very secure.

With U2F, a touch would be required before authenticating with a server, which makes impersonation more difficult. A remote attacker would have to convince me to touch the Yubikey physically. There is a patch in progress making U2F work with SSH directly, but it hasn't been accepted upstream.