Saturday, February 26, 2011

Redundancy of audio packets over lossy network

I wanted to know what people currently do to stream audio packets over a network with packet loss. There are several options for streaming audio, and they vary in their effectiveness against packet loss.

  • AirTunes (aka. RAOP) is the protocol used between iTunes and Airport Express, based on RTSP (for establishing session) and RTP (for transmitting data). My impression is that you could stream MPEG4 AAC and Apple Lossless Audio Codec to Airport Express, though it may be possible to stream uncompressed audio too. Note that audio codec itself introduces algorithmetic delay, since compression is performed by grouping audio data over a time window. It is not clear how the protocol deals with packet loss, since RTP does not define how to deal with it.
  • Netjack connects a jack server to a slave. It can stream CELT or raw audio over UDP. CELT is a low-delay audio codec with packet loss concealment. However, a tradeoff of the low-delay is that it could not resolve low-frequency pitch (~100Hz fundamental), and has to rely on a long term predictor. I'm not sure how this affects audio quality in practice. The packet transport does not seem to attempt recovering from packet loss, but instead relies on CELT codec for packet loss concealment. The slaves audio time clock is synchronized to the jack server.
  • Netjack2 is really a distributed version of jack, using remote computing nodes to apply computationally intensive audio filtering, instead of playing back sound. They implemented a discovery protocol based on multicast, and audio is transmitted over UDP, with no packet loss handling. When using an audio adapter for streaming audio to a slave sound card device, audio resampling is applied to compensate for clock drift.
  • Jacktrip, developed at Stanford CCRMA, connects two jack servers over a network. There doesn't seem to be any time synchronization. Data is transmitted over UDP, and it is possible to transmit redundant data in a rolling window in order to recover from data loss.
  • There is a proposal (quick view) in using Reed-Solomon code to transmit data over UDP at Kyushu University.
I'm particularly interested in technique that could deal with wireless network packet loss, which has the following characteristics:
  • Wireless packet radio implements low-level retransmission of lost or corrupt packets. On the higher level, this is perceived as high packet delay.
  • When packet loss on the higher level does occur, it usually happens in bursts of packets. I estimate that up to 3ms radio gap is possible in normal condition.
Regarding these characteristics, the rolling window redundancy used by Jacktrip would not be effective, since a whole streak of packet loss would not be recoverable under this scheme.

When using Reed-Solomon, if we use byte-sized symbols, then a code block can only be 255 bytes (including parity). If we naively transmit the 255 bytes over UDP as a single packet, it is possible that the whole block would be dropped altogether. But we could transpose the block so that, when transmitting n blocks over m UDP packets, each UDP packet would contain 1/m slice of n blocks. This seems to be the approach taken by the Kyushu University proposal. This scheme, known as Cross-Interleaved Reed-Solomon, is also used on audio CD.

Tuesday, February 15, 2011

The real way to disable Adobe Updater from your Mac OS X

Adobe update manager is really annoying, but most instructions on the web to disable it merely tells Adobe Updater not to report updates; the updater still runs silently. The fact that I'm dedicating system resource every now and then so the Adobe Updater can phone home but not tell me to update is not good enough for me. I want Adobe Updater to stop completely.

To stop Adobe Updater completely, one must understand how it gets run in the first place. The updater is launched by a Mac OS X system service called launchd. To launchd, Adobe Updater is a periodic job. The job file is stored under your ~/Library/LaunchAgents folder. The actual file name is suffixed with a number of random characters, but it starts with "com.adobe.ARM" as the prefix. If you look inside the file (it's a plain text file), you'd see that launchd would run the updater at 12600 seconds interval, or 3.5 hours.
To remove, type these commands in a Terminal window:
cd ~/Library/LaunchAgents
launchctl remove `basename com.adobe.ARM.* .plist`
rm com.adobe.ARM.*
Basically, the idea is, for each launchd plist file in ~/Library/LaunchAgents that you don't want, run launchctl remove on the job name, which is the same as the plist file name without the .plist suffix, then remove the actual .plist file.

While you are at it, there may be other launchd jobs in ~/Library/LaunchAgents left over from stale applications you might have tried before. Feel free to remove them all.

You're welcome.

Edit (Oct 20, 2012): a couple of readers pointed out in the comment that the launchd namespace used by Adobe Updater is now different. I just installed Adobe (Acrobat) Reader XI and found that the name is still com.adobe.ARM.*, but if you have Creative Suite, it might be com.adobe.AAM.* instead. I don't have Creative Suite so I can't verify that.

Furthermore, it appears that when you set Updater preference in Adobe Reader XI to "Do not download or install updates automatically," it now removes the launchd task as well, which means the launchctl and rm commands would no longer be necessary. Kudos to Adobe for figuring that out!

One reader also pointed out that in his case, the updater is installed in the system-wide location /Library/LaunchAgents. In that case, you will need to run “sudo su -” first and type in your own password to gain root privilege (the prompt changes from “$” to “#”) before they can be removed. Be careful the commands you enter as root, as a mistake can irreparably damage your system.

Thanks for keeping me updated y'all.

Friday, February 11, 2011

How many aligned address space can I allocate via mmap?

Suppose I want to allocate a naturally aligned piece of memory through mmap(), say I want to allocate a 16KB block of memory aligned to 16KB, what is the greatest amount of memory inside the address space I could allocate in that way? What if I want to allocate 4MB aligned to 4MB? Or 16MB aligned to 16MB? The table is the result in number of that sized chunks and the total amount in GB.

OS4KB16KB4MB16MB64MB
Mac OS X 10.5.8900227 (3.43GB)225056 (3.43GB)877 (3.43GB)217 (3.39GB)52 (3.25GB)
Linux 2.6.18 (32-bit)1048173 (4.00GB)262040 (4.00GB)1020 (3.98GB)252 (3.94GB)59 (3.69GB)
Linux 2.6.18 (64-bit)6168666 (23.53GB)1540236 (23.50GB)6016 (23.5GB)1504 (23.5GB)376 (23.5GB)

The machine that ran the 64-bit test is an AMD Opteron 8220 SE with 40 bit physical address lines (1024 GB) and 48 bit virtual address lines (262144 GB). However, I can't seem to actually allocate that much memory at all.

The source of the program can be found here (alignedmmap.c). Note that since the address is never touched, this does not allocate any actual memory, and the program is so tiny that it hasn't loaded very many shared objects besides standard C library.

Friday, February 4, 2011

Putting IPv6 to the test

It's been a while since I configured my Airport Express with an IPv6 tunnel to Hurricane Electric's tunnelbroker.net. While this allows me to be IPv6 enabled, I've been wondering what my Internet experience would be like in a world where I can't get IPv4 addresses anymore, now that IPv4 address exhaustion will happen any minute now. Since today I'm sick at home, it's a perfect time to play in between sleep and orange juice.

I turned off IPv4 on my computer. The first surprise is that DNS didn't work. The reason is because, even though I configured the Airport to use Hurricane Electric's IPv6 DNS server 2001:470:20::2, the IPv6 stateless auto-configuration used by Airport Express does not propagate that information, and Airport Express does not in fact provide DHCPv6.

After configuring the DNS manually, it's time to hit Google. All of Google's services work fine as far as I can tell. I can watch YouTube videos. My blog is working, since it's hosted by Google. The bad news is that I could no longer reach most of the search result, which is not a surprise. The search experience is disappointing because I don't know which sites are IPv4 only. For example, when searching for IPv6 stateless auto-configuration, the RFC mirror faqs.org does not work, but ietf.org works. The good news is that the "cache" link allows me to look at the page despite the site being unreachable over IPv6.

Most of the big websites aren't reachable over IPv6: facebook.com, yahoo.com, live.com / bing.com / microsoft.com, wikipedia.org, amazon.com, linkedin.com, ... cnn.com, newegg.com, apple.com, etc. Also, w3.org isn't IPv6 ready, neither is mit.edu (it's not my alma mater, but you'd think they're at the forefront of technology).

To be fair, what is probably going to happen when an ISP runs out of public IPv4 addresses for its subscribers is that they will run a large scale NAT router, and assign private IPv4 addresses instead. You will be getting an address like 10.xx.yy.zz, which is not reachable from the Internet, but you will still be able to reach out thanks to NAT, which stands for network address translation. This situation is already like most home Internet router where your broadband only gives you one public IP address that has to be shared among several devices at home. You home computers already get private IP addresses like 192.168.aa.bb. The problem with large-scale NAT router is that it's probably going to be slower and less reliable—since it has to know how to translate thousands of connections concurrently, and it is a single point of failure—giving you crappy Internet experience.

Another solution is that, when an ISP runs out of IPv4 addresses and starts putting subscribers on IPv6 only, it could simultaneously provide an HTTP proxy server reachable from its own IPv6 network that will allow IPv4 websites to be visited through this proxy. At least the HTTP proxy can be load balanced, so it would not be a single point of failure.

Since the burden of IPv4 interoperability seems to lie on the shoulder of ISPs, I can see why the big websites have been slow to adopt IPv6. But the truth is, you can't count on an end-user's ISP to bring IPv6 users to your IPv4 website reliably, and the IPv6 native sites are going to give a better end-user experience for native IPv6 users.

My own experiment tells me that I'm not ready to retire IPv4 yet (I'm not even using a native IPv6 stack), though I could if I'm only interested in writing blogs and watch YouTube videos.