Canonical Voices

What Ryan Finnie talks about

Ryan Finnie

A year ago, I launched M29, a URL shortener with a twist. Apparently I forgot to announce it here. Whoops.

Normal URL shorteners are fairly simple. You submit a long URL. The service generates a short URL. The long and short URL are placed in a backend database. If you go to the short URL, it redirects to the long URL.

This means that the URL shortener service has a large database of URLs available to it. While 99% of the contents of this database may be mundane, it's still a large, centralized source of information. Very relevant to the recent NSA news, for example.

M29's twist is, except when serving the redirect, it does not know anything about the contents of the long URLs. This is accomplished by generating an AES-128 key, using it to encrypt the long URL, and then splitting the key in two. One half of the key is stored in the backend service, and the other half is encoded as part of the short URL itself. This means the only time the two parts of the key come together is when the short URL is requested for the redirect.

Getting from a long URL to a short URL can be done one of several ways. If you go to m29.us and have Javascript enabled, the client side actually loads an AES libary, builds the key, encrypts the URL, and sends the encrypted URL and half of the key to the server, all processed on the client side. If you don't have Javascript enabled, this task is farmed out to the server side, which generates a random key, encrypts, makes the database insert, returns the short URL, then throws away half of the key. M29 also has a featureful API which lets you do these tasks yourself. (It is also compatible with the goo.gl API, which is easy to work with and has several tools available.)

The net effect is, while I currently have a database of about 10,000 entries, I cannot read them. Source IP and URI logging are not done on the server, so the only way I can find a long URL is if I load a full short URL, which is not possible given just the backend database.

Anyway, this weekend I did some work on M29, including adding a bit.ly-style preview option (append a "+" to the short URL to get its info), among other small feature additions and fixes. It was then I realized, by going to that above short URL (the first URL generated and used in documentation) that the one-year anniversary of the service is today.

Read more
Ryan Finnie

The original Half-Life and Counter-Strike games were quietly released for Linux and Mac OS X last week, and as the maintainer of SteamLink, a repackaging of Half-Life: Uplink for Steam, I went out to see if the mod's files could be installed on these platforms.

Turns out there is a bug in Steam for these platforms, where it tries to launch the Windows version of Half-Life for GoldSrc mods from within Steam. However, Half-Life can be manually launched and pointed at the mod.

I have released a new version of SteamLink as a zip file. If you would like to run Half-Life: Uplink on Linux or OS X, simply download and extract the zip, and run the installer shell script. It will determine the Half-Life installation directory, install the mod, and give you a symlink to a script to launch it.

Read more
Ryan Finnie

I'm working on re-doing my home router / VM server to provide better IO. The goal is to have an SSD as the boot drive, and 4 2TB disks in RAID 10 (4TB usable total) for the VMs. I'll be using md RAID for building it, however I want to be particular about where the drives are physically, for a few reasons:

  • The RAID drives are all SATA 6Gbps drives, but the server's motherboard only has 2 SATA 6Gbps ports. I've got a 2-port PCIe SATA 6Gbps controller on the way[0].
  • I want to place the two drives in each base RAID 1 set on different controllers: one on the motherboard controller, one on the PCIe card controller. This may provide extra performance, but more importantly, having each disk on a different controller protects the array as a whole in case of a controller failure.

Linux md RAID 10 is a relatively new mode. In the past, you would manually create multiple RAID 1 arrays, then combine them in RAID 0, thereby knowing where the disks were placed, since you were doing it yourself. The mdadm RAID 10 method is easier, but I found there is literally no documentation on what drives it uses for the underlying RAID 1 arrays. Using loopback devices and some trial and error, I figured out how the arrays are assembled.

In a nutshell, the underlying RAID 1 arrays are paired two at a time, in order given during creation. If you were to do this:

# mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sd{a,b,c,d}1

sda1 and sdb1 form one RAID 1 array, and sdc1 and sdd1 form another:

|---------------------------|
|           RAID0           |
|---------------------------|
|    RAID1    |    RAID1    |
|-------------|-------------|
| sda1 | sdb1 | sdc1 | sdd1 |
|---------------------------|

One other thing to mention is what happens when multiple drives are lost. Say in this case, both sda1 and sdd1 are lost, resulting in a degraded but functional array:

# mdadm --fail /dev/md0 /dev/sda1
# mdadm --remove /dev/md0 /dev/sda1
# mdadm --fail /dev/md0 /dev/sdd1
# mdadm --remove /dev/md0 /dev/sdd1
|---------------------------|
|           RAID0           |
|---------------------------|
|  RAID1 (D)  |  RAID1 (D)  |
|-------------|-------------|
|      | sdb1 | sdc1 |      |
|---------------------------|

If you were to replace sdd and add it back first, you might think it would go in the second RAID 1 array. But no, it takes the first available degraded slot:

# mdadm --add /dev/md0 /dev/sdd1
|---------------------------|
|           RAID0           |
|---------------------------|
|    RAID1    |  RAID1 (D)  |
|-------------|-------------|
| sdd1 | sdb1 | sdc1 |      |
|---------------------------|

So be careful in this situation, if you care about where the devices are physically laid out.

[0] Half of the order actually arrived Friday, including the SSD and a 4-port PCIe 4x SATA 6Gbps controller. The idea was to place two of the RAID drives on the motherboard SATA 6Gbps controller, and two on the new controller, plus the boot SSD (which is also SATA 6Gbps). My system has 3 PCIe 1x ports and a PCIe 16x port. The 4x card was supposed to go in the 16x port, but I learned after failure that many motherboards do not like non-video cards in the primary 16x port. Oh well. The boot SSD will now go on one of the motherboard SATA 3Gbps ports, and I've got an order in for a 2-port PCIe 1x SATA 6Gbps controller.

Read more
Ryan Finnie

I'm not a programmer, honest, but lately I've had this annoying habit of releasing code. But instead of my normal rambling posts about new announcements, I'll try to keep it brief.

apache-vsl has been one of my "90/90" projects (90% done, just need to finish the remaining 90%) for many years; I believe I started it around 2006, and I finally got the urge to finish and release it to the public. It's an Apache logging daemon (managed by Apache itself), capable of intelligently handling multiple VirtualHosts while being as efficient and extensible as possible. It's specifically built for two use cases -- servers which have a large number of VirtualHosts (hundreds or more), and VirtualHosts which receive a lot of traffic -- but it's easy enough to manage that it's useful with any Apache installation.

You can define time-based log files (monthly, daily, hourly... it's all just strftime), and apache-vsl handles when to log to a new file according to your definition. It manages symlinks to both the current and previous logfiles (for example, access_log.2012-08 -> access_log and access_log.2012-07 -> access_log.old), and has support for program triggers when a rotation happens (say, for running Webalizer against the old logfile, or compressing it).

apache-vsl manages its open files efficiently, so for a server with many VirtualHosts (not all of which may be accessed very often), all of the VirtualHosts' log files won't be open at once, but for VirtualHosts which are accessed frequently, the log filehandle will not constantly be opening and closing.

apache-vsl shares many features with cronolog, but while cronolog is designed more for splitting logfiles after the fact (if you were to use it directly from Apache, you'd need to have Apache open a dedicated pipe for every VirtualHost), apache-vsl is designed to communicate with Apache with a single pipe for multiple VirtualHosts.

A lot of good documentation is available in the manpage, so if you're interested, I suggest you start there.

Read more
Ryan Finnie

Raspberry Pi

Raspberry Pi

My Raspberry Pi, the $35 barebones computer, arrived last week. I bought it for three primary reasons:

  1. It was cheap.
  2. It was interesting.
  3. It was cheap.

I ordered early on a Thursday morning, expecting 3-4 weeks before it shipped per Element14's estimates. So I was quite surprised when I got a shipment notification later in the day. I'm guessing Element14 had a batch in, and was processing the backorders from the last few weeks, and somehow my order got mixed into that. So hey, yay instant gratification!

Shortly after that, Adafruit's clear acrylic case became available again, so I ordered that as well. I was expecting a tighter fit, but instead it bounces around a bit within the case. It's still sturdy enough to do its job, and looks nice, though the top panel no longer has the etched Raspberry Pi logo like I've seen on previous photos.

Raspberry Pi

I also ordered some cables and a powered USB hub from Monoprice (gotta love $5 next day shipping to California/Nevada), and a high power iPad USB charger from OWC, but that turned out to be unnecessary. The 7-port Monoprice USB hub seems to provide enough power to power the Raspberry Pi itself, along with some accessories.

In the above setup, I've got it powering the Raspberry Pi (with HDMI output, plus 16GB SDHC card), a keyboard and mouse, a 4GB USB thumb drive, and my Defcon 20 badge. Not only does the Raspberry Pi boot, but I gave it a stress test for a few hours: Running a Quake 3 Arena 1920x1080 demo on loop (GPU), while simultaneously running Bonnie++ on the USB thumb drive (CPU and IO). It successfully did that for the entire test, so I'm confident in the power performance of the USB hub.

Now, I have no real plans for it; it'll probably sit, powered on in the corner of my office, for if I need to test something on an ARM system. But I do like the concept of its primary purpose. My first computer was a Vic 20 (predecessor to the Commodore 64), and then a Commodore 128 (backwards-compatible successor to the Commodore 64). I have fond memories of those computers, and they helped shape the course of the rest of my life. I like how the Raspberry Pi has both HDMI and Composite output, meaning it supports any TV or monitor made in the last 5 years or so, plus at least the previous 50 years of TVs. (Remember RF adapters?) This really lowers the barrier of computer use for your average child.

However, there is one problem with the Raspberry Pi I really disagree with. Namely, it requires an operating system. A 440MB operating system, as well as the means to get it onto an SD card. An operating system that could easily be destroyed by curious fingers. This could be a huge barrier.

One of the things about my Vic 20 and Commodore 128 was I didn't have any cartridges or pre-bought software. Just the computer, a tape drive, and some CR-90 tapes. All value I got out of them was either by typing in BASIC programs from BYTE or Commodore magazines, or creating programs myself.

The Raspberry Pi really needs an onboard ROM with some sort of simple language on it, such as BBC BASIC, and the ability to save directly to SD cards. Upon boot, if the SD card contains a bootloader pointing to a full operating system, boot that, but if not, load the BASIC ROM.

Read more
Ryan Finnie

You may remember about a year ago when I released twuewand, a TrueRand implementation. TrueRand is a hardware entropy generation technique, implemented in software. In a nutshell, it works by setting an alarm for a few milliseconds in the future, and flipping a bit until the alarm is reached. It works due to the fact that time (your computer's RTC) and work (your computer's CPU) are not linked, so the result when the alarm comes due is unpredictable.

TrueRand was invented in 1995, and had mostly been forgotten for the last decade, until I started doing research on it last year. So it was quite a surprise when I was at Dan Kaminsky's talk at DEFCON a few weeks ago, and one of the topics he brought up was TrueRand. (Go check out his presentation slides; I just want to point out that while I'll be focusing on entropy and debiasing here, he goes into a lot of other interesting topics.)

Dan came to roughly the same conclusion as I did, that entropy sources have gotten worse over time, not better, and systems like VMs are almost completely devoid of entropy. Even more worrying, a paper published this year came to the conclusion that approximately 1 out of every 200 public keys on the Internet are easily breakable, not due to weaknesses in the encryption, but by bad entropy being used when generating the keypair. TrueRand may have been forgotten, but it's needed today more than ever. Dan and I talked for awhile after his talk, and went over a few things by email in the week following. twuewand 2.0's new features are influenced by those discussions.

Dan proposed a number of enhancements for TrueRand, mostly centered around other ideas for measuring variances given only a CPU and RTC, but what caught my eye was his idea of enhancing debiasing.

Many forms of random data are random in the technical sense, but are prone to bias. As an theoretical example, take a gun placed in a sturdy mount and pointed at a target not too far away. Most of the time, shots from it will hit the same spot every time (0), but occasionally they won't (1). So you're left something like 00000001000001001100000010000010; mostly hits, but with random misses. So it's random in a technical sense, but the distribution is heavily weighted toward one side.

The simplest method of debiasing is known as Von Neumann debiasing. Bits are processed as pairs, and any pair that is both 0 or both 1 is simply thrown out. Out of the pairs that are left, {0,1} becomes 0 and {1,0} becomes 1. So in the example above, the Von Neumann debiased output would be 00101. The data is now distributed better, but as you can tell, a lot was lost in the process. This is an extreme example since the data was heavily biased to begin with, but in data without a lot of bias, you still lose at least 50% of the bits (I've found 70-75% in real-world twuewand usage).

Dan thought, "Hmm, that's an awful lot of data simply being thrown out. We can't use the discarded data in the direct output, but perhaps we can use it to better (de-)influence the final output." He came up with a method he called modified Von Neumann, which I refer to in twuewand as Kaminsky debiasing.

The incoming bit stream is still run through Von Neumann, and put into an output buffer. However, all bits (whether they pass Von Neumann or not) are fed to a SHA256 stream. Occasionally (after the input stream is finished, or a sufficient number of bytes are put into the output buffer), the SHA256 hash is computed[1], and used as a 256-bit key for AES-256-CBC encrypting the output buffer. This way, only the bits which pass Von Neumann influence the output directly, but all bits help indirectly influence the output as well.

So twuewand now supports Kaminsky debiasing, and will use it by default if Digest::SHA and Crypt::Rijndael are installed.


Now, I want to clear up a mistake I made in my last post. I said that feeding twuewand output to /dev/urandom on Linux systems influences the primary pool, increasing entropy. First, you can actually write to either /dev/random or /dev/urandom, the effect is the same. But more importantly, entropy is NOT increased by writing to /dev/[u]random. It's merely "stirring the pot". If your system is out of entropy and you are blocking on /dev/random, no amount of writing to /dev/[u]random will unblock. (Directly, that is. If you're banging on your local keyboard to do this, you're slowly increasing entropy, but you could be doing the same thing in a text editor, or to /dev/null.)

Unfortunately, there is no way to increase entropy in the primary pool via standard system command line tools or character devices. However, there is a Linux ioctl, RNDADDENTROPY, which does this. So I wrote a small C wrapper, which takes STDIN and feeds it to the ioctl. This requires root of course. The utility is called, boringly enough, rndaddentropy, and is distributed with the twuewand tarball. It will be built by `make` on Linux systems.

I must point out that this utility gives you a very excellent method to shoot yourself in the foot. The lack of command line tools to directly access the primary buffer is most likely by design, since this bypasses most of the in-kernel filters for random input. Injecting, say, the contents of /bin/ls to the primary pool would be a great way to become one of those 1 in 200 statistics. Only use this utility to feed high quality entropy (such as by twuewand, or something like an entropy key).


Dan Kaminsky will be publishing software in the future called DakaRand, which does much of what twuewand currently does, but incorporates some of his other ideas. He provided me a work-in-progress copy, which looks very interesting, but it is not available to the public yet for a number of reasons. Be on the lookout for that when it is released.

Update (2012-08-15): Dan released DakaRand 1.0 a few hours after I made this post. Go check it out.


[1] In Dan's proposal, after the SHA256 hash is computed, it would then be run through Scrypt's hashing algorithm. This is not done in twuewand for two reasons. First, Crypt::Scrypt does not currently provide a low-level method to just do hashing; instead it wants to create a full digest which is unsuitable for this purpose. Second, Dan has been debating whether this step is necessary or desirable at all, and Scrypt has "undefined effects on embedded hardware".

Read more
Ryan Finnie

[ This was written at 2AM, and I tended to ramble. Hopefully I'll come back tomorrow and clean it up, add links to RFCs and Wikipedia entries where relevant, etc. ]

I've been explaining this a number of times in the last few days, so I figured I'd get this down in blog form to avoid repeating myself (even more). IPv6 is much like IPv4 in most respects, but its autoconfiguration can seem rather alien to people already familiar with IPv4 (that is, DHCP).

On an IPv6 network, the default router runs a service called the Router Advertisement daemon. Its primary purpose is to send out packets saying "hey, I'm the default router for the 2001:db8:1234:abcd::/64 network".

It actually doesn't matter what the router's IPv6 address actually is on 2001:db8:1234:abcd::/64; indeed, it (usually) doesn't even tell your what it is. Instead, it sends out RAs using its "link-local" fe80:: address. Link-local addresses are specific to the LAN segment you're on, and every device is required to have one (derived from the MAC address). It's sort of a middleware layer of doing things, between Layer 2 and Layer 3. But the important thing is you can talk to other devices if you know their link-local fe80:: address (and you're both on the same segment), and you can route to them.

So it really doesn't matter if the router's address is 2001:db8:1234:abcd::1/64 or 2001:db8:1234:abcd::dead:beef/64 or if it doesn't even have an address; if you're doing autoconfiguration, your OS will most likely be routing to it based on its link-local address. For example, my home router's link-local address on the LAN side is fe80::6a05:caff:fe02:4e4d/64, which it uses to send RAs.

By default the router sends an RA every minute or so, so the client device could theoretically set up autoconfiguration without ever talking to the router. But it will also respond to Router Solicitation requests being broadcast to the network. These requests are responded to (via broadcast) with the exact same RA packet it would have sent on a timer, except this time it happened on-demand.

Now, the RA lets the client device know what the LAN's IPv6 network space is. It also sends a series of flags which can make autoconfiguration possible.

The most relevant is the Autonomous address-configuration ("A") flag. This flag provides stateless autoconfiguration, and essentially says "you are permitted to construct your own IP address in the network I mentioned". The client will then use the MAC address of the interface as a basis for constructing an IP address. In the above example, 2001:db8:1234:abcd::/64 was the network. Say my interface's MAC address is 00:19:d2:43:c0:c4. Using the recommended MAC-munging method, this becomes 2001:db8:1234:abcd:219:d2ff:fe43:c0c4/64.

The client then uses Duplicate Address Detection (DAD) to make sure nobody on the network is already using this address. (DAD is also used to make sure the link-local address mentioned above is also unique to the network segment.)

So that's it! The client now has enough information to construct IP information, and in theory it never even had to send a packet. (I say "in theory" because DAD requires sending out a multicast packet. Plus, most OSes send out a Router Solicitation to trigger the Router Advertisement.) In this example, the router, fe80::6a05:caff:fe02:4e4d, sent out an RA stating the network is 2001:db8:1234:abcd::/64, and the "A" flag was set. The client used this information to construct the IP 2001:db8:1234:abcd:219:d2ff:fe43:c0c4/64, which it gave to itself, with the default route being fe80::6a05:caff:fe02:4e4d.

Now what about DNS? In a dual-stack environment, the client probably already has an IPv4 DNS address by now, and can certainly use that to look up IPv6 (AAAA) records. But for pure IPv6, there are two methods for getting DNS servers to the client.

First up is RDNSS, a relatively recent extension to the RA standard. It's simply an extension which gives one or more DNS servers in the RA packet. Simple, but not often used today.

The RA also has a "Managed address configuration" ("M") flag. This simply means, "there is a DHCPv6 server on this network. You may go look for it for more granular control." Also known as stateful autoconfiguration.

DHCPv6 is similar to DHCPv4, but has a number of differences. I'm not going to go into it in detail, but you can do pretty much everything you can do with DHCPv4: pooled address allocations, individual address allocation, DNS servers, domain search lists, etc. It's worth noting that if you have both the "A" and "M" flags in the RA set (and you're running a DHCPv6 server), the client will use both of these methods. That is to say, it'll construct an IP address using stateless means, then request an IP address from the DHCPv6 server, and will bind both of these addresses to the interface.

Side note: IPv6 has no concept of network or broadcast addresses. In this post's example, 2001:db8:1234:abcd::/64 was the network, but 2001:db8:1234:abcd:: can also be a device. However, most networks I've seen continue the old tradition of giving the .1 (or in IPv6's case, :1) address to the default router.

Read more
Ryan Finnie

I solved my problem from a few months ago with my iPhone and iPad not being able to get IPv6 info. It was actually a combination of two problems.

Last year I bought a Netgear N750DB router as an access point to replace my Linksys WRT54GL. Turns out the N750DB will send multicast traffic from a wireless device back to itself. This breaks IPv6 Duplicate Address Detection (DAD), and I was also seeing problems on my Ubuntu laptop. So I went back to the WRT54GL, and while stateless autconfiguration started working on the laptop again, it was still not working on the iPhone and iPad.

The second part was the iOS devices needed to be rebooted. No idea why, but after a hard reboot, they started pulling RA (and DHCPv6, which I had configured in the meantime). I've since bought a Linksys E4200 v1 to replace the N750DB, and everything is now working great.

Note that it's actually hard to tell when iOS has an IPv6 address. There are literally no IPv6 options in iOS, and the IPv6 address itself is not displayed anywhere. The only hint is in the network config page. If you have a DHCPv6 server or giving out RDNSS via RAs, the IPv6 nameserver and/or domain will show up in "DNS" and "Search Domains", though will likely be cut off. The only OS-level way to get meaningful IPv6 address/route information is a utility app called IPv6 Toolkit ($2 but worth it in this case). You can also go to a web site like vsix.us (which of course I wrote) to see what your IP address is.

Note that even after fixing my AP and rebooting my devices, when I went to vsix.us in Safari, it was still preferring v4 over v6. I was able to solve that by clearing all cached info in Safari.

Read more
Ryan Finnie

This weekend, I upgraded the OS on my main colo box, from 32-bit Debian lenny to 64-bit Ubuntu 12.04 precise LTS. This was an installation which I've had for over 8 years. It was originally installed in early 2004, when Debian sarge was in its long beta freeze. It had been dist-upgraded over the years, from sarge to etch to lenny, and was quite well maintained. And over the years, the hardware has been upgraded, from a Pentium 4 with 512MB memory and 80GB usable disk, to a Pentium D with 2GB memory and 160GB disk, to a Core i7 with 9GB memory[1] and 320GB disk.

However, it was still an 8-year-old installation, and I wanted to get to a 64-bit OS. Debian lenny's support was dropped in February, and Ubuntu precise seemed like a good target, so I made the decision late last year to replace the OS.

I bought two 1.5TB WD Black SATA drives and installed precise on them a few weeks ago, in a RAID1 set. My colo box in San Jose has a 3ware 9650SE 2-port SATA 2 card, and I had an identical card available at home. Thankfully 3ware cards store RAID configuration on the disks themselves, so it was just a matter of installing the OS on the surrogate server at home and shipping the disks to San Jose to be physically installed. I also bumped the memory up to its maximum of 24GB, as long as the case was open (memory is insanely cheap right now).

The precise install was a minimal install, with just networking and SSH configured. I then took a backup of the lenny install and put it in /srv/oldos. The idea was once the datacenter swapped out the disks and powered it on, I'd go in and, for example, "chroot /srv/oldos /etc/init.d/apache2 start" for all the essential services. I could then migrate the services into the new install one at a time.

With a few small bumps[2], that strategy worked well. However, a LOT of stuff is on this box, and it took me a most of the weekend to get everything right. Here's a sample of what this server is responsible for:

  • HTTP/HTTPS (Apache, a dozen or so web sites, plus MySQL/PostgreSQL)
  • SMTP (Postfix, plus Mailman, Dovecot IMAPS)
  • DNS (BIND9)
  • XMPP (ejabberd)
  • BitTorrent (bittornado tracker, torrent clients for Finnix)
  • Minecraft
  • Mumble/Murmur
  • rsyncd (Finnix mirrors)
  • 2ping listener
  • The all-important TCPMUX server
  • Various AI, TTS, text manipulation and audio utilities for X11R5 and the Cybersauce Broadcasting Corporation

As of Monday, everything is now running off the new precise install. I installed this server over LVM, and have only allocated about 500GB of the 1.5TB usable space. There is a relatively small 120GB root partition, a 160GB /home partition, and a 250GB /srv partition. The idea is not much more than the OS should be on root (and should never need more than 120GB), /home is self explanatory, and all projects (web sites, etc) go in /srv. If /home or /srv need to be expanded, it can be done remotely relatively easily by umounting them, expanding the partitions using LVM, and running resize2fs. I've also left most of the space unallocated in case I want to run KVM guests some day, though I don't have an immediate need.


[1] Yes, 9GB memory. Triple channel, so it was 3x2GB + 3x1GB modules.
[2] I had two significant problems with the migrations:

  1. The problem with picking a custom third-party Festival voice for the voice of the Cybersauce Broadcasting Corporation is that it looks like it has not been updated in years, and is specific to only a handful of Festival versions. So now I have a minimal, yet 300MB chroot running (64-bit) Debian lenny for the sole purpose of providing Festival's text2wave utility with the custom voice.
  2. I'm using two packages[3] not available in Debian/Ubuntu: a Perl module (AI::MegaHAL) and a PHP module (crack), both of which are published at my PPA. Both are sufficiently ancient and have not been updated upstream for years. Both compiled fine in 64-bit precise and at first appeared to run fine, but would then crash. Turns out both were not 64-bit safe, and required patching. Thankfully this is easy to roll out with PPAs.

[3] Okay, three, technically. I've packaged the tw_cli utility for the 3ware RAID card, but cannot distribute the package through my PPA because it's a non-free binary.

Read more
Ryan Finnie

2ping 2.0 has been released today. User-visible changes are minor, but behind the scenes, a major update to the protocol specification has been implemented, justifying the major version bump:

  • Updated to support 2ping protocol 2.0
    • Protocol 1.0 and 2.0 are backwards and forwards compatible with each other
    • Added support for extended segments
    • Added extended segment support for program version and notice text
    • Changed default minimum packet size from 64 to 128 bytes
  • Added peer reply packet size matching support, turned on by default
  • Added extra error output for socket errors (such as hostname not found)
  • Added extra version support for downstream distributions
  • Removed generation of 2ping6 symlinks at "make all" time (symlinks are still generated during "make install" in the destination tree)

2ping is a bi-directional ping utility. It uses 3-way pings (akin to TCP SYN, SYN/ACK, ACK) and after-the-fact state comparison between a 2ping listener and a 2ping client to determine which direction packet loss occurs.

Read more
Ryan Finnie

One night last week, my cable internet service went down. OK, it rarely happens, and when it does, I have an iPhone with tethering. No big deal normally, I would just have my laptop connect to the tether. However, that night I was in the middle of a few things on my home LAN, and wanted my whole network to have internet access, not just my laptop. Getting wireless access on my home router/server seemed the most preferable. So I dug through the junk crates and found:

  • Two Orinoco Gold 802.11b PCMCIA cards
  • Four Orinoco Silver 802.11b PCMCIA cards
  • A Linksys 802.11b USB dongle which required a USB A to A cable
  • A Linksys 802.11g PCI card
  • About a dozen WRT54G (or similar) router/APs

I immediately threw away the USB dongle due to its odd requirement. I should throw away the Orinoco cards as well, but dammit, they had real value (ten years ago). The PCI card would have worked, except my new router/server actually has no "legacy" PCI slots, just PCI Express slots. I could have cobbled together some sort of bridge with a WRT54G, but it would have been too much work. In the end I toughed it out and waited for the cable internet to return.

The next day I ordered a $17 PCI express 802.11n adapter, and it arrived today. Ubuntu 12.04 "precise" (currently in beta, set to be released next week) saw it immediately, and with a little reading I was able to get it working with my phone. I'm leaving the configs here in case they are useful to someone else:

/etc/network/interfaces:

#auto wlan0
iface wlan0 inet dhcp
    wpa-conf /etc/wpa_supplicant.conf

"auto wlan0" is commented out because I obviously don't want it running on boot, since this is a backup solution. That line could simply be omitted, but I like it there to point out to myself it is not brought up on boot.

/etc/wpa_supplicant.conf:

network={
    ssid="Ryan’s iPhone"
    scan_ssid=1
    key_mgmt=WPA-PSK
    psk="yourkeyhere"
}

iPhone-specific note: that "’" is actually a unicode character, U+2019. "iwlist wlan0 scan" was returning this, which wpa_supplicant was not accepting:

ESSID:"Ryan\xE2\x80\x99s iPhone"

In the end, I hopped on my laptop, associated with the phone and grabbed the SSID from the logs and pasted it in on the server, to preserve the unicode character.

With that in place, "ifup wlan0" worked fine. In the future, during an outage I should just be able to log into the server, run "ifdown eth1", then "ifup wlan0", and everything at home should be run through my phone.

Read more
Ryan Finnie

A copy of this announcement is available at http://www.finnie.org/rfinnie-openpgp-2012-transition.txt, in case the text is mangled here and the signature cannot be verified.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256,SHA1

Wed, 11 Apr 2012 10:30:08 -0700

For a number of reasons, I've recently set up a new OpenPGP key, and 
will be transitioning away from my old one.  My old key was created 
over 10 years ago, as a 1024 bit DSA key with a SHA-1 signatures, both 
of which are considered inadequate today.  My new key is a 4096 bit RSA 
key with SHA-256 signatures.

The old key will continue to be valid for at least 90 days.  It will be 
revoked on or around 2012-07-15, or after the release of Finnix 105, 
whichever is later.  (My old key was used to manage signatures for the 
Finnix project.  This will be split out into a Finnix-specific signing 
key, and will be announced in a separate message.)

However, I would prefer all future correspondence to come to the new 
one, as of today.  I would also like this new key to be re-integrated 
into the web of trust.  This message is signed by both keys to certify 
the transition.

The old key was:

pub   1024D/203ECA25 2001-05-09
      Key fingerprint = B023 7C63 DF28 70AA C3AB  C54A 2996 10A9 203E CA25

And the new key is:

pub   4096R/86AE8D98 2012-04-11
      Key fingerprint = 42E2 C8DE 8C17 3AB1 02F5  2C6E 7E60 A3A6 86AE 8D98

To fetch the full key (including a photo UID, which is commonly
stripped by public keyservers), you can get it with:

  wget -q -O- http://www.finnie.org/rfinnie.gpg | gpg --import -

Or, to fetch my new key from a public key server, you can simply do:

  gpg --keyserver pgp.mit.edu --recv-key 86AE8D98

If you already know my old key, you can now verify that the new key is
signed by the old one:

  gpg --check-sigs 86AE8D98

The new and old keys' primary UIDs are both "Ryan Finnie 
".  This was by design, to ensure you must verify the 
key signatures rather than seeing something like "Ryan Finnie (2012) 
".

If you don't already know my old key, or you just want to be double
extra paranoid, you can check the fingerprint against the one above:

  gpg --fingerprint 86AE8D98

If you are satisfied that you've got the right key, and the UIDs match
what you expect, I'd appreciate it if you would sign my key:

  gpg --sign-key 86AE8D98

Lastly, if you could upload these signatures, I would appreciate it.
You can either send me an e-mail with the new signatures (if you have
a functional MTA on your system):

  gpg --armor --export 86AE8D98 | mail -s 'OpenPGP Signatures' ryan@finnie.org

Or you can just upload the signatures to a public keyserver directly:

  gpg --keyserver pgp.mit.edu --send-key 86AE8D98

Please let me know if there is any trouble, and sorry for the
inconvenience.

Thank you,
Ryan Finnie

[Much of this text was adapted from dkg ,
thank you!]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBCAAGBQJPhcEbAAoJEH5go6aGro2YCqYQAKM2IlO3CgOLPDYIww7tdt0t
TTYgp1ng0oOkRdSKm7maavnVd8Drkys/TgO8DQD/tuf37ZES1Vid7yqQSddAx49/
da+V9EdbCaZOqqVUY0qtW5JTV8xyn67zLwhj06/L+NWf3iP/6ymCzbWrVor2jdtn
Efeylj+T+j5igLOTBkx22d4W3VU787fiMCiwLgDmytwJ66cHR4qR+jWWnsEdVuuF
AVwcs9ELRicppE0p1jMmsr/rKJJAeM0xb1+V+BL685q4XkXRvY6Fg2WC2aoTFJF/
jp94JtlodooWuCuWnNFofqVdYIuSezjki+aRy3KmCFliWaULqL8akdtVlUmA/2gM
PdZE7Acf7JU4TVH/drvY6pbK7zwFIuBA+ESbB4lJEvZFC+Ub2aM7SceDAp2CBd+i
B4+sWkv89ZSDZqGXK2ylTNFU2o2MfQLxZWKZOdq0exZJkb5NSNF22YY8WsMsXpqJ
Ydtt0mxVp57rkhc01Vx4DJ5+OKmCJEgiTj+wnef1RvZh3ayLqkS5wUTkf6S4OLwP
cJT3i+mhAU7CQVFqSnmg98ADiq1SVnWz2rsq4m1e965ST1OpNxicK4g9UO/ePUT2
yBtyEfmFCV98KCADUSdWmD0Nx3uzHxtb+0RMMulPOQszB9VDPxIcNbdcKMLzzcp+
ZwM/dc405Tvdzptf/khgiEYEARECAAYFAk+FwRsACgkQKZYQqSA+yiXbTwCggR1l
9IHQVOKCDEJmot02C8pRFFIAnjvSY/eCeLW3mjvBF8rQUCg80KRJ
=pweu
-----END PGP SIGNATURE-----

Read more
Ryan Finnie

While debugging my IPv6 problem (which I still don't have a resolution for), I registered a domain and made yet another "what is my IP?" site: vsix.us. It looks pretty. You should visit it.

Read more
Ryan Finnie

Dear Lazyweb,

I'm at a loss here. I've got an IPv6 setup at home, with radvd giving out network information. And it basically just works. My laptop associates with the wireless network, sends out a Router Solicitation packet, the router responds with a Router Advertisement (RA), and the laptop gives itself an IP. And for good measure, the router sends out unsolicited RAs every 10 seconds or so.

But my iPhone and iPad are no longer getting IPv6 addresses. They associate with the AP, get a DHCPv4 address, then nothing. I've even tried using the Ip6config app to scan for RAs, but according to it, no RAs arrive. But I can see them from my laptop, capture them with Wireshark, and they look proper. And of course there's no way to get around this in iOS, since there are literally no user-configurable IPv6 options.

I know this worked at one point, because whenever I try searching the web, I keep coming up with my own post from 2010 on getting this working. That being "hey, it's not working... oh, now it is".

Update (2012-06-08): I've figured out what caused this.

Read more
Ryan Finnie

New home server (nibbler)It's been a long time since I've built a computer from scratch, and even longer since I've been excited about it. For the last few years, my home office network has been mostly kitbashed together from other computers, and when I've needed to buy new parts, it's usually been begrudgingly.

However, a few weeks ago, the stars all aligned. I was preparing to upgrade the RAM and drives on my colo box in San Jose after Ubuntu 12.04 is released, and went shopping. Hard drive prices are very high due to the flooding in Thailand last year, but RAM is dirt cheap. I was able to max out my colo server's RAM to 24GB (6x4GB) for $100. That got me thinking about my home office. At any given moment, 7 desktop computers were on: my main Ubuntu workstation, a Windows gaming machine, two G4 Mini Finnix dev machines (PPC), an C2D E7200 Finnix dev machine (x86), an Athlon64 3200+ Debian sid dev machine and an Athlon64 X2 4600+ router / miscellaneous server.

The desktops are used frequently, the G4s are insignificant (they consume 9w each), but the last three servers were all relatively old, very power hungry (and hot), and all perform tasks that need somewhat significant computing power, though usually never at the same time. And at idle, the three of them draw a combined 240w. So I decided now would be a good time to build a virtualization server and combine them. I considered upgrading one of them for the task, but decided it wouldn't work: The E7200, despite being relatively new, doesn't support VT. The 3200+ is nearing 10 years old (it was the first 64-bit consumer processor released) and definitely doesn't support VT. The 4600+ does, but it's pretty old, and the worst power offender.

No, it would be best to start from scratch. I decided to focus on an Intel Sandy Bridge processor, as much RAM as I could get, and a mid-range 6Gbps SATA drive for serving the VMs, with a focus on energy savings primarily and computing power secondarily. Here's what I ended up with:

The parts arrived on Wednesday the 29th, I assembled it last weekend, and installed Ubuntu precise Beta 1 (AMD64 server). I was worried about building a Sandy Bridge system, the platform being relatively new, but nearly everything has worked perfectly so far. The CPU is well supported, the integrated graphics work, and all integrated components on the motherboard work fine. The only issue I've found so far is the system will not reliably power off after shutdown, but I can live with that. Heck, this is even the first machine I've owned where SATA hotplug works correctly.

The 6Gbps Seagate is a LUKS-crypted LVM disk holding the OS installations. The main install contains the routing / VPN / DHCP / etc services, as well as a QEMU/KVM host managed by libvirt. The Finnix dev and Debian sid dev machines have been transferred to this as VM guests. And Finnix build times have been reduced by nearly 2/3rds in the process.

Additionally, the host contains an existing 1.5TB WD Green SATA 3Gbps drive for shared media (ISOs, iTunes collection, movies, etc), and an existing 1.0TB WD Green SATA 3Gbps drive for backups of the other hosts. This drive is mounted in one of the front hotswap SATA sled trays for easy removal. (I learned this lesson late last year when I evacuated my home due to a nearby wildfire, and found that even quick-release case screws and toolless hard drive mounts were difficult to remove when you are short on time.) The other sled tray is simply for miscellaneous uses when needed.

So I replaced three servers, consuming 240w combined idle, with a beefy new server: powerful new CPU, 32GB RAM, and 3 hard drives totaling 4.5TB. I was expecting the idle power draw to be about 120w. But when I plugged it into my Kill-a-Watt, I found the power draw was really... 58w. Wow. And the CPU and case temperatures always hover around 28C. My first feeling was anger with myself; I should have spent the extra few dollars on the i5-2500 or i7-2600, but I was worried about power draw and heat dissipation. Oh well, the i5-2400 is still much faster than anything I've owned before.

I got all the functionality transferred over to the new server last weekend, but haven't done much with it for the last week. (I was pretty sick this week, missing four days of work.) This weekend I hope to make sure everything is in working order, and decommission the other servers.

Read more
Ryan Finnie

A month ago, I started working at Canonical, the makers of Ubuntu. Normally I'd write a long-winded post about my experiences from the last month, how/why I started and so on, but I'll just summarize: it rocks.

I work in the IS Operations group as a system administrator, and while my Launchpad profile has the little "Member of Canonical" badge on it now, I am not an Ubuntu developer (though of course Canonical employs quite a number of Ubuntu developers). That's not to say I don't contribute to Ubuntu; I file bugs, and manage packages as trickle-down from my Debian maintainer status, but these are all contributions that any member of the public community can do. Working in Canonical IS Operations is much like being a sysadmin in any large Ubuntu server shop.

I've gone back through last year's posts and tagged relevant posts with planet:canonical, which are aggregated at Canonical Voices, a feed aggregator for Canonical employees. ("Planet" has become a generic term for feed aggregation sites, though Canonical Voices is not actually running Planet software.) These tagged posts don't necessarily have to do with Canonical or Ubuntu topics, but are a filter for posts I want to appear there; technology topics, mostly.

Last month I made a post to the Finnix blog, explaining how my employment will affect Finnix development (it won't). Since then, Finnix 104 has been released, and I even found out some of the coworkers in my group are Finnix users. Good times!

Read more
Ryan Finnie

I occasionally plug this into Wolfram Alpha:

a^2+b^2=c^2, a/b=16/9, c=27

Click the "approximate forms" solution to get the width and height (a and b) for a rectangle where you know the diagonal (c) and the ratio (16/9). a or b can be specified at the end instead of c if you know the width or height.

I most often use this when I need to get the physical width and height of a monitor panel that I know the diagonal size of (since nearly all monitors are advertised by their diagonal panel size). With that information and the resolution, you can figure out the physical DPI of the monitor. (Not to be confused with the effective DPI of the operating system, which is used for things like converting font points and ems to pixels, and is usually independent of the monitor's size and resolution: 96 DPI for Windows, 72 DPI for Mac OS, and 75 or 100 DPI for X11 historically, though many Linux distros are preset to 96 DPI today.)

Read more
Ryan Finnie

2ping 1.2 has been released, adding ping-style mdev/ewma statistics:

  • Added exponentially-weighted moving average (ewma) and moving standard deviation (mdev) statistics to the summary display

2ping is a bi-directional ping utility. It uses 3-way pings (akin to TCP SYN, SYN/ACK, ACK) and after-the-fact state comparison between a 2ping listener and a 2ping client to determine which direction packet loss occurs.

Read more
Ryan Finnie

twuewand is a true hardware random number generator, written in about a hundred lines of Perl.

No, really.

You can stop laughing now.

First, a little history. This is greatly simplified, and specific to Linux, but the concepts are somewhat universal. Linux has three entropy pools. The first is a hidden, primary entropy pool that directly or indirectly receives entropy from several main sources, described later.

The secondary pool feeds from the primary pool, and is used to drive /dev/random. /dev/random is blocking, meaning if both the primary and secondary entropy pool exhaust, reads from /dev/random block until more entropy is generated.

The third pool is the urandom pool, and functions almost exactly as the secondary pool, but drives /dev/urandom. The key difference is while the urandom pool can draw from the primary pool, it can also reuse entropy to avoid blocking in the case of pool exhaustion.

Now, entropy is gathered from several sources to directly feed the primary pool: keyboard and mouse timings, interrupts, disk activity, and entropy fed back from the other two pools, directly or indirectly. However, consider a server. Most of the time it receives no keyboard or mouse activity, and the interrupts and disk activity are theoretically predictable. But the primary pool can also be influenced by writing to the other pools, and modern Linux distributions take advantage of this. Upon shutdown, a number of bytes are read from /dev/urandom (usually 4096 today) and written to a state file. When the computer is booted again, the OS reads this file and writes the bytes back to /dev/urandom. This isn't exactly completely restoring the state pre-shutdown; remember there are other sources of entropy (including the disk activity needed to read the file), so writing the same 4096 bytes back to the urandom pool merely influences the urandom pool and the primary pool, resulting in entropy that is unpredictable from boot to boot.

Now, consider a LiveCD or a diskless workstation. Without the ability to introduce dynamic entropy from a previous session, the predictability increases a lot. If the computer had a hardware random number generator, we wouldn't have this problem. The hardware RNG could be queried directly, or it could be used to influence a pseudo RNG like the system Linux uses. But very few computers have hardware RNGs, and almost zero consumer-level computers do.

Or do they? Every computer actually has two hardware random number generators, which can be combined to get a stream of random numbers. They are the CPU itself and the real-time clock (RTC).

twuewand is a truerand implementation, first invented in 1995 by D. P. Mitchell. It relies on the fact that the CPU and RTC are physically separate clocked devices, and therefore time and work are not linked. twuewand's operation is very simple. It sets an alarm for sometime in the future (by default 4 milliseconds, as determined by the RTC), and then starts flipping a bit between 0 and 1 (work performed by the CPU). When the alarm is reached, the bit is taken. Voilà, random bit. It then repeats this process for as many bytes as needed.

This process produces a stream of truly random bits. An attacker can alter the amount of work performed by the CPU by introducing his own work during the same time period, but it still does not affect the output in a predictable way. However, this stream is still prone to bias. So after a certain number of bytes are collected, it is run through a cryptographic hash digest, by default SHA512, or MD5 if Digest::SHA is not installed. The hashed data is then output. This "whitens" the data, hopefully decreasing bias while retaining randomness.

twuewand could be used as a primary source of random data, but its primary purpose is intended to be an entropy pool seed. In Linux, you would execute:

twuewand $(cat /proc/sys/kernel/random/poolsize) >/dev/urandom

I wrote twuewand a few weeks ago when I first learned of truerand. truerand is an interesting concept, but it's actually almost never used in the real world anymore. The reason it was invented was to add another source of entropy to entropy pools, but the discovery of the benefits of saving pool data to reintroduce after reboot mostly made it unnecessary. But remember, this source is not available to LiveCDs and diskless workstations. I wrote twuewand for use by Finnix during startup, but hit a major snag. Namely, it's slow. Each bit takes a minimum of 4ms to generate, and that adds up. Generating 4096 bytes takes over 2 minutes. So I'm not going to have Finnix run it during startup, at least not for the full 4096 byte pool size. Perhaps 8 bytes by default, which will take a little over a quarter of a second. It's not as cryptographically secure as filling the entire pool, but it's better than nothing. Either way, twuewand will at least be available in the next version of Finnix if you desire to use it.

(If you don't get the "twuewand" name reference, go watch The Princess Bride.)

Read more
Ryan Finnie

ThinkPad X200s after one yearA year ago, I bought a Lenovo ThinkPad X200s. I boldly proclaimed that it's the best laptop I've ever used, much better than the X61, and combining all the features of a T60 and an X-series subnotebook.

So what's changed in the last year? Absolutely nothing. No plastic has chipped off, all of the LRF (little rubber feet) are still attached, the keyboard is still fully functional, the hinge is still solid, and it's just as tight as when I bought it. The only damage is the palmrest ThinkPad logo had separated its layers, leaving just the silver backing (which is still on amazingly tight). And this is despite the extra torture it received. I did a lot of traveling in the last year, and it's always come with me.

I've got a keeper.

(The Ubuntu sticker hasn't fared as well, but admittedly that is aftermarket.)

Read more