Canonical Voices

What Ryan Finnie talks about

Ryan Finnie

Mobile Rickroll Appliance 6.0

Last week, I attended Defcon 19 in Las Vegas. This year was a pretty good year. It was held at the Rio which was not without problems, but the hotel rooms were nice, the conference space was much larger than any previous year, and overall it was a much better experience that the Riviera from the last few years.

At the last minute before I left for Vegas, I found and revived the Mobile Rickroll Appliance 6.0. The MRRA 6.0 is something I created in 2008, modifying a WRT54GS running OpenWRT with some custom iptables and dnsmasq configuration to make a self-contained Rickrolling platform. The result was hilarious:

(Since then, the sped-up Rickroll video has been replaced with a normal speed one, but otherwise the functionality is exactly the same.)

This year I re-flashed my WRT54GS and took it to Defcon. Before the conference began, I headed to Fry's and picked up a 7.2 Ah 12v battery (the narrow kind used in some UPSes) and various cables and adapters to go from alligator clips on the battery side to the barrel plug on the WRT. The result was a truly mobile (though heavy) Rickroll appliance that could be stored in my backpack.

That Friday morning, I took it to the opening ceremony and talks. At home, the ESSID was "Common Area", but for Defcon I named it "cellshare", to make it look like someone had mobile hotspot enabled on their phone. The MRRA 6.0 is configured so when someone is Rickroll'd, the big front status light will flash for 5 seconds, then turn solid orange, giving me a quick status indicator. This is done by a web bug CGI on the HTML page, which also logs the time since boot (the WRT doesn't have a battery-saved RTC and the MRRA 6.0 doesn't have internet access, so real time is not possible), the HTTP user agent and the HTTP referer (which in this case is what the victim was originally trying to go to).

Over the next 3 hours, I managed to Rickroll over 100 people! Most victims were iPhone users. On the plus side, the way iOS checks for captive portal support, the Rickroll web page will pop up immediately upon association with the AP; they don't even have to open Safari. The downside is the MRRA 6.0 uses FLV, and the iPhone doesn't support Flash, so all they see is a big box and a message saying they've been Rickroll'd. Future releases may use some sort of MP4 solution.

Of the remaining victims, most were laptop users. Most of them were Windows 7 running IE, though a large minority were OS X (you tend to see a lot of Macs at Defcon). Of the sites people tried to visit, cnn.com was the most popular, followed by google.com, facebook.com and apple.com.

Sadly, the battery pack only worked for just over 3 hours. The battery had a capacity of 7.2 Ah @ 12v, and the WRT lists a draw of 1000 mA, so I was expecting a life of at least 7.2 hours (hopefully longer, since I was guessing the 1000 mA figure was max draw, not typical). After 3 hours, the WRT just stopped powering on. The small LED on the barrel plug adapter was still lighting up, but the WRT just would not work unless I was using the wall wart. I suspect it was one of two things. I made no attempt to charge the sealed lead acid battery after buying it from Fry's, so I have no idea what its current capacity was. That possibly combined with a small voltage drop as the battery discharges may have created a voltage that was low enough the WRT didn't like, but still enough to power the barrel plug adapter's status LED. Despite this being Defcon, nobody in our group had a multimeter handy.

Still, I consider the project a success. If you were at any of the first three talks on Friday and got Rickroll'd after associating with "cellshare", I'd like to hear from you!

This weekend I took my original code, modified it for current OpenWRT (the OpenWRT on my WRT was from 2008), re-rendered the Rickroll video to fit on a 4MB WRT device (the original code would only fit on an 8MB WRT -- WRTSL54GS or early WRT54GD -- which are uncommon), and packaged it up. That's right, now you can build your very own Mobile Rickroll Appliance 6.0! Download the tarball, extract, and read the README, which will give you all the info you need to get set up.

Read more
Ryan Finnie

rsnapshot

At my former employer, we had an in-house backup system for backing up Unix servers. It was called speede, and it offered a much better way of maintaining disaster recovery backups than other methods such as tape. Multiple snapshots were taken, by taking the old snapshot, coping it to the new snapshot using hardlinks, and running rsync from the backed up host to the new snapshot. Since rsync works (by default) by copying changed files to a temp file, deleting the old destination file and moving the temp file into place, it cleanly breaks the hardlink without affecting the data in the old inode. The net result is you have multiple snapshot directories that look like completely independent directory trees, but are space efficient since the majority of the files share the same inodes between snapshots. If you have 200MB of data in snapshot 1 and 5MB of files change between snapshot 1 and snapshot 2, only 205MB is stored on disk.

(Apple uses the same type of process for Time Machine, by the way.)

speede was started sometime in 2002. I started at the company in 2004, and took over development of it in early 2005. Over the next 6 years, a lot was added, such as run concurrency, more granular options, etc. It was an awesome system. We even planned on releasing it in 2009 as open source, but company politics put an end to that in the 11th hour (we were in the process of getting acquired by another company at the time).

After that, I decided to do a semi-cleanroom re-coding of it and instead releasing that, calling it bahlgs ("Backups Are Hard, Let's Go Shopping!"). Unfortunately, that became one of my "90% done, just need to finish the other 90%" projects, and it was never released.

I had heard about rsnapshot awhile ago, which did nearly the same thing as speede. But I never looked much into it, since we had an elegant system that did exactly what we needed.

I ran an early dev copy of bahlgs on my home router / server, backing up a few home servers, my colo box, my Linode, etc. Today I upgraded my home server, using it as an excuse to reinstall the OS at the same time. Rather than setting up that copy of bahlgs again, I decided to take a better look at rsnapshot.

It seems to be pretty decent, and would be capable of functioning as a replacement for my home backup system. But at the same time I was thinking back to my previous employer, where I maintained a datacenter of approximately 150 servers. And I realized that rsnapshot wouldn't have worked for that use:

speede created snapshot directories in the format snapshot-YYYYMMDD-HHMM, with a symlink from current to the latest snapshot. rsnapshot uses a format like daily.0, where "daily" is the type of snapshot and "0" is an incrementing integer, with 0 always being the most current. That avoids the need for a "current" symlink, but makes it harder to see what dates correspond to which snapshots.

speede had hardcoded logic for the last N daily backups, plus the first snapshot of the month. It's something I wanted to make more configurable, but it served our backup needs. rsnapshot allows for a configurable number of hourly/daily/weekly/monthly snapshots, which is more configurable, but the runs are not connected to each other. That is, if I choose to do daily and monthly snapshots, the backup is run twice on the day the monthly snapshots are run.

speede had concurrency support, allowing for a maximum number of concurrent rsyncs (6 by default), and hosts could be placed into concurrency groups with separate concurrency limits for them. For example, limit group "vmhost5" to 2 concurrent rsyncs, since if all 6 runs happened to be against guests of VM Host #5, it would impact all guests on the host.

rsnapshot seems to have no concurrency support. That would be a killer for my old employer, where we had three backup servers, each running 6 concurrent rsyncs, backing up about 150 servers, and it would still take about 8 hours each night. This can be partially mitigated in rsnapshot by using multiple configuration files and dividing the servers up into multiple cron runs done concurrently, but speede was smarter and used a queue system so one large backup wouldn't hold up others.

I'm definitely not putting down rsnapshot. It seems very useful, and even has features that would have been nice for speede, except I never got around to coding them into speede myself (such as pre/post per-backup scripts). But again, I'm not lamenting not taking a look at rsnapshot when I was with my old employer, since the lack of concurrency support would have been a deal breaker.

Both speede and rsnapshot are coded in Perl, so I may look into adding the "missing" functionality myself, and submitting it upstream. But for now, I think I'll install rsnapshot at home.

Read more
Ryan Finnie

I am not a programmer by trade. I'm a system administrator, which involves knowing just enough about ten thousand different technologies to get by. A literal Jack of all trades, master of "none". (Well, few; of course I have specialties.)

That is not to say I don't know programming languages. As a quick list, I'm competent with Perl, PHP, Python, Javascript and Bourne shell, and know enough about C and Java to be able to add code here and there, but not be able to write a program from scratch. For the most part I can code well, but wouldn't want to do it for a living. (Which I tried back in 2000. Hated it, looking back.)

Now this post isn't all ego stroking (only about 90%). Being active in Free software, I often contribute patches to software projects. Usually one-line patches here and there, which is useful but not necessarily exciting. However, there are the occasional times where submitting a relatively small patch makes a huge improvement, and those are the sorts of contributions that really stick with me. Submitting these types of patches (and getting positive feedback from the developers or other users) honestly has a euphoria effect that lasts a long time.

A few examples:

* Last weekend I submitted a patch for Minecraft Overviewer, which is a map renderer that makes isometric tiles out of Minecraft worlds, and wraps them in a Google Maps interface. The problem is Minecraft has its own coordinates system that many users rely upon (X/Y/Z, where oddly Y is altitude), whereas Gmaps uses latitude/longitude coordinates. Overviewer shoehorned that into the Gmaps style by defining the world as a latitude and longitude between -1.0 and 1.0. I wrote some code that, while internally everything still uses Gmaps-style lat/lng, the user interface shows/accepts Minecraft-style X/Y/Z coordinates everywhere.

* A few Debian versions ago, they switched the installation CD from a command line-based bootloader (straight syslinux) to a menu (syslinux menu.c32). However, menu.c32 didn't support conditional menu item selection based on 32/64 bit processors, which was lamented in the prerelease post. I saw this, thought "I know syslinux pretty well, that would probably be about a 10 minute patch!" Of course, 10 minutes turned into a few hours, but it was still relatively easy. I submitted a bug report with the patch, it was accepted into the syslinux package within an hour, and pulled into debian-installer an hour after that, ready for the night's daily builds. To date, I don't think I've ever seen Debian move that fast.

* The default functionality of Spamassassin is, when a spam is found, to wrap the original message in an RFC822 MIME container, add the spam report, and let it continue on. This is so the original message is perfectly preserved and can be extracted by many mail readers in case of false positives, submitting to an RBL, etc. That functionality was mine. In fact, I'm toward the top of the CREDITS file under "Major contributions", and tend to get the occasional mail from people who say "can you help me install?" or "how dare you block my perfectly legit penis enlargement mail!" by people who randomly mail the addresses they find in that file.

* And of course there's Finnix, which, while not a small patch contribution, produces the same effect. I'm always happy when I hear someone who uses it to get out of a tight situation, or lets them do their job easier.

Read more
Ryan Finnie

I forward my mail from my personal server to Gmail. For the most part, it would work okay, but SPF is a problem. Specifically, SPF completely breaks legitimate email routing if "properly" configured. And since we're talking about the people emailing you being the ones "properly" configured, there's little you can do about it. (Have I mentioned how much I hate SPF?) My mail server receives mail, then routes it to Gmail. The problem is when the original sender's SPF policy is set to strict, Gmail receives it, looks up the SPF policy, notices it's coming from my server instead of the permitted sender server, and marks it as spam.

In hindsight, the creators of SPF "planned" for this, and created SRS, which effectively rewrites the SMTP Envelope FROM (but not From:), from, for example, user@sender.tld to SRS0=ABCD=EF=sender.tld=user@relay.tld.

I implemented this about a month ago on my mail server, and things seemed to be going well at first. But over time, Gmail started marking more and more legitimate mail as spam. Probably due to their servers not being aware of SRS, and seeing a single domain (relay.tld) sending a bunching of mail with From: addresses like facebook.com, etc. In the end, nearly 100% of mail was being marked as spam, so I switched back to simple relaying.

Now, what many email providers do is support a procedure with a name like "inbound gateway", "trusted relay", etc, where you specify which IP addresses are permitted to relay mail to the final destination. This isn't a "whitelist", it basically tells the mail server, "for mail coming from this IP address, do not consider the connecting IP address at all in relation to spam determination, good or bad". The problem is, only some Google mail services provide this:

Free Gmail: NO
Free Google Apps Gmail: NO
Paid Google Apps Gmail: YES (from the Apps control panel: Service Settings, Email, Inbound gateway)
Postini: YES (but requires a call to Postini, and getting them to approve the IP globally, not just for your service)

So I'm considering switching to the paid Google Apps, just to get this behind me. I like Gmail's interface, and more importantly, I'm almost totally reliant on Google for spam filtering, as no other service seems to do as well (barring the problems described here), considering I've had my primary email address for nearly 15 years, and it receives literally thousands of spam per day.

Read more
Ryan Finnie

2ping 1.1 has been released, with a few small changes:

  • Host processing delays sent by the peer are no longer considered when calculating RTT
  • Changed ID expiration (for which no courtesty was received) time from 10 minutes to 2 minutes
  • Manpage fix: correct UDP port number listed
  • Added an RPM spec file

2ping is a bi-directional ping utility. It uses 3-way pings (akin to TCP SYN, SYN/ACK, ACK) and after-the-fact state comparison between a 2ping listener and a 2ping client to determine which direction packet loss occurs.

Read more
Ryan Finnie

Firefox 4.0 was released this week, and while the Hampr extension is technically compatible with it out of the box, it's not ideal. Firefox 4.0 for Windows defaults to hiding the menu bar in favor of an "app menu", a dropdown next to the tab bar. Because the Hampr extension's default mode (on Windows and Linux) is to replace the Bookmarks menu with the Hampr menu, the user will never see the Hampr interface unless they've changed the default Firefox behavior and are showing the main menu bar.

I will go into some more detail about the tech below, but for the impatient, there are one of two things you can do immediately to fix this:

A) Enable the main menu bar.

  1. Click "Firefox", hover over "Options", and click "Menu Bar".
  2. Hampr will be waiting for you on the main menu bar, no restart necessary.

B) Switch the Hampr extension to "Toolbar button" mode.

  1. Click "Firefox", then "Add-ons", then "Extensions". Click "Options" on the line for Hampr.
  2. For "Display mode", select "Toolbar button". Click "OK" for the restart warning, then "OK' for the options dialog.
  3. Click "Firefox", hover over "Options", and click "Toolbar layout".
  4. Drag the Hampr button (which looks like a yellow folder) to a convenient location, for example next to the address/search bar.
  5. Click "Done" and restart Firefox.

Hampr has two more modes, "Add to main menu bar" and "Add to bookmarks menu", but unfortunately both are made useless by Firefox hiding the main menu bar. The first is obvious, but "Add to bookmarks menu" would seem useful, since the Firefox bookmarks menu is shown in four locations, depending on layout:

  • From the main menu bar, if the main menu bar is visible.
  • From the app menu, if the main menu bar is hidden.
  • From the bookmarks dropdown button on the address bar, if the main menu bar is hidden.
  • From the bookmarks toolbar, if the bookmarks toolbar is visible and the main menu bar is hidden.

The problem is, while they all look identical, they're actually four completely separate copies as far as Firefox is concerned. The Hampr extension only knows how to hook into the bookmarks menu of the main menu bar, so "Add to bookmarks menu" only displays on the main menu bar's bookmarks menu.

I will look into adding new functionality to hook into these menus, but it will probably take a lot more code, so it's not going to be immediately. In the meantime, enabling the main menu bar is probably the easiest choice if you are an avid Hampr user.

(Firefox 4.0 for Linux doesn't seem to default to hiding the main menu bar. And due to severe technical restrictions with OS X (namely, a menu bar dropdown cannot be changed while it is open), main menu bar integration was never available for Mac to begin with.)

Read more