Canonical Voices

What Alex Chiang talks about

Posts tagged with 'tech tuesday'

alex

I wanted somewhere easy to dump technical notes that weren’t really suitable for this blog. I wanted a static HTML generator type of blog because the place to dump my notes (people.canonical.com) isn’t really set up to run anything complex for a multitude of reasons, such as security.

I also didn’t want to just do it 1990s style and throw up plain ASCII README files (the way I used to) because I envision embedding images and possibly movies in my notes here. At the same time, the closer I can get to a README the better, and so that seems to imply markdown.

After a brief fling with blacksmith where absolutely nothing worked because of a magical web 2.0 fix-everything-but-the-zillions-of-pages-of-existing-docs rewrite, I wiped the blood and puke from my mouth and settled on octopress.

Octopress was much better, but it was still a struggle. It’s a strange state of affairs that deploying wordpress on a hosted site is actually *less* difficult than configuring what *should* be a simple static HTML generator. Oh well.

Here are some notes to make life easier for the next person to come along.

Deploying to a subdir, fully explained
One wrinkle of hosting on a shared server using Apache conventions is that your filesystem path for hosting files will probably get rewritten by the web server and displayed differently.

That is:

    unix filesystem path                 =>  address displayed in url bar
    /home/achiang/public_html/technotes  =>  http://people.canonical.com/~achiang/technotes

The subdir deployment docs talk about how to do this, but the only way I could get it to work is by issuing: rake set_root_dir[~achiang/technotes] first. So the proper sequence is:

rake set_root_dir[~achiang/technotes]

vi Rakefile	# and change:
	-ssh_user       = "user@domain.com"
	+ssh_user       = "achiang@people.canonical.com"
	-document_root  = "~/website.com/"
	+document_root  = "~/public_html/technotes"

vi _config.yml	# and change:
	-url: http://yoursite.com
	+url: http://people.canonical.com/~achiang/technotes

rake install
rake generate
rake deploy	# assuming you've setup rsync deploy properly

Once you’ve tested this is working, then optionally set rsync_delete = true. But don’t make the same mistake I made and set that option too soon, or else you will delete files you didn’t want to delete.

Finally, once you have this working, the test address for your local machine using the `rake preview` command is http://localhost:4000/~achiang/technotes.

Video tag gotchas
One nice feature of Octopress is the video plugin it uses to allow embeddable H.264 movies. I discovered that unlike the image tag which apparently allows for local paths to images, the video tag seems to require an actual URL starting with http://.

Therefore:

    {% video /images/movie.mp4 %}	# BROKEN!

However, this works:

    {% video http://people.canonical.com/~achiang/images/movie.mp4 %}

I’ll work up a patch for this at some point.

Misc gotchas
The final thing I tripped over was https://github.com/imathis/octopress/pull/1438.

I’ll update here if upstream takes the patch, but if not, then you’ll want the one-liner in the pull request above.

Summary
After the initial fiddly bits, Octopress is good enough. I can efficiently write technical content using $EDITOR, the output looks modern and stylish, and it all works on a fairly constrained, bog-standard Apache install without opening any security holes in my company’s infrastructure.

Read more
alex

Today I spent a little bit of time playing with sbuild and after an hour or so, decided I hated it. Tried to figure out why people recommend it, and it seems like the best answer is, “it’s the closest to what the buildds use”. I guess that’s a fair answer, but out of the box, sbuild feels clunky to me.

Luckily, Michael Terry is jawesome and wrote these really great pbuilder wrapper scripts and now they’ve landed in Quantal.

If you want to know why I ? them so, check out my contra answer on askubuntu:

Why use sbuild over pbuilder?

And if you want to speed up your pbuilder even moAR, then check out PbuilderHowto.

Maybe I don’t know what I’m doing so if you have tips or corrections, add them over there. If you see mterry out somewhere, buy him a beer!

Read more
alex

After wandering around for a bit, I’ve settled back in San Francisco on a more or less permanent basis. Part of the moving process was finding an ISP and it seems like Comcast is the best option (for my situation). I signed up for their standard residential service, and remote teleworking continued on quite merrily… except for one tiny wart.

We use Google Plus hangouts quite extensively on my team including a daily standup with attendance that hovers between 5 to 10 people. The first time I tried a hangout with my new Comcast service, it was unusable with extreme lag everywhere, connection timeouts, and general unhappiness.

I had a strong hunch that I was suffering from bufferbloat, and a quick ping test confirmed it (more on that later). Obviously I wanted to fix the problem, but there is a lot of text to digest for someone that just wants to make the problem go away.

After a bit of irc whingeing and generous help from people smarter than me, here are my bufferbloat notes for the impatient.

background
Bufferbloat is a complex topic, go read the wiki page for excruciating detail.

But the basic conceptual outline is:

  • a too large buffer on your upstream may cause latency for sensitive applications like video chat
  • you must manage your upstream bandwidth to reduce latency (which typically means you intentionally reduce upstream bandwidth)
  • use QoS in your router to globally reduce upstream bandwidth (not for traffic shaping!)

diagnosis
Ensure your internet connection is idle. Then, start pinging google.com. Observe the “time” field, which will give you a value in ms. Watch this long enough to get an intuitive feel for what is a normal amount of latency on your link. For me, it hovered consistently around 20ms, with some intermittent spikes. You don’t need to be exact. If the values swing wildly, then you’ve got other problems that need to be fixed first. Stop reading this blog and call your ISP.

While the ping is running, visit http://testmy.net/upload and kick off a large upload, say 15MB or more.

If your ping times increase by an order of magnitude and stay there (like mine did to around 300ms), then you have bufferbloat.

This isn’t as rigorous as setting up smokeping and making pretty graphs, but trust me, it’s a lot faster and way easier. Thanks to Alex Williamson for this tip.

mitigation
You will need a router that can do QoS.

The easiest solution is to spend $100 and buy a Netgear WNDR3700 which is capable of running CeroWRT. Get that going and presumably you’re done, although I can’t verify it since I am el cheapo.

I didn’t want to spend $100 and I had an old Linksys WRT54GL lying around. Install Tomato onto it. (Big thanks to Paul Bame for helping me (remotely!!) recover a semi-bricked router.) Now it’s time to tune QoS.

In the Tomato admin interface, navigate to QoS => Basic Settings. Check the “Enable QoS” box and for the “Default class” dropdown list, change it to “highest”.

Figure out your maximum upload speed. You should be able to obtain this number after a few upload tests at testmy.net that you did in the previous step. Enter your max upload speed into the “Outbound Rate / Limit” => “Max Bandwidth” field. Make sure you use the right units, kbits/s please!

Finally, in the “Highest” QoS setting under Outbound, set your lower and upper bounds. I started with 50% as a lower bound and 60% as an upper bound.

Put a large fake number in for “Inbound Limit” and change all the settings there to “None”. These settings don’t seem to affect latency.

Click “save” at the bottom of the page — you do not need to reboot your router.

Re-run the google.com ping test + large upload test at testmy.net. Your ping times under load should remain relatively unchanged vs. an idle line. Congrats, you’ve solved your bufferbloat problem to 80%.

Update (7/29/2012): Thanks to John Taggart for pointing out a more rigorous page on QoS tuning for tomato.

Now you can experiment with increasing the lower and upper bounds of your QoS settings to get more upstream bandwidth. As always, make a change, save, re-run the ping + upload test, and check the results. Remember, the goal is to keep latency under load about equal to what it is on an idle line.

Now your colleagues will thank you for the increased smoothness of your video chats, although remembering to brush your teeth and put pants on is the “last mile” problem I can’t solve for you.

Read more