Canonical Voices

Posts tagged with 'random'

Dustin Kirkland

Tomorrow, February 19, 2014, I will be giving a presentation to the Capital of Texas chapter of ISSA, which will be the first public presentation of a new security feature that has just landed in Ubuntu Trusty (14.04 LTS) in the last 2 weeks -- doing a better job of seeding the pseudo random number generator in Ubuntu cloud images.  You can view my slides here (PDF), or you can read on below.  Enjoy!

Q: Why should I care about randomness? 

A: Because entropy is important!

  • Choosing hard-to-guess random keys provide the basis for all operating system security and privacy
    • SSL keys
    • SSH keys
    • GPG keys
    • /etc/shadow salts
    • TCP sequence numbers
    • UUIDs
    • dm-crypt keys
    • eCryptfs keys
  • Entropy is how your computer creates hard-to-guess random keys, and that's essential to the security of all of the above

Q: Where does entropy come from?

A: Hardware, typically.

  • Keyboards
  • Mouses
  • Interrupt requests
  • HDD seek timing
  • Network activity
  • Microphones
  • Web cams
  • Touch interfaces
  • WiFi/RF
  • TPM chips
  • RdRand
  • Entropy Keys
  • Pricey IBM crypto cards
  • Expensive RSA cards
  • USB lava lamps
  • Geiger Counters
  • Seismographs
  • Light/temperature sensors
  • And so on

Q: But what about virtual machines, in the cloud, where we have (almost) none of those things?

A: Pseudo random number generators are our only viable alternative.

  • In Linux, /dev/random and /dev/urandom are interfaces to the kernel’s entropy pool
    • Basically, endless streams of pseudo random bytes
  • Some utilities and most programming languages implement their own PRNGs
    • But they usually seed from /dev/random or /dev/urandom
  • Sometimes, virtio-rng is available, for hosts to feed guests entropy
    • But not always

Q: Are Linux PRNGs secure enough?

A: Yes, if they are properly seeded.

  • See random(4)
  • When a Linux system starts up without much operator interaction, the entropy pool may be in a fairly predictable state
  • This reduces the actual amount of noise in the entropy pool below the estimate
  • In order to counteract this effect, it helps to carry a random seed across shutdowns and boots
  • See /etc/init.d/urandom
dd if=/dev/urandom of=$SAVEDFILE bs=$POOLBYTES count=1 >/dev/null 2>&1


Q: And what exactly is a random seed?

A: Basically, its a small catalyst that primes the PRNG pump.

  • Let’s pretend the digits of Pi are our random number generator
  • The random seed would be a starting point, or “initialization vector”
  • e.g. Pick a number between 1 and 20
    • say, 18
  • Now start reading random numbers

  • Not bad...but if you always pick ‘18’...

XKCD on random numbers

RFC 1149.5 specifies 4 as the standard IEEE-vetted random number.

Q: So my OS generates an initial seed at first boot?

A: Yep, but computers are predictable, especially VMs.

  • Computers are inherently deterministic
    • And thus, bad at generating randomness
  • Real hardware can provide quality entropy
  • But virtual machines are basically clones of one another
    • ie, The Cloud
    • No keyboard or mouse
    • IRQ based hardware is emulated
    • Block devices are virtual and cached by hypervisor
    • RTC is shared
    • The initial random seed is sometimes part of the image, or otherwise chosen from a weak entropy pool

Dilbert on random numbers

Q: Surely you're just being paranoid about this, right?

A: I’m afraid not...

Analysis of the LRNG (2006)

  • Little prior documentation on Linux’s random number generator
  • Random bits are a limited resource
  • Very little entropy in embedded environments
  • OpenWRT was the case study
  • OS start up consists of a sequence of routine, predictable processes
  • Very little demonstrable entropy shortly after boot

Black Hat (2009)

  • iSec Partners designed a simple algorithm to attack cloud instance SSH keys
  • Picked up by Forbes
  • (2012)

  • Minding Your P’s and Q’s: Detection of Widespread Weak Keys in Network Devices
  • Comprehensive, Internet wide scan of public SSH host keys and TLS certificates
  • Insecure or poorly seeded RNGs in widespread use
    • 5.57% of TLS hosts and 9.60% of SSH hosts share public keys in a vulnerable manner
    • They were able to remotely obtain the RSA private keys of 0.50% of TLS hosts and 0.03% of SSH hosts because their public keys shared nontrivial common factors due to poor randomness
    • They were able to remotely obtain the DSA private keys for 1.03% of SSH hosts due to repeated signature non-randomness

Dual_EC_DRBG Backdoor (2013)

  • Dual Elliptic Curve Deterministic Random Bit Generator
  • Ratified NIST, ANSI, and ISO standard
  • Possible backdoor discovered in 2007
  • Bruce Schneier noted that it was “rather obvious”
  • Documents leaked by Snowden and published in the New York Times in September 2013 confirm that the NSA deliberately subverted the standard

Q: Ruh what can we do about it?

A: For starters, do a better job seeding our PRNGs.

  • Securely
  • With high quality, unpredictable data
  • More sources are better
  • As early as possible
  • And certainly before generating
  • SSH host keys
  • SSL certificates
  • Or any other critical system DNA
  • /etc/init.d/urandom “carries” a random seed across reboots, and ensures that the Linux PRNGs are seeded

Q: But how do we ensure that in cloud guests?

A: Run Ubuntu!

Sorry, shameless plug...

Q: And what is Ubuntu's solution?

A: Meet pollinate.

  • pollinate is a new security feature, that seeds the PRNG.
  • Introduced in Ubuntu 14.04 LTS cloud images
  • Upstart job
  • It automatically seeds the Linux PRNG as early as possible, and before SSH keys are generated
  • It’s GPLv3 free software
  • Simple shell script wrapper around curl
  • Fetches random seeds
  • From 1 or more entropy servers in a pool
  • Writes them into /dev/urandom

Q: What about the back end?

A: Introducing pollen.

  • pollen is an entropy-as-a-service implementation
  • Works over HTTP and/or HTTPS
  • Supports a challenge/response mechanism
  • Provides 512 bit (64 byte) random seeds
  • It’s AGPL free software
  • Implemented in golang
  • Less than 50 lines of code
  • Fast, efficient, scalable
  • Returns the (optional) challenge sha512sum
  • And 64 bytes of entropy

Q: Golang, did you say?  That sounds cool!

A: Indeed. Around 50 lines of code, cool!


Q: Is there a public entropy service available?

A: Hello,

  • Highly available pollen cluster
  • TLS/SSL encryption
  • Multiple physical servers
  • Behind a reverse proxy
  • Deployed and scaled with Juju
  • Multiple sources of hardware entropy
  • High network traffic is always stirring the pot
  • AGPL, so source code always available
  • Supported by Canonical
  • Ubuntu 14.04 LTS cloud instances run pollinate once, at first boot, before generating SSH keys

Q: But what if I don't necessarily trust Canonical?

A: Then use a different entropy service :-)

  • Deploy your own pollen
    • bzr branch lp:pollen
    • sudo apt-get install pollen
    • juju deploy pollen
  • Add your preferred server(s) to your $POOL
    • In /etc/default/pollinate
    • In your cloud-init user data
      • In progress
  • In fact, any URL works if you disable the challenge/response with pollinate -n|--no-challenge

Q: So does this increase the overall entropy on a system?

A: No, no, no, no, no!

  • pollinate seeds your PRNG, securely and properly and as early as possible
  • This improves the quality of all random numbers generated thereafter
  • pollen provides random seeds over HTTP and/or HTTPS connections
  • This information can be fed into your PRNG
  • The Linux kernel maintains a very conservative estimate of the number of bits of entropy available, in /proc/sys/kernel/random/entropy_avail
  • Note that neither pollen nor pollinate directly affect this quantity estimate!!!

Q: Why the challenge/response in the protocol?

A: Think of it like the Heisenberg Uncertainty Principle.

  • The pollinate challenge (via an HTTP POST submission) affects the pollen's PRNG state machine
  • pollinate can verify the response and ensure that the pollen server at least “did some work”
  • From the perspective of the pollen server administrator, all communications are “stirring the pot”
  • Numerous concurrent connections ensure a computationally complex and impossible to reproduce entropy state

Q: What if pollinate gets crappy or compromised or no random seeds?

A: Functionally, it’s no better or worse than it was without pollinate in the mix.

  • In fact, you can `dd if=/dev/zero of=/dev/random` if you like, without harming your entropy quality
    • All writes to the Linux PRNG are whitened with SHA1 and mixed into the entropy pool
    • Of course it doesn’t help, but it doesn’t hurt either
  • Your overall security is back to the same level it was when your cloud or virtual machine booted at an only slightly random initial state
  • Note the permissions on /dev/*random
    • crw-rw-rw- 1 root root 1, 8 Feb 10 15:50 /dev/random
    • crw-rw-rw- 1 root root 1, 9 Feb 10 15:50 /dev/urandom
  • It's a bummer of course, but there's no new compromise

Q: What about SSL compromises, or CA Man-in-the-Middle attacks?

A: We are mitigating that by bundling the public certificates in the client.

  • The pollinate package ships the public certificate of
    • /etc/pollinate/
    • And curl uses this certificate exclusively by default
  • If this really is your concern (and perhaps it should be!)
    • Add more URLs to the $POOL variable in /etc/default/pollinate
    • Put one of those behind your firewall
    • You simply need to ensure that at least one of those is outside of the control of your attackers

Q: What information gets logged by the pollen server?

A: The usual web server debug info.

  • The current timestamp
  • The incoming client IP/port
    • At, the client IP/port is actually filtered out by the load balancer
  • The browser user-agent string
  • Basically, the exact same information that Chrome/Firefox/Safari sends
  • You can override if you like in /etc/default/pollinate
  • The challenge/response, and the generated seed are never logged!
Feb 11 20:44:54 x230 2014-02-11T20:44:54-06:00 x230 pollen[28821] Server received challenge from [, pollinate/4.1-0ubuntu1 curl/7.32.0-1ubuntu1.3 Ubuntu/13.10 GNU/Linux/3.11.0-15-generic/x86_64] at [1392173094634146155]

Feb 11 20:44:54 x230 2014-02-11T20:44:54-06:00 x230 pollen[28821] Server sent response to [, pollinate/4.1-0ubuntu1 curl/7.32.0-1ubuntu1.3 Ubuntu/13.10 GNU/Linux/3.11.0-15-generic/x86_64] at [1392173094634191843]

Q: Have the code or design been audited?

A: Yes, but more feedback is welcome!

  • All of the source is available
  • Service design and hardware specs are available
  • The Ubuntu Security team has reviewed the design and implementation
  • All feedback has been incorporated
  • At least 3 different Linux security experts outside of Canonical have reviewed the design and/or implementation
    • All feedback has been incorporated

Q: Where can I find more information?

A: Read Up!

Stay safe out there!

Read more
Jussi Pakkanen

Steve Denning has held a wonderful presentation on how management should be be done in the 21st century. His main point is that instead of making money, the main goal of a company should be to delight its customers. He reasons that if the main goal is making money, it leads into a corporate structure that abhors innovation which makes the corporation vulnerable to agile startups.

The video is extremely recommended for everyone who deals with management in any way (including those who are being managed). The material in the presentation may seem very familiar to you either because it mirrors your own working experience or because you recognize the patterns it presents in other areas as well.

One well known piece of popular culture resonates very strongly with the presentation: the original Star Wars trilogy. The Galactic Empire is a good analogy of a large, established corporation that is being challenged by a small but nimble Rebel Alliance.

No need to plan, just do what you are told

Let us start our analysis by contrasting the meeting and decision making processes of the Empire and the Alliance. Probably the most famous meeting scene in the entire trilogy happens in Star Wars when Empire officials are discussing the rebellion and the lost Death Star plans. A simple overview of the meeting tells us that it will not be a successful one.

The meeting happens around one huge table. It is very probable that people can’t hear what participants on the other side of the table are saying. This is even more probable when you notice that most people are old generals who probably have poor hearing. They also look like they would rather be anywhere else than at the meeting. However the issues they are discussing are vital and the outcomes will shape the entire future of the Empire. More specifically the end of it.

As far as we can tell, the point of the meeting is to determine what to do should the Alliance really have a copy of the Death Star plans. All that everyone remembers of the meeting is Force choking. This is quite sad, because the issues raised are important ones. Lord Vader has not been able to find the lost tapes. Neither has he been able to find the rebel base. We also know that he has no real, workable plan to achieve these goals. Rather than work on the problem with a group of military experts Vader instead chooses to save his face by shooting the messenger.

The end result is that no actual work gets done. There are no contingency plans, no alternative approaches, nothing. The entire strategy of the Empire, the largest, most powerful organization in the galaxy, is It Will Work Because I Say It Will Work, now STFU and GBTW. It is also made painfully clear that questioning the choices of the Dear Leader is hazardous.

Compare this with the Rebel Alliance. All pieces of information and opinions are dealt with in a positive manner. When someone suspects that the targeting computer would not be able to hit the Death Star’s exhaust port, Luke does not attack him but rather gives a personal example showing how it is possible. When Han reports that a probable imperial probe droid has found them, the people in charge trust him and start working on evaluation. He could have gotten a response along the lines of do you have any idea how much resources we have spent to build this base, do you expect us to abandon all that based on a hunch. Even C-3PO, the lowest rung of the rebel ladder, can give his signal analysis results directly to the main command and his expertise is valued.

The Rebel Alliance is about solving problems, trust and agility. The Empire is about top-down leadership and management by mandates and shouting. We all know which one of them won in the end.

Failing with people: Management by Vader

If we view the Empire as a corporation then we can say that the Emperor is roughly the chairman of the board whereas Darth Vader is the CEO. He is in charge of the operational branches of the Empire. His decisions define the corporate culture. Let us examine his leadership using the battle of Hoth as a case study.

From a management point of view the largest conflict is between Darth Vader and Admiral Ozzel. Details on Ozzel’s military career are spotty but we can assume that he has gone through the Imperial Military/Space Navy/whatever Academy, gotten good grades, worked hard on his career and eventually has reached the rank of Admiral. Darth Vader, by comparison, is a whiny kid from a backwards desert planet with no formal military training and who has gotten his current rank through nepotism [1].

The conflict between these two comes to its peak when Vader feels that Ozzel has flown the fleet too close to the suspected rebel base. Let’s think about that for a second. The mission they are engaged in is basically a surprise attack. The goal is to catch the enemy unaware, crush them fast and prevent any escape attempts. This being the case flying in hyperspace as close as possible to the target planet and attacking immediately is the right thing to do. The alternative approach, and the one apparently preferred by Vader, would be to come out of hyperspace far from the planet and then fly slowly closer and attack. This strategy would have given the rebels ample time to jump every ship to hyperspace long before the Star Destroyers could have fired a single shot.

From a management point of view this single episode has many failures. First of all Vader did not trust his employees but rather started telling them what to do through micromanagement. Secondly even though he had a very clear vision on how the assault should be handled, he did not explain it to his underlings beforehand. He just magically assumed that they would do the right thing. Maybe he had forgotten that only Sith Lords have the ability to read people’s minds. Thirdly, when the fleet had left lightspeed, punishing Ozzel was the stupidest thing he could possibly do. The attack plan was in motion, nothing could change that. Any punishment should have happened only after the campaign. Summary executions in the middle of troop deployment only serves to weaken morale.

Kinda makes you wonder if the only reason Vader choked Ozzel was that he could be used as a scapegoat in case of failure.

This kind of behavior happens again and again through the trilogy. During the assault on Death Star, Vader explicitly tells professional TIE fighter pilots not to shoot at their targets, either because he wants all the glory to himself or because he thinks they are too stupid to hit anything. He orders the entire fleet inside an asteroid field causing billions of credits worth of damages and several thousand deaths. They could just have waited outside the asteroid field because it is well established that one can’t jump to hyperspace from inside the field. He also micromanages the search by demanding constant progress updates.

Come to think of it, almost every single management and executive decision Vader makes is wrong. He would have run the entire Empire down to the ground even if he hadn’t killed the Emperor. In corporations this kind of manager is unfortunately all too common. The higher up the chain he is, the more damage he can do. If he holds major amounts of stock, things are even worse because then he becomes really hard to get rid of.

Motivating the masses: the case of Stormtrooper apathy

One of the most ridiculed aspects of the Star Wars trilogy are the stormtroopers and especially their shooting accuracy. In corporations stormtroopers correspond to regular low level workers. The ones that actually get all the grunt work done. If you compare stormtroopers to rebel fighters, you find that rebel forces are consistently better. They shoot more accurately, have more imagination and just generally get things done better.

One might speculate that this is because all the top talent goes to the Rebel Alliance because it is the hot new cool stuff. In reality they are both recruiting from the same talent pool. Moreover it can be speculated that the best of the best of the best would go to the most glamorous and prestigious schools i.e. The Imperial Academy. Why would they instead join, effectively, a terrorist cell with a very low life expectancy unless they have a personal bone to pick with the Empire. [2]

The basic skill level of a stormtrooper is pretty much the same as the average Joe in the rebel alliance. And yet they perform terribly. As an example, let’s examine the scene in Star Wars just after our heroes have escaped from the trash compactor. They run into a group of seven stormtroopers. A few shots are fired and the entire group starts running away. If one examines the footage closely, at the time they start their retreat they can only see Han and bits of princess Leia. Luke and Chewie are behind a wall.

Think about that for a while. These are professional soldiers that come across some hippie and a girl. They are armed with deadly force, are specifically trained and have massive strength in numbers. Yet their instinctive decision is to run away. It’s kind of like having police officers who hide in their parent’s basements whenever they hear that a crime has been committed.

What could be the reason for this? The answer is simple: motivation. The common man inside the stormtrooper uniform probably does not care about the goals of the Empire. He just wants to get his paycheck and go home. What he really does not want is to get killed in any way. If you look at the behavior and motivation of stormtroopers throughout the series, doing everything possible not to get killed is pretty high on the list. Underachieving in their every day tasks is part of this because success means promotion, which means bigger probability of dealing with Vader, which in turn means higher probability of death by random Force choking than death in battle.

If this is the structure of your organization, the question is not why aren’t the workers performing well. The question is why would any sane person want to perform well.

There is one reason. We can deduce that by examining the cases where stormtroopers behave like an actual, efficient, deadly fighting force. There aren’t many of these, but let’s start with the beginning of Star Wars, the assault on princess Leia’s Corellian cruiser. The assault force knows what they are doing, shoot accurately, and take over the ship very efficiently.

There are a few other cases where this happens as well, but they are quite rare. There is one thing they have in common, though. The troops perform well only when Vader is personally overseeing them. This is classic management by fear. Every single troop knows that if they fail, they will get force choked to death. They might get choked even if they do just ok, just to set an example. So they really do their best.

The biggest problem with this approach is that it does not scale. Vader can’t be everywhere. Things work fine when he’s there. When he leaves, an entire legion of his best soldiers gets defeated by a dozen teddy bears with stone age technology.

Actually, let me take that back. The biggest problem is not the lack of scalability. The biggest problem is that Vader probably thinks that his troops are truly the best of the best. Why wouldn’t he? Whenever he is around, things work smoothly and efficiently. Who’s going to tell him that his so called elite troops are in fact complete garbage? Captain Needa?

This is the reason companies like Toyota and Google thrive. They care about their employees. They want them to participate in the decision making process. They want them to be part of the family, so to say, rather than being resources to be shifted around, shouted at and and summarily executed (though I don’t think any Fortune 500 company does executions at the present time).

The meaning of (a company’s) life

The main thesis of Steve Denning’s presentation is that the common view that a company’s purpose is to make money is flawed. Instead they should be delighting their customers. Making money is a result, not the goal. With that in mind, let’s ask a simple question.

What is the ultimate purpose of the Empire?

We hear very little about their goals on health care or education. As far as we can tell, the Empire is only the manifestation of the Emperor’s lust for power. He doesn’t care about the people. His only interest is in the power trip he gets from bossing them around. Just like certain corporations see their customers only as sponges to squeeze as much money out of as possible.

There are consequences.

When Luke talks to Obi-Wan for the first time he says “It’s not that I like the Empire, I hate it. But there’s nothing I can do about it now.” His delivery seems to indicate that this is a common attitude towards the Empire.

Remind of any corporations you know?

If we accept the special editions as canon, once the Emperor died, people started spontaneously partying in the streets, knocking down statues and shooting fireworks. After the tipping point everyone dropped the Empire like it was going out of fashion. One imagines that even people high up on the Empire’s chain of command would go around stating how they have always secretly supported the goals of the Rebellion.

The world is full of companies that have used their dominant position to extract money with inferior products. They have focused on cost cutting and profit maximisation rather than improving their customers’ lives. And they have been successful for a while. Once there has been a competitor that do cares about these things, the dominant player has usually collapsed. For an example see what has happened to Nokia after the release of the iPhone.

The only protection against collapse is to make your customers consistently happy. Should someone come out tomorrow with a new magical superphone that is up to 90% better than iPhone. Would current iPhone users switch out in masses? No they would not, because they have bought their current phone because it was the best for them, the one they really wanted. Not because it was the “crappy-but-only-possible” choice.

If your company is producing products of the latter type, your days are already numbered, but you just don’t know it yet. Just when you think you are at the height of your power, someone will grab you without warning and throw you over a railing. Most likely you will blame your failure on them. But you are wrong. You have brought your downfall on yourself.

Also, you are dead.


[1] Assuming that the Force Darth Sidious uses to inseminate Shmi Skywalker comes from himself. Somehow. In a way I don’t really want to know.

[2] The prequels seem to indicate that stormtroopers are clones. However that is probably not the case anymore in the time frame of the original trilogy. The original clones all spoke with the same voice. Stormtroopers speak with different voices. There are also variations in size and behavior. If they were clones, i.e. dispensable cannon fodder, it would make even less sense for them to be concerned about self-preservation.

Read more
Jussi Pakkanen

The main point of open source is that anyone can send patches to improve projects. This, of course, is very damaging to the Super Ego of the head Cowboy Coder in charge. Usually this means that he has to read patch, analyze it, understand it, and then write a meaningful rejection email.

Or you could just use one of the strategies below. They give you tools to reject any patch with ease.

The Critical Resource

Find any increase in resources (no matter how tiny or contrived) and claim that to be a the most scarce thing in the universe. Then reject due to increased usage.

A sample discussion might go something like this:

- Here’s a patch that adds a cache for recent results making the expensive operation 300% faster.

- This causes an increase in memory usage which is unacceptable.

- The current footprint is 50 MB, this cache only adds less than 10k and the common target machine running this app has 2GB of memory.

- You are too stupid to understand memory optimisation. Go away.

 The suffering minority

When faced with a patch that makes things better for 99.9% of the cases and slightly worse for the rest, focus only on the 0.01%. Never comment on the majority. Your replies must only ever discuss the one group you (pretend to) care about.

- I have invented this thing called the auto-mobile. This makes it easier for factory workers to come to work every morning.

- But what about those that live right next to the factory? Requiring them to purchase and maintain auto-mobiles is a totally unacceptable burden.

- No-one is forcing anyone. Every employer is free to obtain their own auto-mobiles if they so choose.

- SILENCE! I will not have you repress my workers!

 The Not Good  Enough

Think up a performance requirement that the new code does not fulfill. Reject. If the submitter makes a new patch which does meet the requirement, just make it stricter until they give up.

- This patch drops the average time from 100 ms to 30 ms.

- We have a hard requirement that the operation must take only 10 ms. This patch is too slow, so rejecting.

- But the current code does not reach that either, and this patch gets us closer to the requirement.

- No! Not fast enough! Not going in.

 The Prevents Portability

Find any advanced feature. Reject based this feature not being widely available and thus increases the maintenance burden.

- Here is a patch to fix issue foo.

- This patch uses compiler feature bar, which is not always available.

- It has been available in every single compiler in the world since 1987.

- And what if we need to compile with a compiler from 1986? What then, mr smartypants? Hmmm?

The Does not Cure World Hunger

This approach judges the patch not on what actually is, but rather what it is not. Think of a requirement, no matter how crazy or irrelevant, and reject.

- This patch will speed up email processing by 4%.

- Does it prevent every spammer in the world from sending spam, even from machines not running our software?

- No.

- How dare you waste my time with this kind of useless-in-the-grand-scheme-of-things patch!

The absolute silence

This is arguably the easiest. Never, ever reply to any patches you don’t care about. Eventually the submitter gives up and goes away all by himself.

Read more
Jussi Pakkanen

The Ubuntu Advantage™

As a touch developer people often ask me what makes our touch stack better than the rest. As exhibit A I present this image of one of our competitor’s products.

This was found in Orlando’s Hard Rock Cafe.

Read more