Comparing build speeds of different code bases

Some C++ code bases seem to compile much more slowly than others. It is hard to compare them directly because they very often have different sizes. Thus it is hard to encourage people to work on speed because there are no hard numbers to back up your claims.

To get around this I wrote a very simple compile time measurer. The code is available here. The basic idea is quite simple: provide a compiler wrapper that measures the duration of each compiler invocation and the amount of lines (including comments, empty lines etc) the source file had. Usage is quite simple. First you configure your code base.

CC='/path/to/smcc.py gcc' CXX='/path/to/smcc.py g++' configure_command

Then you compile it.

SMCC_FILE=/path/to/somewhere/sm_times.txt compile_command

Finally you run the analyzer script on the result file.

sm-analyze.py /path/to/sm_times.txt

The end result is the average amount of lines compiled per second as well as per-file compile speed sorted from slowest to fastest.

I ran this on a couple of code bases and here are the results. The test machine was a i7 with 16GB of ram using eight parallel compile processes. Unoptimized debug configuration was always chosen.

                   avg   worst     best
Libcolumbus     287.79   48.77  2015.60
Mediascanner     52.93    5.64   325.55
Mir             163.72   10.06 17062.36
Lucene++         65.53    7.57   874.88
Unity            45.76    1.86  1016.51
Clang           238.31    1.51 20177.09
Chromium        244.60    1.28 49037.79

For comparison I also measured a plain C code base.

                   avg   worst     best
GLib           4084.86  101.82 19900.18

We can see that C++ compiles quite a lot slower than plain C. The main interesting thing is that C++ compilation speed can change an order of magnitude between projects. The fastest is libcolumbus, which has been designed from the ground up to be fast to compile.

What we can deduce from this experiment is that C++ compilation speed is a feature of the code base, not so much of the language or compiler. It also means that if your code base is a slow one, it is possible to make it compile up to 10 times faster without any external help. The tools to do it are simple: minimizing interdependencies and external deps. This is one of those things that is easy to do when starting anew but hard to retrofit to code bases that resemble a bowl of ramen. The payoff, however, is undeniable.

Zero overhead pimpl

Pimpl is a common idiom in C++. It means hiding the implementation details of a class with a construct that looks like this:

class pimpl;

class Thing {
private:
  pimpl *p:
public:
 ...
};

This cuts down on compilation time because you don’t have to #include all headers required for the implementation of this class. The downside is that p needs to be dynamically allocated in the constructor, which means a call to new. For often constructed objects this can be slow and lead to memory fragmentation.

Getting rid of the allocation

It turns out that you can get rid of the dynamic allocation with a little trickery. The basic approach is to preserve space in the parent object with, say, a char array. We can then construct the pimpl object there with placement new and delete it by calling the destructor.

A header file for this kind of a class looks something like this:

#ifndef PIMPLDEMO_H
#define PIMPLDEMO_H

#define IMPL_SIZE 24

class PimplDemo {
private:
  char data[IMPL_SIZE];

 public:
  PimplDemo();
  ~PimplDemo();

  int getNumber() const;
};

#endif

IMPL_SIZE is the size of the pimpl object. It needs to be manually determined. Note that the size may be different on different platforms.

The corresponding implementation looks like this.

#include"pimpldemo.h"
#include<vector>

using namespace std;

class priv {
public:
  vector<int> foo;
};

#define P_DEF priv *p = reinterpret_cast<priv*>(data)
#define P_CONST_DEF const priv *p = reinterpret_cast<const priv*>(data)

PimplDemo::PimplDemo() {
  static_assert(sizeof(priv) == sizeof(data), "Pimpl array has wrong size.");
  P_DEF;
  new(p) priv;
  p->foo.push_back(42); // Just for show.
}

PimplDemo::~PimplDemo() {
  P_DEF;
  p->~priv();
}

int PimplDemo::getNumber() const {
  P_CONST_DEF;
  return (int)p->foo.size();
}

Here we define two macros that create a variable for accessing the pimpl. At this point we can use it just as if were defined in the traditional way. Note the static assert that checks, at compile time, that the space we have reserved for the pimpl is the same as what the pimpl actually requires.

We can test that it works with a sample application.

#include<cstdio>
#include<vector>
#include"pimpldemo.h"

int main(int argc, char **argv) {
  PimplDemo p;
  printf("Should be 1: %d\n", p.getNumber());
  return 0;
}

The output is 1 as we would expect. The program is also Valgrind clean so it works just the way we want it to.

When should I use this technique?

Never!

Well, ok, never is probably a bit too strong. However this technique should be used very sparingly. Most of the time the new call is insignificant. The downside of this approach is that it adds complexity to the code. You also have to keep the backing array size up to date as you change the contents of the pimpl.

You should only use this approach if you have an object in the hot path of your application and you really need to squeeze the last bit of efficiency out of your code. As a rough guide only about 1 of every 100 classes should ever need this. And do remember to measure the difference before and after. If there is no noticeable improvement, don’t do it.

On style checkers and warnings as errors

The quest for software quality has given us lots of new tools: new compiler warnings, making the compiler treat all warnings as errors, style checkers, static analyzers and the like. These are all good things to have. Sooner or later in a project someone will decide to make them mandatory. This is fine as well, and the reason we have continuous integration servers.

Still, some people might, at times, propose MRs that cause CI to emit errors. The issues are fixed, some time is lost when doing this but on the whole it is no big deal. But then someone comes to the conclusion that these checks should be mandatory on every build so the errors never get to the CI server. Having all builds pristine all the time is great, the reasoning goes, because then errors are found as soon as possible. This is as per the agile manifesto and universally a good thing to have.

Except that it is not. It is terrible! It is a massive drain on productivity and the kind of thing that makes people hate their job and all things related to it.

This is a strong and somewhat counter-intuitive statement. Let’s explore it with an example. Suppose we have this simple snippet of code.

  x = this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);

Now let’s suppose we have a bug somewhere. As part of the debugging cycle we would like to check what would happen if x had the value 3 instead of whatever value the function returns. The simple way to check is to change the code like this.

  x = this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);
  x = 3;

This does not give you the result you want. Instead you get a compile/style/whatever checker error. Why? Because you assign to variable x twice without using the first value for anything. This is called a dead assignment. It may cause latent bugs, so the checker issues an error halting the build.

Fair enough, let’s do this then.

  this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);
  x = 3;

This won’t work either. The code is ignoring the return value of the function, which is an error (in certain circumstances but not others).

Grumble, grumble. On to iteration 3.

  //x = this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);
  x = 3;

This will also fail. The line with the comment is over 80 characters wide and this is not tolerated by many style guides, presumably because said code bases are being worked on by people who only have access to text consoles. On to attempt 4.

//x = this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);
  x = 3;

This won’t work either for two different reasons. Case 1: the variables used as arguments might not be used at all and will therefore trigger unused variable warnings. Case 2: if any argument is passed by reference or pointer, their state might not be properly updated.

The latter case is the worst, because no static checker can detect it. Requiring the code to conform to a cosmetic requirement caused it to grow an actual bug.

Getting this kind of test code through all the testers is a lot of work. Sometimes more than the actual debug work. Most importantly, it is completely useless work. Making this kind of exploratory code fully polished is useless because it will never, ever enter any kind of production. If it does, your process has bigger issues than code style. Any time spent working around a style checker is demotivational wasted effort.

But wait, it gets worse!

Type checkers are usually slow. They can take over ten times longer to do their checks than just plain compiling the source code. Which means that developer productivity with these tools is, correspondingly, several times lower than it could be. Programmers are expensive, having them sitting around watching preprocessor checker text scroll slowly by in a terminal is not a very good use of their time.

Fortunately there is a correct solution for this. Make the style checks a part of the unit test suite, possibly as part of an optional suite of slow tests. Run said tests on every CI merge. This allows developers to be fast and productive most of the time but precise and polished when required.