Re: C++ build systems
On Sep 15, 11:30 pm, Joshua Maurice <joshuamaur...@gmail.com> wrote:
I'm curious how all of you actually build your code. GNU Make,
SCONS, bjam, vcproj files, etc.?
I use GNU Make and some pretty hairy generic makefiles. It's
far from perfect, but I've yet to find anything really better,
and it seems to be about the most portable around.
Specifically, I'm very curious how close your build system is
to my "ideal" build system. My ideal build system has two
requirements, fast and correct. Correct means that a full
build at any time produces the same results as a build from a
completely clean local view. Fast means fast. When builds can
take hours or more, I don't want to look at an error, and
wonder if it was because I didn't build from clean,
potentially wasting more hours as I do a sanity check rebuild.
All build systems can be made correct by having it first
completely clean out the local view, then building. However,
this is also the polar opposite of fast. I think it's fair to
say that the only way to achieve correct and fast is to have
an incremental build system, a build system that builds
everything which is out of date, and does not build anything
already up to date.
The problem is that it is very, very difficult to do this and be
correct. Most build systems I've seen compromise somewhere, and
will occasionally rebuild things that aren't necessary.
(From what I've heard, Visual Age is the exception. But I've
never had the chance to try it, and from what I've heard, it
also compromizes conformance somewhat to achieve this end.)
(Arguably, that definition is biased. It infers the use of
file timestamps, but there could exist fast and correct build
systems which do not rely upon file timestamps.)
I didn't see anything about file timestamps in it. In fact, my
reading of what you require pretty much means that anything
based on file timestamps cannot be used, since it will
definitely result in unnecessary recompilations (e.g. if you add
some comments to a header file). Most make's do a pretty good
job of being correct, but because they depend on file
timestamps, they occasionally recompile more than is necessary.
I think it's also fair to say to achieve fast, you need to
have a parallel build system, a build system that does the
build steps in parallel as much as possible.
GNU make (and many, probably most others) are capable of
partially parallelizing the build, starting compilations of
independent modules in parallel, for example. On single
systems, I've not found that this buys you a lot. Some build
systems (Sun's make, for example, and I think the make for
Clearcase) are capable of distributing the build---starting the
parallel compilations on different machines in the network; that
can be a bit win, especially if you have a relatively large
network. On Unix based systems, at least, this can be simulated
relatively easily using shell scripts for the compiler, but the
simulation will generally only be a first approximation---IIUC,
Sun's make will search out machines with little activity, and
Something like "ease of use" is a third aspect of my ideal
build system. A naive build system built upon GNU Make which
requires developers to manually specify header file
dependencies may be technically correct, but it would be
ridiculously error prone and a maintenance nightmare.
Which is why GNU make (and most other makes) have provisions for
automatically generating the dependencies.
I don't know a better way to describe what I want than
"idiot-proof". If you miss a header file dependency, the next
incremental build may produce correct results, and it may not.
I want a build system where the incremental build always will
be correct (where correct means equivalent to a build from
completely clean) no matter the input to the build system.
(However, if the build system itself changes, then requiring
all developers to fully clean would be acceptable.)
There are two ways of using automatic dependency checking with
GNU make. The first is to specify a special target (typically
"depends"), which is invoked to update the dependencies. The
second updates the dependencies automatically at each make. The
first depends on the user explicitly invoking the target when
any header file usage is changed, which while not as error prone
as requiring the programmer to specify the dependencies
directly, is still open to error. The second has (or had) a
distinct impact in build times.
I currently use the first, because I don't change header use
that often, and the impact on build times was very significant
when I tried the second. The place where I just worked,
however, recently implemented the second, and the impact on
build times seemed quite acceptable, at least on modern machines
with modern compilers, so I'll probably change, when I can find
the time. Note, however, the building the dependencies requires
some collaboration from the compiler; otherwise, you need a
separate run of the compiler and some more or less fancy shell
scripts. (This is what I currently use with VC++: I've not
found an option which would generate the dependencies otherwise,
so I use /E, with the output piped to a shell script. It works
well, but I suspect that trying to do it every time I compile
will impact build times considerably.)
Lastly, I want it to be portable to basically anything with a
C++ implementation. My company supports nearly every server
and desktop platform known to man (Ex: Z/OS, HPUX, AIX,
Solaris, WIN, Linux, several mainframes, and more), and
several common Make-like tools purport to not run on all of
these systems. Preferably, I would like to not be bound to the
system specific shell either like Make. (Although you could
just require all developers to install sh and force sh usage,
at least for the makefiles.)
GNU make, itself, isn't really bound to a shell. On the other
hand, the commands that you use to rebuild the code will be
interpreted by the shell if they contain any metacharacters;
under Unix, this will always be /bin/sh, unless you specify
otherwise, but on other platforms (Windows, at least), it uses
the environment variable SHELL. My own build scripts
(makefiles) make extensive use of the shell, particularly for
executing tests, so I do require a Unix-like shell. Globally,
this is probably the simplest solution anyway: require the
installation of a Unix like shell, and set the environment
variable to point to it.
For example, for your build systems, how much of the above
does it do?
1- Does it correctly and automatically track header file
Not automatically, but it can be programmed to do so.
2- If a DLL changes, will you relink all dependent DLLs, transitive
and direct, or will you only relink direct dependent DLLs?
I'm not sure. I avoid dynamic linking when it's not necessary.
3- Can your developer inadvertently break the incremental
build in such a way that it succeeds on his local machine but
fails on the build machine because he missed specifying a
The developer doesn't specify the dependencies. On the other
hand, because they are generated by a specific target, and (at
least in my case) are platform dependent, it's quite easy for
them to be up to date on one platform, and not on another.
4- Is your (unit) test framework incorporated into the build,
so that you can run only the tests which have depend upon a
The unit test framework is part of the build procedure, with the
"install" target dependent on the "runtest" target (which only
succeeds if all of the unit tests pass). I'm not sure what you
mean with regards to the second part---I've not done so, but it
wouldn't be too difficult to modify the makefiles so that the
entire install procedure is skipped if nothing has been changed
in the component. Other than that, however, I don't want to be
able to install a component where other components can see and
use it without executing all of the unit (regression) tests on
5- Can your build system generate vcproj files on windows? (I
personally like the Visual Studios IDE, and it would be a
shame to lose this by going to a fast, correct build system.)
No idea. I don't even know what a vcproj file is. (In theory,
it should be able to, but I don't know how. And with regards to
the Visual Studios IDE, I've used it a couple of times on my own
code, without any vjproj files in the project itself---I just
open a more or less dummy project, then indicate where it should
find the relevant files.)
Thus far, I have not found a satisfactory answer to these
questions in the open source world. Thus, I've been developing
my own answer for the last couple of months, where I hope to
achieve (almost) all of these goals.
Handling simple dependencies at the file level is pretty
trivial. Handling things like the automatic generation of
dependencies, and requiring a minimum of input from the
developers in each component, can rapidly become very