Re: How to set up a fast correct java build?
This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.
---910079544-873830374-1263075790=:22299
Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8BIT
On Fri, 8 Jan 2010, Joshua Maurice wrote:
On Jan 8, 12:11?pm, Tom Anderson <t...@urchin.earth.li> wrote:
On Fri, 8 Jan 2010, Joshua Maurice wrote:
I'm working on a project now with over 20,000 java source files, in
addition to more than 4,000 C++ source files, some forms of custom code
generation, an eclipse build, and probably other things I don't know
offhand.
How do you build your java code?
With a clean build. I have a much smaller project than you.
My short and serious answer is that your project is too big. It's big
enough that trying to manage it as a single entity is utter madness. You
need to find a way to split it into smaller independent parts, each with a
more tractable build problem. You may balk at this, and indeed, it would
be difficult, but if you don't do it, there is simply no way that you're
going to be able to do builds without pain, not with all the build magic
in the world. Sorry.
I agree this is the "best" approach. However, that would require
changing my company's mindset and culture, and doing significant
refactoring of code. The culture here is that they don't believe in
design by contract, in general purpose reusable components, and and
such, we've coded ourselves into a tangled mess.
Fair enough. I would at least considering doing the Change Employer
refactoring on yourself. :)
I've been reassigned to try and speed up build times, but before I was
working on one of the things at the base of the dependency tree. When I
make a change, I generally had to build everything and run all tests
because the tests for my component were not comprehensive, mostly
because of this culture of no design by contract. Instead, any small
change made at the root may have subtle nuances and break code far far
away, either at compile time or test time.
I disagree with your assessment though that no magic bullet will make
these problems go away. I don't call it a magic bullet, but an
incremental parallel build would work wonders. If I had that, compile
times would go down drastically, like 2+ orders of magnitude.
It will speed up builds. My point was that there will be other huge pains
resulting from this giant syncytial code structure, to do with local
changes having nonlocal effects. Even if a change never outright breaks
something, it means each programmer has to maintain a lot of global
knowledge just to be able to work locally. Still, as you say, them's the
breaks.
Frankly, the current state of affairs in the Java community is not
acceptable, and even laughable, given that solutions to these problems
(fast build, correct build) are known and have been known for many, many
years in the context of C and C++.
And have been necessary purely because compiling C is slow, and C's weak
dynamic binding means you have to build everything all at once. Java
projects simply work in smaller bits, and do complete builds quickly.
Before you dismiss this as a cop-out, consider that there are some really
very big java systems out there - they got built, and you don't hear a
great wailing and gnashing of teeth in the java community about the pain
of building. Indeed, try an experiment - find someone with experience of
both C and java, working in the normal modes for both, and ask him which
build experience he'd prefer. Your problem stems from working on java in a
C mindset.
I'll bite. How long do you think it would take to recompile 20,000
java files, and just the java files? I'm about to get numbers on that
anyway, and I'll share if / when I get them.
I downloaded HtmlUnit snapshot build 1674 (the most recent). It contains
419 source files, totalling 2.9 MB. I built it with a script that said:
find src -type f -name \*.java | xargs javac -g -classpath 'lib/*' -d build/classes
(where javac is from Sun's JDK 1.6.0_16, running on Ubuntu jaunty)
I ran that, then ran it again under 'time', which reported:
real 0m16.697s
user 0m16.281s
sys 0m0.688s
Assuming compilation time scales linearly (which it probably doesn't - i
imagine it's slightly superlinear), that would suggest that compiling 20
000 files would take 796 seconds, which is 13 minutes and 17 seconds.
That's certainly not something i'd want to do with every file i change,
but it's something i'd be prepared to do several times a day. At work, we
have a build that takes 20-25 minutes (including database setup and so
on), and we run that one to five times a day when developing; it's a pain,
but it's the right amount of time to go and get a fresh cup of coffee,
tend to some minor administrivia, read up on some technical point that
came up while coding, etc.
Oh, and the build time for your case would be lower than my prediction if
your build machine is beefier than mine - an Eee PC 1001HA, with a 1.60
GHz Intel Atom N270 CPU, 1 GB of RAM, and i'm guessing a 5400 rpm disk.
Depends on whether you can afford to spend more than 200 quid on it, i
guess.
I agree with most of your assessment except with one minor
qualification: if your codebase is pure java, then it works out pretty
well. You can just get it all in Eclipse, and you have a wonderfully
good incremental compiler. However, my codebase is not all java. It has
custom inhouse codegen which makes Java.
Assuming you don't rebuild the generator (and if you are, and it changes
slowly, that's a prime candidate for factoring out into a separate build),
and that the input to the generation is a set of files which don't depend
on anything else, then this can be dealt with by some fairly
straightforward make, before any java build.
I assume that once the generation is done, there's nothing else blocking
the compilation of the java - there may be dependencies on C++ via JNI,
but the nature of JNI is that the compile-time dependency is from the java
to the C++, as not vice versa.
It has Java classes implemented in JNI, so some of the C++ compile
depends on Java compile for javah.
So after the java build, you have the javah run, and then the C++ build.
A lot of the tests are reverse: the Java tests depend upon the C++ code,
Again, a runtime rather than a compile-time dependency, i assume.
itself some of which is generated by a Java tool, which itself depends
on more C++ and Java code being built.
I'm guessing (i) that the generated C++ is test code of some description,
and that none of the main C++ depends on it, (ii) that the tool's
dependency on the C++ is again runtime, not compile-time, and that (iii)
the C++ it depends on (what i'm calling the 'main C++') has no further
dependency on generated code except javah output, and via that the java
autogeneration.
If my assumptions are correct, your phases are:
- Autogeneration of the java
- Compilation of all java
- javah
- Compilation of the non-generated C++
- Generation of the test C++
- Compilation of the test C++
That is indeed a lot of work. But it looks to me like the worst of it will
be generating and compiling the C++. BICBW.
That's ignoring entirely the biggest mess in it all: our Eclipse GUI
plugins thingy build.
Yeah, you're hosed.
Maybe in it all it could recompile the java every time. It might work,
as long as I define proper hackery to have the javah step run iff the
java source file has changed, not the class file, to not trigger
rebuilds of C++ stuff. (Would that work?)
Not triggering rebuilds is good, but i'm not sure i'd tie it to the source
file changing. I'd be tempted to always run javah (i think it's pretty
quick), but only to rebuild dependent C++ if the contents of the .h file
have changed. Running javah will make a fresh file, which will have a new
timestamp, even when the contents are the same, so you need a step to weed
this out. I'd set up two locations for javah .h files, one for javah to
write into, and one for make to read from, then add a stage which copies
files from the former to the latter iff they have different contents. And
by 'stage', i mean 'this shell script':
#! /bin/bash
set -eu
SRC_DIR=src
DST_DIR=dst
for header in $SRC_DIR/*
do
header=$(basename $header)
if [[ ! -f $DST_DIR/$header ]]
then
echo Copying new header $header
cp $SRC_DIR/$header $DST_DIR/$header
else
cmp -s $SRC_DIR/$header $DST_DIR/$header || {
echo Copying changed header $header
cp $SRC_DIR/$header $DST_DIR/$header
}
fi
done
Or some suitable improvement thereof.
That said, thinking about it, tying the javah to the .java is probably
both simpler and faster. Oh well.
However, one of my goals is still to get the build system overhead down
to seconds ideally, though I think 1-3 minutes is a more reasonable
goal. Something where I can hit "build" from root after making a change,
instead of the current situation where every developer thinks he knows
more than the build system, and only builds subfolders which he knows
are affected by his change. However, when the developer misses
something, and it gets to the official build machine streaming build, it
causes lots of lost time. I want the developer to no longer feel obliged
to "hack it" and instead let the build system do its job.
How often do developers check in to the trunk from which the main build
gets done? It would be enough to require them to do a successful full
rebuild and test immediately before doing that; being able to do it
frequently while developing would be very nice, but is not essential. If
they're checking in 0.5 - 2 times a day (roughly what we do where i work),
then a 20-30 minute build would be tolerable. If you're integrating more
frequently than that (which is of course a good thing), then it probably
isn't tolerable.
If the working cycle fits my description, you could even enforce this with
checkin hooks - every candidate checkin gets built and tested before it's
accepted.
Speaking of Eclipse, perhaps the way to go might be to just do the
entire build in Eclipse. Write Eclipse build plugins for C++, for our
custom codegen, etc. I'll have to seriously consider that.
Eclipse has some degree of C++ support - it uses an external compiler, and
possibly an external make tool too.
Your only real problem is wiring in the code generation (including javah).
I'm not any kind of Eclipse build expert, but i know there's a way to get
Eclipse to do vaguely make-like rebuilding via arbitrary processes - we
had a project which had some JAXB, and rebuilt the java binding objects if
the schema changed. I suspect this is done via ant, which doesn't have
good make-like incremental rebuilding built in, but can be used to
manually write build scripts which only rebuild when necessary, by looking
at timestamps. If there's a way to get that ant script to trigger further
rebuilding in Eclipse, then this should basically do the job.
tom
--
Socialism - straight in the mainline!
---910079544-873830374-1263075790=:22299--