Re: A simple unit test framework
James Kanze wrote:
On May 7, 2:02 am, Branimir Maksimovic <b...@hotmail.com> wrote:
On May 6, 3:03 am, Gianni Mariani <gi3nos...@mariani.ws> wrote:
James Kanze wrote:
...
Yes, but nobody but an idiot would pay you for such a thing.
Thread safety, to site but the most obvious example, isn't
testable, so you just ignore it?
Common misconception.
1. Testability of code is a primary objective. (i.e. code that can't be
tested is unfit for purpose)
2. Any testing (MT or not) is about a level of confidence, not absoluteness.
I have discovered that MT test cases that push the limits of the code
using random input does provide sufficient coverage to produce a level
of confidence that makes the target "testable".
I have exactly opposite experience. Multi threading program failures
show up only with specially crafted input.
That is I have to look into code first and then make tests to
specifically target places in code .
But, such errors are usually most simplistic ones, those are common
threading errors, and can be usually catched by eye.
Even when you know exactly where the error is, it's sometimes
impossible to write code which reliably triggers it. Consider
std::string, in g++: if you have an std::string object shared by
two threads, and one thread copies it, and the other does [] or
grabs an iterator at exactly the same time, you can end up with
a dangling pointer---an std::string object whose implementation
memory has been freed. The probability of doing so, however, is
very small, and even knowing exactly where the error is, I've
yet to be able to write a program which will reliably trigger
it.
Are you sure. I ran into that problem in an earlier version of gcc's
std::string. I have never seen it since and I did review the
std::string code at 3.0 and I was satisfied that all was well.
std::string is not thread safe but this should work:
std::string global1( "A" );
std::string global2( global1 );
void thread1()
{
global1 += "1";
}
void thread2()
{
global2 += "A";
}
Consider some variations on DCL. In at least one case, you only
get into trouble if one of the processors has read memory in the
same cache line as the pointer just before executing the
critical code. Which means that the function can work perfectly
in one application, and fail when you link it into another
application.
Yes, but it's exactly this type of test that finds bugs like that.
As I mentionned in a response to another poster, it may just
depend on what you consider acceptable quality. I've worked on
critical systems a lot in the past. With contractual penalties
for downtime. So I tend to set my standards high.
Interestingly enough, however, it turns out that developing code
to such high standards is actually cheaper than just churning it
out.
Yes. Better tested code is easier to develop with.
... As a rough, back of the envelope figure: correcting an
error found in code review costs one tenth of correcting the
same error found in unit tests,
What are you smoking ? Sure you can find some obvious bugs in a code
review, but I would have already run the unit tests before I do the code
review.
... which costs one tenth of
correcting the same error found in integration tests, which
costs one tenth of correcting the same error found in the field.
We can haggle on how many orders of magnitude field errors cost but I'll
agree it's at least 2 orders above unit tests.
Not to mention the advantages of the transfer of knowledge which
takes place in code review. The result is that any time an
integration test finds an error, it is considered first and
foremost an error in the process; we analyse the process to find
out where it failed. And there is even a tendancy to do this
for unit tests, although perhaps not as systematically.
[...]
It does require a true multi processor system to test adequately.
Not just mp system, but *all* possible mp systems, which is
of course impossible.
It's perhaps worth pointing out that most current low-end MP
systems use a more or less synchronized memory model. This is
not true, however, for Alpha processors, nor the Itanium, nor, I
think, top of the line Sparcs, nor, probably future desktop
processors. That means that the fact that code works on a
current four processor system based on Intel 32 bits means
nothing with regards to future processors.
Only in low level code. Code written above the iron layer should work
just fine.