Re: Serious concurrency problems on fast systems
On 6/1/2010 12:21 AM, Kevin McMurtrie wrote:
I've been assisting in load testing some new high performance servers
running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is
suspending threads for time-slicing in very unfortunate locations. For
example, a thread might suspend in Hashtable.get(Object) after a call to
getProperty(String) on the system properties. It's a synchronized
global so a few hundred threads might pile up until the lock holder
resumes. Odds are that those hundreds of threads won't finish before
another one stops to time slice again. The performance hit has a ton of
hysteresis so the server doesn't recover until it has a lower load than
before the backlog started.
You've got "a few hundred threads" all calling System.getProperty
over and over again so rapidly that they get in each other's way? Why?
Seems a very odd situation, but ...
Okay: As a first cut, write your own PropertySurrogate class.
Have it slosh all the System properties into a HashMap instead of
System's Hashtable, and change the System.getProperty calls to
PropertySurrogate.getProperty instead. Since HashMap isn't
synchronized, threads calling PropertySurrogate.getProperty won't
block each other.
If this over-and-over reading and re-reading of the System
properties is because you suspect somebody's changing them, you
could have one thread that just keeps iterating over the System
properties, watching for changes. When it sees a change, it would
update the PropertySurrogate HashMap accordingly. This veriant
needs synchronization, but you could use a ReadWriteLock to allow
multiple read-only threads to access the map simultaneously; they'd
be locked out (briefly) only when the monitoring thread was in the
middle of an update.
Still, it seems to me that what you should be looking at is
the reason for all these repetitive getProperty calls. It sounds
like reading the clock every microsecond to see if it's 2014 yet.
--
Eric Sosman
esosman@ieee-dot-org.invalid