Re: JMX OutOfMemoryError: Requested array size exceeds VM limit

From:

Eric Sosman <esosman@comcast-dot-net.invalid>

Newsgroups:

comp.lang.java.programmer

Date:

Tue, 17 Dec 2013 17:33:50 -0500

Message-ID:

<l8qjg7$pd5$1@dont-email.me>

On 12/17/2013 4:32 PM, mikesmith813@gmail.com wrote:

I have a Java server application that consistantly throws an "OutOfMemoryError: Requested array size exceeds VM limit" and crashes about every 70 hours.

I'm running on Java 1.7 release 45 with a beefy 18 CPU Linux Server with lots of memory. My max heap is set to 50GB and using the new G1GC collector. Profiling with JVisualVM shows that the app is not using more than half of the heap ever and PermSpace looks fine.

Clearly, something is trying to allocate an array of an incorrect huge size but I can't determine what. My code creates objects and sends across JMX but the objects are already allocated in memory and handed to JMX at Exception time. I'm wondering if this is a JMX bug but I cannot find anything online. Any thoughts/ideas are appreciated! Stack trace below:

Job_Executor6:class com.sun.jmx.remote.opt.util.JobExecutor java.lan.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.grow(Unknown Source)
at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at com.sun.jmx.remote.generic.ObjectWrappingImpl.wrap(ObjectWrappingImpl.java:30)
at java.management.remote.generic.ServerIntermediary$RequestHandler.handleNotifyReqMessage(ServerIntermediary.java:683)
at java.management.remote.generic.ServerIntermediary$RequestHandler.execute(ServerIntermediary.java:626)
at java.management.remote.generic.ServerSynchroMessageConnectionImpl$RemoteJob.run(ServerSynchroMessageConnectionImpl.java:266)
at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:208)
at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:59)

     I'm not familiar with JMX, but from the stack trace it's
clear that somebody is trying to serialize a big object (or a
big object graph) to a ByteArrayOutputStream. There's a byte[]
underlying that stream, and (like any other Java array) it can
have no more than 0x7fffffff elements -- just shy of 2GB. So:

     1) Why are you trying to serialize a 2GB object (or graph)?
        Does one of the objects being serialized have a reference
        to something enormous, like all of Snowden's files?

     2) If you really must serialize something that large, do you
        need to store it all in memory before sending it wherever?
        That is, could you write directly to a socket stream or a
        file instead of storing it up in a ByteArrayOutputStream
        first?

     3) A 2GB network transmission or disk write (however it's
        marshalled) will take macroscopic time: >3.5 minutes at
        10MB/sec. Just sayin' ...

--
Eric Sosman
esosman@comcast-dot-net.invalid