Re: How to store the data of a large map<string, int>?
liujiaping wrote:
On Aug 6, 11:31 am, Barry <dh...@126.com> wrote:
liujiaping wrote:
Hi, all.
I have a dictionary-like file which has the following format:
first 4
column 7
is 9
a 23
word 134
...
Every line has two columns. The first column is always an English
word, and the second is an integer number. The file is a text file,
and contains tens of thousands of lines. My program has to read this
file, and I use the container map<string, int> to store the data.
Every time my program runs, the file is read. But since the file is
too large, the speed is very slow. Is there any other efficient way to
organize the data of the file to make it fast to read?
Any help is appreciated. Thank you.
I think you have to convert your text file into binary mode, built as a
dictionary indexed file.
You can have such structure to serialize your data into the dictionary file
struct Foo {
unsigned int word_len;
char* word;
int key;
};
and index to Foo object into an integral value so you can search it
fast, like hashing, to build a index file.
You can reference StarDict, an open source dictionary, it will give you
some hints.
Thanks for ur advice. But how to write the struct Foo to a binary
file?
since you load the text file into vector<map<string, int> >
say word_map
struct Serializer {
Serializer(ofstream& ofs) : ofs(ofs) {}
void operator() (pair<string, int> const& p) const {
string::size_type len = p.first.size();
ofs.write((char const*)&len, sizeof(string::size_type));
ofs.write(p.first.data(), len);
ofs.write((char const*)&p.second, sizeof(int));
}
ofstream& ofs;
};
ofstream ofs("out.dict", ios::binary);
for_each (word_map.begin(), word_map.end(), Serializer(ofs)));
Can the function fwrite() do that? And given the binary file, how do
you
read from it to the struct Foo? Is there any example about it?
word_map;
void load(iftream& ifs) {
while (!ifs.eof()) {
string::size_type len = -1;
ifs.read((char*)&len, sizeof(string::size_type));
assert(len <= 1024);
char buf[1024]; // maximum buffer for a word
ifs.read(buf, len);
string word(buf, len);
int key;
ifs.read((char*)&key, sizeof(int));
word_map.insert(make_pair(word, key));
}
}
////////
In addiction, if you wanna have index file for your data (then you can
load just one specific word data), you have to write the hash code(have
to be unique) and the file offset in the dict data file, when you're
loading one word, just look up in the index file, then locate the word
data with offset in the data file.