Re: Memory allocation failure in map container
Saeed Amrollahi <amrollahi.saeed@gmail.com> wrote:
I am working on an application to manage Tehran Securities Exchange
organization.
In this program, I try to manipulate a lot of institutional investors
(about 1,500,000). The Investors are a struct like this:
struct Investor {
int ID;
std::wstring NID;
std::wstring Name;
std::wstring RegNum;
std::wstring RegDate;
std::wstring RegProvince;
std::wstring RegCity;
std::wstring Type;
std::wstring HQAddr;
// the deblanked or squeezed info
std::wstring sq_Name;
std::wstring sq_RegNum;
std::wstring sq_RegProvince;
std::wstring sq_RegCity;
};
The investors are stored in an Access Database and I read them into
memory at the program loading time. Because, each investor is
registered in a City, I tried
to use a map container: map a city to the collection of investors:
map<wstring, vector<Investor>*> g_Investor; // map city name to the
investors located in
For the sake of efficiency, I used pointer to vector as a value of
the map.
Using a pointer as you do above is not inherently any more efficient
than map<wstring, vector<Investor> > would be.
the following code fills the map:
void InvestorSearchEngine::FillInvestors(const vector<Investor>& m)
{
So at this point, 'm' has 1,500,000 members? I can only assume that it
isn't referring to some temporary structure. If it isn't, then one thing
that may help is to have the map hold a vector of Investor* instead of
Investor. That way you won't have two copies of each Investor object.
However, if the vector that 'm' refers to changes, you will have to
rebuild g_Investor all over again. There is probably a better solution.
I see that your class is called "InvestorSearchEngine." If the goal of
this class is to help clients retrive Investors that meet specific
criteria, then keeping a huge map of all the Investor objects by city
seems inappropriate.
try {
for (vector<Investor>::const_iterator cit = m.begin(); cit !=
m.end(); ++cit) {
if (g_Investor.find(cit->second.RegCity()) != g_Investor.end()) {
// so, there is a node with reg. place already
map<wstring, vector<Investor>*>::iterator it =
g_Investor.find(cit->second.RegCity());
it->second->push_back(cit->second);
}
else { // the registered city is new
vector<Investor>* vp = new vector<Investor>();
vp->push_back(cit->second);
g_Investor[cit->second.RegCity] = vp;
}
}
}
catch(const std::bad_alloc&)
{
wofstream LogFile("D:/log.txt", std::ios_base::app);
LogFile << "Memory exhausted ... " << L'\n';
LogFile << g_Investor.max_size() << L'\n';
LogFile.close();
}
Unfortunately, sometimes, the memory allocation is failed and the
"Memory exhauseted ..."
message was written to log file.
It sounds like you need to sacrifice some of this preloading in order to
save space.