Re: Memory footprint of std::vector<std::vector<T> >
On 10 Des, 23:35, The Librarian <7h3.l1br4r...@gmail.com> wrote:
On Dec 10, 3:24 pm, Rune Allnor <all...@tele.ntnu.no> wrote:
I have a data structure which I naively implemented as
std::vector<std::vector<T> >. Right now I'm testing it
with 8000 vectors of vectors, each of which contain on the
order of 1000 elements, ending up somewhere on the
order of 350 MB of memory. I am nowhere near exhausting
any meory limits.
The program was compiled with default 'release' settings
(VS2008) and takes forever to run.
Starting from VS2005, Microsoft has introduced a new feature called
"checked iterators" that affect several STL classes.
As a consequence, every time you access an element of std::vector<T>,
the vector bounds are checked.
The situation is even worse for std::vector<std::vector<T> > because
double checks are performed (I think it requires 4 comparison for
every element access).
In order to disable this "feature", you should set the symbol
_SECURE_SCL to 0 before including the stl headers.
This is an example taken from MSDN:
// checked_iterators_4.cpp
// compile with: /EHsc
#define _SECURE_SCL 0
#include <vector>
#include <iostream>
using namespace std;
int main() {
vector<int> v;
v.push_back(67);
int i = v[0];
cout << i << endl;
};
Apart from this, I suggest you to turn on all the optimizations (in
particular SSE2, it can make a big difference).
The SSE2 optimization seems to have at least halved the
data load + processing time.
The #define _SECURE_SCL 0 trick caused a segmentation
fault when I used it only in the main trasnslation unit;
it might work better if I include it in every file.
I'll have to try the std::deque and reserve() on the
vectors, although there is probably more time to be
saved by reimplementing some of the processing routines,
although these don't take very long (about the same time
it takes to convert from text to binary numbers when
I tested the program with a different setup).
When I run the program I watch memory consumption in
the task manager. The numbers update approximately
twice per second.
From 0 MB data loaded to about 200 MB data loaded,
the memory ticker increases about 1.7-1.8 MB/update.
After 200MB it increases about 0.3-0.4 MB/update.
Which is why I suspected reallocations of continuous
blocks of memory to be the culprit.
BTW, is there a way to adjust the size of the ifstream
buffer? I know I will be working with files with
sizes on the order of hundreds of MBytes, so I can
just as well allocate, say, 10 MB for the stream buffer
up front to reduce overhead on file I/O.
Rune