Re: fgets() vs std::getline() performance
Jeff Koftinoff wrote:
Wu Yongwei wrote:
Why do you say `no exception thrown'? I would expect a
std::bad_alloc, and, when it is not caught, an abort().
You would expect correct based on a conforming system.
However, any system that employs lazy memory allocation
{allocation of memory pages at page fault time instead of
malloc() time} , behaves differently.
Yes, but you really shouldn't allow such machines to be
connected to the Internet. It's the overcommit which is the
problem, not the code using getline(). Normally, any Linux
machine connected to the network should have the value 2 in
/proc/sys/vm/overcommit_memory. (I would, in fact, recommend
this on any Linux system, unless some of the applications
being run require overcommit or were designed with overcommit in
mind. The process which gets killed isn't necessarily the one
using too much memory; I have heard of a least one case where a
critical system process needed to login was killed.)
Note that not having overcommit isn't a panacea either. I can
remember Solaris 2.2 hanging for around 5 minutes, thrashing
like crazy but not advancing in any visible way, in one of the
stress tests I did on it. (This seems to have been fixed in
later versions. At least, my stress test caused no problems
with Solaris 2.4.)
[...]
It is a security hole when it joins the the lazy allocation
problem.
Overcommit is a very serious security hole. Which has nothing
to do with getline or fgets (unless you are considering the
possibility of writing the entire program without using any
dynamic memory).
It allows an untrusted user to kill servers, potentially even
other unrelated admin servers running on the same system.
The untrusted user doesn't get a choice with regards to which
processes are killed. For that matter, nor does the trusted
user:-). Anytime you run a system with overcommit, any program
which uses dynamic memory, OR forks, OR does any one of a number
of other things which could cause memory to be allocated, may
crash the system anytime it runs. Common sense says that you
don't run anything important on such machines.
Yes, it is a problem with those system's designs. But it is a
real one that affects many real servers. It can not be
stressed more that catching std::bad_alloc is not always
enough!
IIRC, Andy Koenig wrote an article about the general problem a
long time ago, in some OO journal. In general: except on
specially designed systems, you can't count on catching
bad_alloc and recovering; there are generally cases of memory
allocation failure that escape its detection (the stack is an
obvious example). On the other hand, many programs don't
require 100% certainty of catching it, and on a well designed
system, if you exceed available memory and don't manage to catch
it, the system will still shut you down cleanly and free up most
resources. Also, if you're familiar with the system, you may be
able to avoid some of the problem areas---I've written programs
for Solaris where I could guarantee no insufficient memory due
to stack overflow after start-up.
When this problem could be a real issue, there are ways to
go around it. For example, using custom allocators. The
point is that C++ does not strangely force an arbitrary
limit on how long a line could be. And the system limitation
could be put somewhere else than the processing logic.
Unfortunately, the interface of getline() and std::string also
do not allow me to put a limit on how long the line could be.
And if I can use this space to respond to James Kanze's comment:
James Kanze wrote:
I agree that a version of the function with a maximum length
would be nice. Or simply specifying that extractors respect
setw too---that would be useful for a lot of other things as
well. But I don't see it really as a security hole, any more
than the possibility of any user function to allocate more
available memory than it should is. If untrusted users have
access to your program, you'll have taken the necessary steps
outside the program to prevent DOS due to thrashing.
What would the necessary steps be then to ensure maximum line
length in my protocol parses utilizing iostream and
std::string?
One obvious solution would be to overload getline with an
additional parameter specifying maximum length. A perhaps more
general solution would be to systematically recognize the width
parameter on input---this would also be useful for reading files
with fixed width fields, rather than separators.
(Note that depending on the implementation, inputting with >>
into an int might suffer similar problems, if you feed it too
many digits. (I would expect any good implementation to stop
storing digits once it recognizes that it has more than enough,
but all the standard says is that if you feed it a number which
is too big, you have undefined behavior.)
Should I write my own getline()? Or read one character at a
time?
It depends on what you are doing. I'd start by getting rid of
overcommit:-). AIX stopped using by default a long time ago
(but you can still turn it on, on a user by user basis, if you
need it). Most Linux distributions seem to default to using it
(which seems fairly irresponsible), but it's easy to turn off
(globally). Solaris and HP/UX don't use it. So you safe on the
major Unix platforms (True Unix for Alpha? SGI? I don't know.)
After that, it depends on the application, and what you're using
the standard streams for. My applications are mainly large,
reliable servers, and istream is used only for reading the
configuration file---if it crashes, it crashes during
initialization, due to an operator error (bad config file), but
the server doesn't go down once it is running. Unless it hits
some odd case in the system or the system library that I
couldn't protect against.
I also write a lot of little quicky programs for my own use.
There too, if they crash because of an excessively long line,
it's no big deal. (But if I were serious about it, I'd set up a
new_handler to emit a nice message before terminating the
program.)
I, for one, would love to have a std::string() class that had
an option of setting a maximum allowable length.
That's another issue. In many applications, it would, in fact,
be useful to have fixed length strings: if I'm reading from a
non-scrolling text field in a GUI, and using the data as a
column in a data base, I can pretty much limit the maximum
length to something that would generally fit on the stack. I
agree that what we need is a set of string classes, with
different policies with regards to length and memory management,
and with full convertability between them (and an exception if
an implicit conversion doesn't fit).
--
James Kanze GABI Software
Conseils en informatique orient?e objet/
Beratung in objektorientierter Datenverarbeitung
9 place S?mard, 78210 St.-Cyr-l'?cole, France, +33 (0)1 30 23 00 34
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]