Re: newbie question about data I/O

On Sep 20, 11:58 am, "Ross A. Finlayson" <ross.finlay...@gmail.com>
wrote:


ible


pen(), read ready, off


ate boundary


try" block has its types


 >> record; // <- the function to be composed to read the


Still, that doesn't really show that it's useful, because the idea is
to have the parser on the istream, and then the file with it's
serialization to the physical recording medium has then more semantics
to handle like read and write cacheing and flushing and so on, and as
well then the raw reads and write with the disk API.

Read and writing the disk records should have another exception
catching block on the outside try for the I/O exception, then that is
also for socket I/O, because there are all kinds of exceptions to
consider on the transport layers. With the inner istream templates,
if they can autogenerate expected composable extracto tr functions,
then it is kind of like boost::spirit the use of overloading >> and <<
even more, making composable the specification that evaluates to the
input evaluator, statically there in the expression block.

Also there are other I/O events to consider.

Then the user is to combine the while block templatized. What that
means, is,

collection A, B, C;

input >> A >> B >> C; // <- this should be any order

Then the while block is encapsulated there. So on the first pass it
collects the istream that is imbued with the property of waiting to
run its input extractors over the collections' composed input
extractor. Within the template used with the convention, the types
add to the semantics of the input type, where it can be extended from
the in-place input type.

Then, the collection type has to have the operator >> for the
composable input extractor, that for expressivity has the semantics in
the convention to configure the function object to handle exceptions
on reading the input and input events.

istream& template <typename T> operator >> (istream& i, collection& c)
{
    class istream_property
    {
        void* operator new (size_t void ); // this is just a pointer, but
it's automatic with the class
        // where to write about the istream?
    }
it would be static but not typesafe, the property. that is about
composing the extractors for the input.

    // don't read i, just compose with the other collections to read the
data
    // set a private variable in i, imbue the istream.

    carry forward

    return i;// because i is still the same just subclassed functionally
}

Then in templates the pre-composed input extractors have the idea
being that for each of the kinds of records there are, that they're
read from the file.

records

collection_type

collection

records += these;

input >> new_records;

compare (new_records, old_records);

old_records.compare(new_records);

collection_type c = new_records( templatized_type);

The istream's go to the extractor composers before there is ever
called istream read, which return a function object that is the while
loop with the error frame.

filename_t filename = "input.txt";

filespec_t filespec = "input.txt" + "input.dat" + "input.xml"; // <-
concatenate or combine the contents of these files
// <- also combine their extractor specifications?
filespec_t filespec += "schema.xsd" + record_line_delimited
(field_tab_delimited); // <- apply the write schema to the file spec

// then get the table representations for the data source/sink

template <typename reader_type>
{
// ifstream input_file(file_name) // <- file name, file des,

template <typename > class input : istream;

input in (filespec); // <- default an ifstream, file name, file des,
socket des

// here there is composed up the exception handlers for the I/O
events

template <typename > collection_type{
    class T::collection;
    class T::record;
    class T::extractor : extractor_base, composable

}

}

// read(in);

in >> records >> out; // why is this extractor from the input to the
output, it could read to end or so, check the exception state of the
input

Then, it's not just records, it's transforming the records between
input and output. After reading records, there still could be unread
records. After writing the records to output, there still could be
unwritten records.

template <typename > operator >> (&input i,

Then, with covariant return types, the compiler picks up

input : istream& operator >> (istream& i, collection<record,
inputspec>);

Then writing

in >> records;

Then all the contents of the input, however they are composed from the
input specification, have had the input spec's extractors composed.
Then, they are combining with the records, the contents from the input
are the inputs to algorithms

Then, it's about the input type, exhausting the input, or waiting for
there to be more data in the input.

Now, the input extractors, should be the standard I/O stream
extractors, with the semantics, for reading and writing the literals.

Then, the point is getting the data into A, B, and C. Consider for
example XML with many different namespaces in the same file. These A,
B, and C collections of records

Yet, if the user wrote it

in >> records;
in >> records;

Basically it's a reference instead of a pointer, it's not clear how to
imbue the ifstream with a constructor from the input specification, so
the extractors can be generated using standard conventions for the
input extractors, in ease of use of overloading the input extractors.

Then, as to how to generate the file specifications, an example was
shown to compose them from literals like "filename.txt" and to
describe the specification of their content with the composable input
extractors.

input_specification spec =
{"filename.txt",record_line_delimited<field_tab_delimited> };
     // <- here there are type converters for the literal array

input in(spec);
records<record_line_delimited<field_tab_delimited> > records; // <-
make records, overload records
// these are fixed column records

while(in) // <- here compose all the exception handlers, or imagine if
an event jumps to here
{
    in >> records;
}

Hopefully the user could go like so

name n;
specification spec("filename.txt");

input in(spec);

records (Markers, locations, T, mu); // those parameters are of a type
that describe the contents, maybe just their names

in >> records;

Then, the records collection has what were the contents of the files
as records, row-tabular records, with fixed column widths. So, the
forward definitions to be implemented require a specification along
the lines of

tabular_data + row_major + row_record + record_line_delimited +
field_tab_delimited

Then, there is are questions about how to define the field types, how
to match them (eg string, int, double). There is the name of the
column, that's a name type, where there are considerations of names
about what names can be. Names are basically strings, with the notion
that names have a small maximum size, some MAX_NAME of a thousand
characters or so. Realistic name size expectations might be along the
lines of MAX_PATH, etcetera, the maximum length of a path
specification, for strings longer than that they are more the data
than the names.

Then, the input extractors are to work with the standard extractors
and fit within the standard extractor conventions.

ifstream ifs;

ifs >> records;

Then, also there is a consideration if the user might have written

ifs >> data;

// ifs.read(); // <- no, the read function of an input file stream
means something specific
read<ifstream>(ifs); // <- user might expect all of the input to be
read into the static data

and as well how the users have already written many input extractors.
I wonder how to make templates to then tie the extractors to the
record to then make that simply composable.

Then, for example in an input data file that is field tab delimited
within each record, where each item/record has an entry for each
field, the input extractor is to read it that way as well as reading
the contents. The user might want a function to which they could pass
other headers of the tables in the file data. Other generators of the
input might use spaces instead of tabs, this input has spaces and
tabs, defining the records on newlines. For code that uses its own
record types already, it would be nice to combine the extractors with
the records in their definition, for compositing the transforms, and,
to combine them in their default way which gets into, for example, in
terms of pointer ownership and serialization of object relationships,
regenerating local offsets.

That is about how the user might have a different form of input data
and expect it to be made coherent with data from a source with
different tables. That is about mapping the names of the tables in
transforms, with the record placement transforms. As well it is about
handling the composition of the extractions.

The template for compositing the extractors for the record can be
defined, how can it be defined automatically? That is about the
objects having the extractor from the input. The built-in types have
extractors defined with semantics. So, to later re-use them without
modifying user code then the idea is how to leave it say

while(in >> records); // <- empty process loop?

That the operation doesn't discard the input stream reference, instead
that it is instrumented in-place so that it compiles the templates
inline with user code.

Then, in normal user semantics, the object can be evaluated
automatically.

So, if the user has defined a record, then just bring it into the
template file with the templates to make the inserters, then the
inserters set themselves as the handlers in the standard input stream
extractor semantics.

c_file_has_structs.h:

union field // <- equalize / flatten alignment
{
    word; // <- no union names, syntax error?
    void*;
    int;
    char*; // <- C string, null-terminated
    size_t; // <- specialize to size of buffer and C string
    offset;
    float;
};

struct RECORD
{
    array<field> array; // <- objectize, make automatic
};

// <- records have fixed field width, computing record off field array

struct RECORD
{

};

Use the C semantics for structured storage of data, also read COBOL
records. Consider link table record type extraction.

Basically is C string pointer, and C data buffer pointer and data
buffer size pointer, with sharing the size pointer. Then, there are
the automatic types.

For that, build up the variadic template macro invocations. Here is a
way. It is about using alignment requirements on the struct, and the
guarantees of the sizeof operator on the typename. It gives the
alignment guarantees, the sizeof. It is used in the templates because
the numerics are evaluated before the templates. Generate then with
the templates the conversion of the struct type, by cancellations of
its alignment offsets, the extractor.

That is then about making it semantically equivalent to

RECORD r;

collection<RECORD> records;

input >> records;

input<RECORD>;

do { input }
||
do { !!input } // <- it's boolean but it checks the object
||
do { input() } // <- why try? in the templates, it's compiled
{
    while input;
}

// no, maybe instead

if (input || input() || !!input )
{
    input;
}

The point of writing that so many times is that each of those could be
different calls, or, each much the same or conditional on the others,
where before or after they are the same thing.

Then, there is maintain the automatic linearization of the program
components where there is the serialization of the compositor order
with the compositor components as those are the extraction operations
that work with the standard C++ library I/O stream istream extractor
semantics.

struct RECORD_WITH_C_STRINGS{
    char* name;
    char* record; // <- also these go in a union?
    void* data;
    size_t length; // <- this also hidden
    unsigned ; // <- also
};

autocompose_input_extractor.hpp:

#include "template_pre_defs.hpp"
#include "c_file_has_structs.h"
#include "template_post_defs.hpp"

template <typename T> record;

// auto-local anonymous namespace

operator >> ( istream* in, char* a name is a sequence, consider how to
instrument in the enum ranges

Don't want to use a char* for the extractor operator, it's a C-
string.

So, for RECORD_WITH_C_STRINGS, the standard extractors that are
generated via templates from it

operator >> (istream& in, extractor_string& s)

or

operator >> (input& in, char* name) // these for extractors of names
(fields)

or

operator >> (input& in, record& record) // these for extractors of
records

where input extends istream and is the imbued istream interface that
is supposed to be generated from its own include files.

A problem with the second one is that then the char* character pointer
has its pointer ownership undefined. Templates can be defined to work
on the reference but then the template doesn't cancel to const.

record<RECORD> records;

file f("input.txt");

f >> records;

Then say you want to write the records, where you would then perhaps
expect to read them even again later.

file f("output.txt") save;

records >> save;

Or probably more normally

ostream out = ofstream("output.txt") out // <- templatize in function
out, anonymize identifier

records >> out;

Then, if more records were on the other size of the out, the template
compiler might have already generated the semantic overload.

template <typename ostream_type> operator >> (record<RECORD>& records,
ostream& out);
{
    ostream_type this

    // this template works on ostreams so it goes into templates that
work on ostreams
    ostream out =

    // auto-generated record inserters to the output stream

    // if necessary, record the record specification that is part of the
specification that was used to construct the input reference.

    // then, there is a consideration as to how to make concise and
readable combined specifications, with using comments, and compiler
erasure

    // record all the changes in the semantic stream on the translation
for the records and parser support

    // combine auto-generated to cancellation on the automatic record
redefines with the compiler unit base

    records >> out;

}

Here is where, there are the standard semantics, and then there are
other program semantics that the existing inserters and extractors
already have defined, which is the point of using the standard
semantics with the standard C++ library.

So, the fields are built up to form the records from the C structs,
automatically. That is about a container class that is defined on
both sides of C language input in C++ non-compiling blocks under
program acceptance has the build up of the redefines to support the
automatic types.

Now, about categorizing the C++ input and output semantics, with the
standard streams, I was reading a book about the I/O stream semantics,
with for example setting the exception handlers, and being very
careful in handling the exception handlers to discover semantics
because the automatic generation of the inserters and extractors is
simply using alignment guarantees of the template evaluation of sizeof
with the fields in the struct. That is where there is a keyword in
the compiler to get the offsets of fields in a struct, for example
offset_of, that accepts ordinal inputs. So if I can use sizeof in
templates then I want to use offset of, but, not by the name/
identifier of the field of the struct, but by its natural layout
including for example unsigned layout. Then, inserters and extractors
can be worked out about other features of the structure type, for
automatically generating standard inserters and extractors from any C
struct that is plain old data, POD struct. Yet, the alignment size of
void* and char* is the same. This and other features have then more
definitions for the field type, because to generate the automatic
inserters and extractors for the I/O stream, if they are object
relations then to make them plain old data their entire relation of
object state is relativized to local coordinates with default to names
to give the objects small offsets in a cache line layout. So, with a
difference between, for example integer and floating point types where
those built-in types of the language could be in separate arithmetic
processing units including different clocks, the runtime's profile on
its alignment of those types can maintain object translation profiles
for writing the data inline from the maintained record.

Then, there is a consideration how to define records in terms of:
existing records in C/C++. Basically, it's an integer. So the types
of the data structures like the buffers with the extent or the string
with its extent have lots of semantics about the strings because
compared to binary data more often in the content prefixed buffers,
there is no reason to recount the length of the string in the record
when it is constant.

Yet, to indicate the intent of the programmer in making a thing about
the type that is the string type but also the mapping of the semantics
of the string type

Using the ostream semantics with the overloaded operator that way is
not the same as the const return of the reference

Then, here is a point about templates from before about generating the
templates even though they aren't valid, so when they're canceled,
they actually fit, for the automatic generation of the bridge
functions that on the translations input >> output the object
representations are maintained so it is actually a const reference to
the object state. The templates are interpreted but they aren't
generated until they're used, so, in the forward definitions, they are
for later definitions. That is just kind of pointless, looking for a
bug in the undefined behavior. Really it's probably reasonable to
figure that on the template evaluation maybe there could be the file
specification, where then defining a global general reference function
for only the auto generated compositors for the insertion and
extraction would then have the auto generated templates that exist
based upon the inherent semantics have the same expression

while (in)
{
    class in_exception: io_exception;
    try
    {
        in >> records;
    }
    catch (in_exception&)
    {

    }
};

Then, have the operation on the records, serially read the input files
and there have validators on all the inputs. The validator semantics
for the data types then are about the content layout. With the
tabular data, these are constant reads of records where every record
has a value for each field, so each record has an identical field
layout.

Also if the user writes

in >> records;

Then the programmer might want that to automatically express

template istream& operator >> (istream& in, auto& a)
{
    // a is the return value type
    a(in); // <- look, it's class constructor, or function object ()
    operator()(auto& this); // <- is this here, cancel this with template
    inline (); // <- run self
}

The templates generate the definition differently, then consider
canceling along how the construction initializers are generated.

Then, with the template expansion, it is about computing the field
offsets that match those of the struct. The use of the struct keyword
has some kind of serialization about a convention with looking around
for attributes that the function has it as a record with fields. So,
it's size alignments are to be translated simply. The specifications
are to be automatically generated with the hashes on the inputs. Use
principal component analysis on the inputs, then, use that to emit a
code for the state of the small space state. The specifications are
generated thus by linearizing the data. So, the name referencing is
on the fields with the name length vis-a-vis the input record length
with the above types. Also, the program simply maintains its file
access record in reusing the file, with access records on the file.
That's about then, treating the actual input file classes like structs
with using their sizeof and offsets of members in making from their
class definition these object auto-completion templates. That's
about, making a template namespace, then importing the standard C++
headers, and making fro them these templates by including their file
contents inline in the compilation unit of the C++ compiler, also with
the template features. Then, where it is attributed for the input
specification that the natural types to maintain symbolic reference
along the inputs and then there are advisement functions on the built-
in semantics of the type, to naturally serialize and reload the
staticly generated memos, then the memos can bank swap with the other
ones in the memory transactions around the memory barriers or memory
boundaries. They're static, so it is not convenient to realign the
stack for them to maintain more records than a small fixed amount of
records. Yet, if it is the same file name, for example, as an input
at the beginning of the program, came in as an istream for the
component library to read and write, if the memo compares the same
file name input and knows it's a constant input then it should use the
facility to check the input for constancy before reading it if the
dependent components would be unchanged when it's constant. Then, the
error or feedback or error handling has that the file has to be re-
read, or rather, that the difference must be known, the difference of
the states of the data that the files represent in terms of the
intrinsic semantics the guarantee of the related input data to all the
functions that depend on reading the file. Otherwise, if it just
knows a difference to the file, for example that it is just a line
appended to the end of the file at a given offset, then maybe lines
should be cache lines.

That brings up the utility of static result memoization for iterator
constancy along shared iterators. The compositors develop single
iterator pass components. The presence of the row vectors should tree
back to the records. What that means, is, the input record validators
on the localized object layout with the constancies on their
references maintain the satisfying input record identity. The idea
here is to build up from the file specification, as an example of the
input data, the memoized structures even for that file, where the
automatic types in the program advise that the program inputs are
interpreted. Then, where the automatic and supported types can
maintain the mutual satisfiability of their access then the results
don't need to be computed about vectorizing the input transform
instructions, so the compositor has total freedom to serialize input
from the collected I/O operations. Then, there is a consideration
about there being the empty handler on catching the io_exception
reference. If it's the empty handler, then, it just catches the
particular io_reference then in terms of the const referency of the
handling it tosses any other exception reference or non-io_reference
exception. Other exception handlers might expect that any function
between it and the handler would not do anything, but that's not
necessarily the same as having an empty block in the handler.

Now, there are the standard input extractors for the built-in types
including maybe pointers but the references are to be avoided. As
well, any object defined or templates-that-fit-the-templates extractor
is an automatic compositor.

Then, what seems worthwhile is to figure out how to present the
standard objects to the standard algorithms for template generation so
that then the automatic layout of the standard objects maintains in-
place efficient I/O reads of the data. That way, the istream that is C
++'s std::istream has a templatized subclass that is an input
abstraction so that when users of the standard I/O streams use the
default built-in I/O stream semantics

template <typename T> class input : T;

input<ifstream> in(input_file);

then has maybe the dual semantics for the first constructor variable
of the auto-layout type where the references have their extents
computed and const maintained so the input here is input_file, but the
input_file could also be the specification, generating converters to
constructor parameters, because in other cases the constructor of the
input template would better be initialized otherwise.

Excuse me, I digress from describing how maintaining the input
comparators is to work with the specification of the data in terms of
the fields and records thus that on constancy their processing results
are maintained, because the templates use the const reference paths
instead of non-const reference paths and then there is the idea of the
maintaining the const semantics in the non-const semantics.

Then, for the blocks, they could be in the const dependency chains
naturally, where, when a function-static access is in

operator >> (istream& in, data& d)
{
    // maintain the const reference so to advise the caller that inputs
are idempotent
}

then

operator >> (const istream& in, data& d)

then consideration of

operator << (ostream& out, const data& d); // <- data is const or non-
const
operator << (ostream& out, data& d);

there are questions about how the compiler organizes references, but
actually that is probably defined behavior, about the priority of
selection of matching the function prototype for the function overload
by the parameter list, and as well warning on conventions.

Here, then I have introduced a data type, don't think a forward
definition. The data type here, now that is basically a collection of
records, but, it's also maybe to be compatible with the semantics of
raw pointers so that somehow it could define that it doesn't realias
pointers into the offsets of function objects, maybe exporting
function objects.

I hope that this doesn't seem an unreasonable course of consideration,
I wrote it today and I've read it several times.

Thanks,

Ross F.