Re: Can extra processing threads help in this case?
"Hector Santos" <sant9442@nospam.gmail.com> wrote in message
news:O%23HmXKf2KHA.5212@TK2MSFTNGP04.phx.gbl...
Peter Olcott wrote:
http://en.wikipedia.org/wiki/Priority_inversion
If there are no shared resources then there is no
priority inversion.
Try and provide a valid counter-example of priority
inversion without shared resources.
You don't to have a dead lock to reveal problems. You can
get Race Conditions with classic SYNC 101 mistakes like
this that depends on time synchronizations:
if (NumberOfHighPriorityJobsPending !=0)
nanosleep(20);
20 milliseconds
Since you like wikipedia, read:
http://en.wikipedia.org/wiki/Race_condition
Whats the point of the above? Are you expecting that the
value will turn 0 in the nanosleep(20) which is wrong
anyway. Is that 20 seconds or 20 nanaseconds? Did you
really mean?
if (NumberOfHighPriorityJobsPending !=0)
usleep(20);
In either case, you are are in for a RUDE awakening with
that.
You probably mean:
while (NumberOfHighPriorityJobsPending !=0)
usleep(20);
which COULD be fine, but you should use an optimized
kernel object here to wait on.
if (WaitForSingleObject(hPriorityEvent, INFINITE) ==
WAIT_OBJECT) {
/// do whatever
} else {
/// Not what I expected
}
When you wait on a kernel object, you won't be spinning
your thread like you do above.
Event driven is better. I would prefer that the high
priority jobs has absolute priority over the lower priority
jobs. Even better would be if this could be done
efficiently. I think that process priority would work well
enough. That would depend on how the kernel scheduler works,
the frequency and duration of the time slices.
You are not explaining with much of any reasoning why you
think that one alternative is better than another, and
when I finally do get you to explain, it is only that
your alternative is better than your misconception of my
design, not the design itself.
No, your problem is that you are stuck with a framework
One design constraint that won't be changed until system
load requires it is that we must assume a single core
processor with hyperthreading.
Many Threads to 1 FIFO/OCR process
and everyone is telling you its flawed and why. I'm tried
different
When they finally get to the why part I point out their
false assumption. A priority Queue may be a great idea with
multiple cores, I will not have those.
ways using your WORK LOAD which you accepted and began to
change your TPS.
But you still going to overflow your Many to 1 design,
especially if you expect to use TIME to synchronize
everything.
This is not a given, but, using time to synchronize is not
the best idea. It could possibly waste a lot of CPU. So then
four processes with one getting an 80% of the relative share
and the other three sharing about 7%.
Exactly what are these ways, and precisely what have I
failed to account for?
You been told in a dozen ways why it will fail! You are
OFF in your timing of everything for the most part. You
think you can achieve what you want with a Many Thread to
1 OCR process design at the TPS rates and work load you
think you can get.
You can't!
Four processes four queues each process reading only from
its own queue. One process having much more process priority
than the rest. Depending upon the frequency and size of the
time slices this could work well on the required single core
processor.
On a quad-core it would have to be adapted possibly using a
single priority queue so that the high priority jobs could
possibly be running four instances at once.
I know full and well that the biggest overhead of the
process is going to be disk access. I also know full and
well that tripling the number of disk access would likely
triple overhead. I am not sure that SQLite is not smart
enough to do a record number based seek without requiring
an index. Even if SQLite is not smart enough to do a
record seek without an index, it might still be fast
enough.
This is what I am saying, WE TOLD YOU WHAT THE LIMITS OF
SQLITE are and you are not listening. You can do a ROW
lookup, but you can't do a low level FILE RECORD POSITION
AND BYTE OFFSET like you think you need, but really don't.
As long as the ROW lookup maps to the file byte offset we
are good. If the ROW lookup must read and maintain an index
just to be able to get to the rows in sequential order, this
may not be acceptable.
I also told you that while you UPDATE an SQLITE database,
all your READS are locked!
You refuse to comprehend that.
I knew this before you said it the first time. The practical
implications of this is that SQLite can't handle nearly as
many as simultaneous updates as other row locking systems.
Their docs said 500 transaction per second.
Again, you can SELECT a row in your table using the proper
query, but it isn' a direct FILE ACCESS with BYTE OFFSET
idea and again, SQLITE3 will lock your database during
updates so all your REQUEST SERVER will be locked in
reading/writing any table while it is being updated by
ANYONE.
If it doesn't require a separate index to do this, then the
record number maps to a byte offset. Since record numbers
can be sequenced out-of-order, in at least this instance it
must have something telling it where to go, probably an
index. Hopefully it does not always make an index just in
case someone decides to insert records out-of-sequence.
You (and Hector) are definitely right on some things.
We are right on EVERYTHING discussed here. There has been
nothing you stated or posted that indicates any error in
all suggestions to you.
You and Joe are most often wrong by making false assumptions
about the details of my design and its requirements.
IDEAL: Many Threads to Many Threads
WORST: Many Threads to 1 thread
I guess that I am currently back to alternative two which is
many threads or a web server to four OCR processes via four
FIFOS on a single core machine, one process having much more
process priority than the others.
A multi-core processor would probably involve the same thing
except have a single priority queue in-between.