Re: Distributing a Task
O. Olson wrote, quoted or indirectly quoted someone who said :
I have a (computational) task that can be easily parallelized due to
lack of dependencies. I have written this in Java, and I am using
multiple threads on a single computer to get it done.
Roedy Green <see_webs...@mindprod.com.invalid> wrote:
see <http://mindprod.com/jgloss/distributedcomputing.html>
O. Olson wrote:
Thanks Roedy. I think the Java Parallel Processing Framework
http://www.jppf.org/index.php is what I wanted. I have not tried it
out though but it looks like it will do what I want.
If you roll your own, you can do it fairly readily with RMI or web services,
or more robustly but with a little more difficulty with Jini or message
queues. Internodal bandwidth and latency are significant concerns regardless
of framework or other solution - you must ensure that gains from
parallelization offset the management and communication costs between
cooperating tasks. Your scenario seems well-matched for this treatment.
Point-to-point messages passed with send-receive-reply semantics are very
effective at coordinating internodal tasks.
Consider a task administrator, Adam, that receives messages from task agents
spread across the LAN or WAN. Adam listens, say as a socket server, for a
message that says, "I am available" from a distributed worker, Willy. Adam
replies to Willy with a work paquette - a set of inputs to be processed in
parallel.
Adam also receives "work done" paquettes - "available" messages with attachment.
Adam is very fast at handling messages from workers - it just replies with
work paquettes to "available" messages and puts work results into an output bin.
When Adam runs out of work to distribute, it holds off on replies to the
workers, who languish in wait state for replies with work paquettes.
Adam is limited by capacity to hold pending worker messages. It can reply
with a "quit" paquette to order a worker to shut down. It can birth new
workers if demand goes up. Its strategies are essentially those of work
queues in the regular concurrent libraries.
On the drain side, Adam is also a listener. The client for processed chunks,
Clara, sends to Adam for completed paquettes, and Adam replies with results as
available. The same considerations apply on the output side about worker pool
management and suspension of replies until output is available. In many
typical scenarios the output pool is of fixed size, a single channel or very
few, as in visual displays. The worker pool is dynamic, availing itself of
resources in parallel as possible or necessary.
In this model, Adam is a listener only. Clients on input (worker or producer)
and output (consumer) send to Adam for a reply with work to do or that's been
done, respectively.
--
Lew