Non Blocking Communication in MPI-1 and Modes
Nonblocking communications allow the overlap of computation
and communication - typically, large numbers of operations
could go on during the time it takes to send a message ... or
at least, that is likely what you will have found out from
Exercise 2.
A non-blocking point-to-point communication typically has the form
- Post the operation and immediately return.
- Do some other (local) work, such as computations.
- Probe for completion of the communication.
This allows overlap of communication and computation, particularly on systems
where a PE consists of two processors, one responsible for computation and
the other handling communications.
When using nonblocking send/receives, MPI uses a handle of type
MPI_Request to give something that you can later on test against. So the
invocation
MPI_Isend(void *msg, int count,
MPI_Datatype datatype,
int dest,
int tag,
MPI_Comm comm,
MPI_Request *request);
will return the handle in the variable "request".
You should not mess with "request" except through MPI function calls.
After posting the nonblocking send, you can later on wait for completion
of the send by posting
MPI_Wait(MPI_Request *request, MPI_Status *status)
A similar procedure is used for MPI_Irecv().
You can also just check for completion, rather than having to
wait for completion, by:
MPI_Test(MPI_Request *request,
int *flag,
MPI_Status *status);
Here flag = true if the operation indicated by "request" is complete,
false otherwise. You would typically use a test instead of a wait
when there are multiple messages pending, and you want to process
them as soon as they come in - rather than in a somewhat arbitrary
order typically imposed by a loop.
Other sorta useful variations on this are
- MPI_Waitany(): gives an array of handles, and blocks until
any one of them completes. On return, tells you which one
them triggered the return.
- MPI_Waitall(): blocks until all operations in the request block complete.
- MPI_Testall(), MPI_Testany(): these are the testing versions of the
previous two.
There is a zoo of MPI functions that can be used: MPI_Waitsome, MPI_Testsome,
etc. However, it is best to learn the basic ones and understand how
they work, than to try to learn all of the operations. Furthermore, clearly
you could implement a (probably inefficient) version of MPI_Waitany() using
MPI_Test() - and that is just what many vendors have done.
MPI Modes
As we have seen, blocking and non-blocking do not necessarily correspond
to what naive thinking might indicate. This is because in MPI, the
implementation
has been carefully separated from the logical view of the library.
So where you may think of particular implementations when you think of
blocking/nonblocking, there is no such guaranteed mapping.
Communication Modes in MPI
Much closer to the implementation is the mode used. Until now
we have used standard mode, but there are three others. Here are the essentials:
- Standard mode is the default. The vendor's implementation decides if
outgoing messages are buffered by the system. If buffered, blocking send
can complete before the corresponding receive is posted.
If not, blocking send cannot complete until the corresponding receive is
posted and the entire message is moved to the destination. This means
that MPI code that works alright on one machine may deadlock on another,
if you have not carefully arranged the send/receive pairs.
- Buffered mode has the user allocating buffer space for messages,
taking the place of a possible system buffer. Here a blocking send can complete
before the corresponding receive is posted, since once the message is
copied to the buffer the user is safe to overwrite the message space.
The function MPI_BUFFER_ATTACH() attaches a buffer for MPI to use (which
only allows one at a time).
- Synchronous mode allows a blocking send to start before the
corresponding receive is posted, but can complete only if the receive
operation has started. If the receive is also synchronous, this forces
both processes to sychronize at the communication point.
- Ready mode allows the send to start only if the matching receive
has already been posted. It completes when the send buffer is empty. This
allows the system to possibly do a direct copy of the outgoing message
to the receiving message buffer, without doing any system buffering.
Ready mode is dangerous; if the corresponding receive has not been posted,
then the behaviour is undefined: the message may be delivered later, discarded
silently, or a core dump results at the choice of the MPI implementor.
The last three are for the send functions, and consist of prepending a
B,S,or R to the corresponding send function. They all use the same corresponding
receive function as for standard mode. All four modes have blocking and nonblocking
versions.
- Last Modified: Thu 08 Feb 2018, 07:17 AM