MPI_Gather(void *sendbuf, /* starting address of send buffer */ int sendcount, /* number of elements in send buffer */ MPI_Datatype sendtype, /* data type of send buffer elements */ void *recvbuf, /* address of receive buffer */ int recvcount, /* number of elements for any single receive */ MPI_Datatype recvtype, /* data type of recv buffer elements */ int root, /* rank of reciving process */ MPI_Comm comm) /* communicator group */Almost always recvtype and sendtype are the same, and sendcount and recvcount will be equal. The last two specify the number of elements from each process, not the total amount of data received. This means, e.g., if sendcount = recvcount = 1, then p data items will be accumulated in recvbuf. They will also be accumulated in rank order.
MPI_Allgather() does the similar thing, except that the data is accumulated into recvbuf on every process in the communicator. There is a shortcoming with MPI_Allgather(); the sendcount must be identical on every process. For the CG algorithm with the order n of the problem not a multiple of the number of processes, that means you would need to "pad" one or more segments of the gathered vector so that they all have the same size. To handle this, there is a vector variant:
MPI_Gatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, int root, MPI_Comm comm)Here recvcount has been replaced by a vector recvcounts, such that recvcounts[i] contains the number of entries sent by process i. It also allows you to place the data almost wherever in the recvbuf you desire, using the displs[] array. The data sent from process i is placed in the ith portion of the receive buffer recvbuf on process root. The ith part of recvbuf begins at offset displs[i] elements into recvbuf. The receive buffer is ignored for all non-root processes, but should be provided.
WARNING: This operation must not cause any overlapping of writes in the recvbuf on root. That means you cannot allow the specification of counts, types, and displacements to let two different processes to have part of their send data writing to the same location in recvbuf.
The "All" version of MPI_Gatherv is straightforward, given the above ideas.
When you specify only a single datatype, as with MPI_Send, but a sendcount larger than 1, then MPI creates the type signature as the concatenation of sendcount copies of the MPI datatype specified.
For collective communications, the type signatures of all participating processes must be identical.