-
Notifications
You must be signed in to change notification settings - Fork 1
/
ghosts.h
194 lines (163 loc) · 8.52 KB
/
ghosts.h
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
// This file is part of the ESPResSo distribution (http://www.espresso.mpg.de).
// It is therefore subject to the ESPResSo license agreement which you accepted upon receiving the distribution
// and by which you are legally bound while utilizing this file in any form or way.
// There is NO WARRANTY, not even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
// You should have received a copy of that license along with this program;
// if not, refer to http://www.espresso.mpg.de/license.html where its current version can be found, or
// write to Max-Planck-Institute for Polymer Research, Theory Group, PO Box 3148, 55021 Mainz, Germany.
// Copyright (c) 2002-2009; all rights reserved unless otherwise stated.
#ifndef GHOSTS_H
#define GHOSTS_H
/** \file ghosts.h Ghost particles and particle exchange.
In this file you find everything concerning the exchange of
particle data (particles, ghosts, positions and forces) for short
range interactions between the spacial domains of neighbouring
nodes.
<h2> How does this work </h2>
The ghost communication transfers data from cells on one node to cells on another node during
the integration process. Note that data can only be transferred from one cell to one other, and
the contents of the other cell will be overwritten. This communication is invoked by the integrator,
at four different times in an integration step. These steps are reflected in \ref cell_structure structure,
since they are treated differently by different cell systems:
<ul>
<li> ghost_cells is used to transfer the cell sizes, i. e. makes sure that for all later transfers
the target cell has the same size as the source cell.
<li> exchange_ghosts sets up all information on the ghosts that is necessary. Normally transfers the
(shifted) position and particle properties.
<li> update_ghosts is used to update the particle properties if no particles have been moved between cells.
Therefore only the positions are transferred, otherwise this normally looks pretty much like the
exchange_ghosts.
<li> collect_ghost_forces finally is used to transfer back forces that were exerted on a ghost particle. They
are simply added again to the force of the real particle. The communication process is therefore inverted
with respect to exchange_ghosts and update_ghosts, i. e. sending is replaced by receiving and the other way
round.
</ul>
The particle data that has to be transferred, and especially from where to where, heavily depends on the cell system.
In Espresso, this is abstracted in form of ghost communicators (\ref GhostCommunicator) and ghost communications
(\ref GhostCommunication). The ghost communicators represent the four communications above and consist of the data to
transfer (which is determined by their type) and a list of ghost communications.
The data types are described by the particle data classes
<ul>
<li> GHOSTTRANS_PROPRTS transfers the \ref ParticleProperties
<li> GHOSTTRANS_POSITION transfers the \ref ParticlePosition
<li> GHOSTTRANS_POSSHFTD is no transfer class, but marks that an additional vector, the shift, has to be
added to the positions. Can only be used together with GHOSTTRANS_POSITION.
<li> GHOSTTRANS_MOMENTUM transfers the \ref ParticleMomentum
<li> GHOSTTRANS_FORCE transfers the \ref ParticleForce
<li> GHOSTTRANS_PARTNUM transfers the cell sizes
</ul>
Each ghost communication describes a single communication of the local with another node (or all other nodes). The data
transferred can be any number of cells, there are five communication types:
<ul>
<li> GHOST_SEND sends data to one other node
<li> GHOST_RECV recvs data from one other node. In the case of forces, they are added up, everything else is overwritten
<li> GHOST_BCST sends data to all nodes
<li> GHOST_RDCE recvs data from all nodes. In the case of forces, they are added up, everything else is overwritten
<li> GHOST_LOCL transfer data from a local cell to another local cell. In this case, the first half of the cells are the sending cells,
the other half are the corresponding receivers.
</ul>
Note that for the first four communications you have to make sure that when one node sends to another, that the sender has GHOST_SEND and the
receiver GHOST_RECV. In the case of GHOST_BCST rsp. GHOST_RDCE, all nodes have to have the same communication type and the same master
sender/receiver (just like the MPI commands).
A special topic are GHOST_PREFETCH and GHOST_PSTSTORE. For example, if all nodes broadcast to the other, the naive implementation will be that
n_nodes times a GHOST_BCST is done with different master nodes. But this means that each time n_nodes-1 nodes wait for the master to construct
its send buffer. Therefore there is the prefetch flag which can be set on a pair of recv/send operations. If the ghost communication reaches a
recv operation with prefetch, the next send operation (which must have the prefetch set!!) is searched and the send buffer already created.
When sending, this precreated send buffer is used. In the scenario above, all nodes create the send buffers simultaneously in the first
communication step, thereby reducing the latency a little bit. The pststore is similar and postpones the write back of received data until a
send operation (with a precreated send buffer) is finished.
The ghost communicators are created in the init routines of the cell systems, therefore have a look at \ref dd_topology_init or
\ref nsq_topology_init for further details.
*/
#include <mpi.h>
#include "particle_data.h"
/** \name Transfer types, for \ref GhostCommunicator::type */
/************************************************************/
/*@{*/
/// send to a single node
#define GHOST_SEND 0
/// recv from a single node
#define GHOST_RECV 1
/// broadcast, the node entry gives the sender
#define GHOST_BCST 2
/// reduce, the node entry gives the receiver
#define GHOST_RDCE 3
/// transfer data from cell to cell on this node
#define GHOST_LOCL 4
/// mask to the job area of the transfer type
#define GHOST_JOBMASK 15
/// additional flag for prefetching
#define GHOST_PREFETCH 16
/// additional flag for poststoring
#define GHOST_PSTSTORE 32
/*@}*/
/** \name Transfer data classes, for \ref GhostCommunication::type */
/************************************************************/
/*@{*/
/// transfer \ref ParticleProperties
#define GHOSTTRANS_PROPRTS 1
/// transfer \ref ParticlePosition
#define GHOSTTRANS_POSITION 2
/** flag for \ref GHOSTTRANS_POSITION, shift the positions by \ref GhostCommunication::shift.
Must be or'd together with \ref GHOSTTRANS_POSITION */
#define GHOSTTRANS_POSSHFTD 4
/// transfer \ref ParticleMomentum
#define GHOSTTRANS_MOMENTUM 8
/// transfer \ref ParticleForce
#define GHOSTTRANS_FORCE 16
#ifdef LB
/// transfer \ref ParticleLatticeCoupling
#define GHOSTTRANS_COUPLING 32
#endif
/// resize the receiver particle arrays to the size of the senders
#define GHOSTTRANS_PARTNUM 64
/*@}*/
/** \name Data Types */
/************************************************************/
/*@{*/
typedef struct {
/** Communication type. */
int type;
/** Node to communicate with (to use with all MPI operations). */
int node;
/** MPI communicator handle (to use with GHOST_BCST, GHOST_GATH, GHOST_RDCE). */
MPI_Comm mpi_comm;
/** Number of particle lists to communicate. */
int n_part_lists;
/** Pointer array to particle lists to communicate. */
ParticleList **part_lists;
/** if \ref GhostCommunicator::data_parts has \ref GHOSTTRANS_POSSHFTD, then this is the shift vector.
Normally this a integer multiple of the box length. The shift is done on the sender side */
double shift[3];
} GhostCommunication;
/** Properties for a ghost communication. A ghost communication is defined */
typedef struct {
/** Particle data parts to transfer */
int data_parts;
/** number of communication steps. */
int num;
/** List of ghost communications. */
GhostCommunication *comm;
} GhostCommunicator;
/*@}*/
/** \name Exported Functions */
/************************************************************/
/*@{*/
/** Initialize a communicator. */
void prepare_comm(GhostCommunicator *comm, int data_parts, int num);
/** Free a communicator. */
void free_comm(GhostCommunicator *comm);
/** Initialize ghosts. */
void ghost_init();
/** do a ghost communication */
void ghost_communicator(GhostCommunicator *gc);
/** Go through \ref ghost_cells and remove the ghost entries from \ref
local_particles. Part of \ref dd_exchange_and_sort_particles.*/
void invalidate_ghosts();
#ifdef GHOST_FLAG
MDINLINE int ifParticleIsGhost(Particle *p){
return p->l.ghost;
}
#endif
/*@}*/
#endif