- Posts: 3
Domain decomposition Error
- Marius
-
Topic Author
- Offline
- Fresh Boarder
Less
More
5 years 6 hours ago #8117
by Marius
Domain decomposition Error was created by Marius
Dear Martini-User,
during my preparations for a bunch of coarse grained 'Peptide-Membrane"-Simulations a Error occurred. If I start my mdrun simulations I get this message:
Fatal error:
There is no domain decomposition for 20 ranks that is compatible with the given box and a minimum cell size of 3.28188 nm
Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
Look in the log file for details on the domain decomposition
For more information and tips for troubleshooting, please check the GROMACS
website at www.gromacs.org/Documentation/Errors
I use Gromacs 5.1.2 on a PC with 20 threads. The membrane consists of three different molecules (POPC, CHOL, MTSM (Sphingomyelin with 16:0/16:0 Chains)), which I generated with insane.py. There are 196 lipids in each leaflet and the box size is 9x9x11nm. The minimization and the grompp command worked well, but the mdrun command doesn't. The peptid is constrained during the equilibration.
Tried Solutions:
-Somebody had the same problem before, but the solution doesn't work for me (set -rdd 1.4).
-If I set -nt 12 it works fine, but maybe there is a solution where I can use all of my threads
-If I set -rcon 1.84 manually, instead of using the estimated value 2.6 from gromacs, it works fine. But I am not sure if this influence the outcome of my simulations.
Maybe some of you know the problem or maybe you can help me with my question about -rcon.
Thank you very much for your time and your effort.
With regards,
Marius
during my preparations for a bunch of coarse grained 'Peptide-Membrane"-Simulations a Error occurred. If I start my mdrun simulations I get this message:
Fatal error:
There is no domain decomposition for 20 ranks that is compatible with the given box and a minimum cell size of 3.28188 nm
Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
Look in the log file for details on the domain decomposition
For more information and tips for troubleshooting, please check the GROMACS
website at www.gromacs.org/Documentation/Errors
I use Gromacs 5.1.2 on a PC with 20 threads. The membrane consists of three different molecules (POPC, CHOL, MTSM (Sphingomyelin with 16:0/16:0 Chains)), which I generated with insane.py. There are 196 lipids in each leaflet and the box size is 9x9x11nm. The minimization and the grompp command worked well, but the mdrun command doesn't. The peptid is constrained during the equilibration.
Tried Solutions:
-Somebody had the same problem before, but the solution doesn't work for me (set -rdd 1.4).
-If I set -nt 12 it works fine, but maybe there is a solution where I can use all of my threads
-If I set -rcon 1.84 manually, instead of using the estimated value 2.6 from gromacs, it works fine. But I am not sure if this influence the outcome of my simulations.
Maybe some of you know the problem or maybe you can help me with my question about -rcon.
Thank you very much for your time and your effort.
With regards,
Marius
Please Log in or Create an account to join the conversation.
- bart
-
- Offline
- Admin
Less
More
- Posts: 98
5 years 5 hours ago - 5 years 5 hours ago #8118
by bart
Replied by bart on topic Domain decomposition Error
In short, try different amounts of mdrun -nt 'your_number' and use the one which works. GROMACS domain decomposition is pretty solid and usually fiddling with it yields little to no improvement. Below is the long answer.
First I will state some basic principles, we will then use those principles to see what is going on.
1) A true MD simulation using perfect math would scale at n^2 (every particle would interact with every particle). To prevent this poor scaling we introduce a cut-off and only calculate particle-particle interactions within the cut-off this results in linear scaling of the code with respect to the amount of particles.
2) Domain decomposition is the slicing of your complete box into its sub domains. Each threat could possible get one of those chunks. However, since there are interactions which could span the boundary of a box, the neighboring boxes are all also taken into account. Therefore each threat gets a box and the directly neighboring boxes. This will only result in no errors if the size of the original part*2 > cut-off. (if this would not be the case it could be that I would have to calculate an interaction between a particle that is in the active domain and a particle that is even one box further away than my neighbor -- of which I have no knowledge as the active threat for the target domain.)
Now we know a little about what is going on, we can see that the actual size of the box can be limiting on the maximum domain decomposition. For if I add more domains on a box of the same size, the individual domains will have to become smaller. With the danger that they become smaller than 2x the cut-off. If this happens you are sure to fail. Thus a specific box size has a theoretical upper limit when it comes to the amounts of threats you can use (divide x,y,z by cut-off) using this scheme of domain decomposition. There is however one possible solution, which is possible using the MPI scheme of gromacs, where you put more than one core on the same domain. This will result in more communication and does not always speed things up as much as you would like, however, this is the only thing left. (or simple turning off hyper threading might also improve things in these situations!)
I hope this was helpful and cheers!
First I will state some basic principles, we will then use those principles to see what is going on.
1) A true MD simulation using perfect math would scale at n^2 (every particle would interact with every particle). To prevent this poor scaling we introduce a cut-off and only calculate particle-particle interactions within the cut-off this results in linear scaling of the code with respect to the amount of particles.
2) Domain decomposition is the slicing of your complete box into its sub domains. Each threat could possible get one of those chunks. However, since there are interactions which could span the boundary of a box, the neighboring boxes are all also taken into account. Therefore each threat gets a box and the directly neighboring boxes. This will only result in no errors if the size of the original part*2 > cut-off. (if this would not be the case it could be that I would have to calculate an interaction between a particle that is in the active domain and a particle that is even one box further away than my neighbor -- of which I have no knowledge as the active threat for the target domain.)
Now we know a little about what is going on, we can see that the actual size of the box can be limiting on the maximum domain decomposition. For if I add more domains on a box of the same size, the individual domains will have to become smaller. With the danger that they become smaller than 2x the cut-off. If this happens you are sure to fail. Thus a specific box size has a theoretical upper limit when it comes to the amounts of threats you can use (divide x,y,z by cut-off) using this scheme of domain decomposition. There is however one possible solution, which is possible using the MPI scheme of gromacs, where you put more than one core on the same domain. This will result in more communication and does not always speed things up as much as you would like, however, this is the only thing left. (or simple turning off hyper threading might also improve things in these situations!)
I hope this was helpful and cheers!
Last edit: 5 years 5 hours ago by bart.
Please Log in or Create an account to join the conversation.
- Marius
-
Topic Author
- Offline
- Fresh Boarder
Less
More
- Posts: 3
5 years 5 hours ago #8119
by Marius
Replied by Marius on topic Domain decomposition Error
Thank you very much for your insightful explanations!
I think, that helps.
With regards,
Marius
I think, that helps.
With regards,
Marius
Please Log in or Create an account to join the conversation.
Time to create page: 0.092 seconds