Segmentation Fault When Doubling Domain and Mesh Size

[tambeyash]

Hello xcompact3d team and community,

I am new to computational fluid dynamics and I am currently learning about the xcompact3d library.
I previously ran the Channel example successfully with the domain and mesh configuration as in the examples/channel/input_DNS_Re180_LR_explicittime.i3d. For my latest run, I doubled the domain size in all three directions and also doubled the mesh size in each direction, so that the grid spacings (dx, dy, dz) remain constant. Other parameters were kept the same. However, this time my simulation crashed with a segmentation fault. Here is the relevant portion from the log:

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 3 PID 2805614 RUNNING AT cn0001
=   KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 19 PID 2805630 RUNNING AT cn0001
=   KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 23 PID 2805634 RUNNING AT cn0001
=   KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 31 PID 2805642 RUNNING AT cn0001
=   KILLED BY SIGNAL: 11 (Segmentation fault)
====================================================================================

The previous (smaller) simulation ran fine with the same computational resources.

All that changed is the domain and mesh size (both doubled in every direction).

Input files and parameters were updated for the new domain and mesh.

Questions:

• Are there any guidelines for scaling up the domain and mesh with respect to MPI ranks and decomposition to avoid such segmentation faults?

• Could this be due to insufficient memory per rank, or are there other common issues to check when increasing simulation size?

• Is there a recommended way to troubleshoot these MPI segmentation faults in xcompact3d?

Any advice or references to similar cases would also be appreciated!

Thanks in advance for your help.

Let me know if you’d like to add more information or specify your hardware/environment details.

[mathrack]

Hello,

The channel case is a case with a non-regular grid. Thus, dy is not constant. If the default is : nx, ny, nz and Lx, Ly, Lz. Then you should double the domain size only in the periodic direction : 2nx, ny, 2 nz and 2 Lx, Ly, 2 Lz

Of course, it is also possible to change the domain size and the mesh in the y direction. However, the direction y is not periodic. Thus, the number of cells should be ny = n+1 with n a number efficient for FFT (power of 2 is the best)

I think there was a segmentation fault because the initial ny is odd and the new one is even. In this case, ny should always be odd.

Best regards,
Mathrack

[tambeyash]

I did keep ny as odd. The default is nx,ny,nz = 128,65,64 respectively. I kept ny,ny,nz=256,129,128 respectively.

[mathrack]

Please provide the exact diff between input_DNS_Re180_LR_explicittime.i3d and your input file.

Please also provide the output of the code. How many time steps for the segmentation fault to appear ?

EDIT : and the number of CPUs and if you have a release / debug build. Basically the data alllowing to reproduce the segmentation fault.

[tambeyash]

Original
! Mesh
nx=128                ! X-direction nodes
ny=65                 ! Y-direction nodes
nz=64                 ! Z-direction nodes
istret = 2            ! y mesh refinement (0:no, 1:center, 2:both sides, 3:bottom)
beta = 0.259065151    ! Refinement parameter (beta)

! Domain
xlx = 8.              ! Lx (Size of the box in x-direction)
yly = 2.              ! Ly (Size of the box in y-direction)
zlz = 4.              ! Lz (Size of the box in z-direction)

My input file
! Mesh
nx=256                ! X-direction nodes
ny=129                 ! Y-direction nodes
nz=128                 ! Z-direction nodes
istret = 2            ! y mesh refinement (0:no, 1:center, 2:both sides, 3:bottom)
beta = 0.259065151    ! Refinement parameter (beta)

! Domain
xlx = 16.              ! Lx (Size of the box in x-direction)
yly = 4.              ! Ly (Size of the box in y-direction)
zlz = 8.              ! Lz (Size of the box in z-direction)

Rest all the parameters were kept the same. The output log is in as mentioned in my original question.

EDIT: The segmentation fault occurred after 19550/ 100000 time-steps

[mathrack]

The diff command is supposed to output something like this :

diff input_DNS_Re180_LR_explicittime.i3d ../input_DNS_Re180_LR_explicittime.i3d 
15,17c15,17
< nx=256                ! X-direction nodes
< ny=129                 ! Y-direction nodes
< nz=128                 ! Z-direction nodes
---
> nx=128                ! X-direction nodes
> ny=65                 ! Y-direction nodes
> nz=64                 ! Z-direction nodes
[...]

The output log from your simulation is supposed to look like this :

mpirun -n 6 ./xcompact3d input_DNS_Re180_LR_explicittime.i3d 
 Xcompact3d is run with the provided file -->input_DNS_Re180_LR_explicittime.i3d
 ===========================================================
 ======================Xcompact3D===========================
 ===Copyright (c) 2018 Eric Lamballais and Sylvain Laizet===
 ===Modified by Felipe Schuch and Ricardo Frantz============
 ===Modified by Paul Bartholomew, Georgios Deskos and=======
 ===Sylvain Laizet -- 2018- ================================
 ===========================================================
 Git version        : v5.0-9-gffbc596
 ===========================================================
 Simulating channel
 ===========================================================
 Channel forcing with constant flow rate (CFR)
 Re_cl                  :          4200.000

[...]
many lines
[...]

Segmentation fault

[tambeyash]

The difference:

< nx=256                ! X-direction nodes
< ny=129                 ! Y-direction nodes
< nz=128                 ! Z-direction nodes
---
> nx=128                ! X-direction nodes
> ny=65                 ! Y-direction nodes
> nz=64                 ! Z-direction nodes
22,24c22,24
< xlx = 16.              ! Lx (Size of the box in x-direction)
< yly = 4.              ! Ly (Size of the box in y-direction)
< zlz = 8.              ! Lz (Size of the box in z-direction)
---
> xlx = 8.              ! Lx (Size of the box in x-direction)
> yly = 2.              ! Ly (Size of the box in y-direction)
> zlz = 4.              ! Lz (Size of the box in z-direction)

The starting of output log file:

 Xcompact3d is run with the default file -->input.i3d
 ===========================================================
 ======================Xcompact3D===========================
 ===Copyright (c) 2018 Eric Lamballais and Sylvain Laizet===
 ===Modified by Felipe Schuch and Ricardo Frantz============
 ===Modified by Paul Bartholomew, Georgios Deskos and=======
 ===Sylvain Laizet -- 2018- ================================
 ===========================================================
 Git version        : v5.0-9-gffbc596
 ===========================================================
 Simulating channel
 ===========================================================
 Channel forcing with constant flow rate (CFR)
 Re_cl                  :          4200.000
 xnu                    :        0.00023810
 ===========================================================
 p_row, p_col           :         0       0
 ===========================================================
 Time step dt           :        0.00500000
 Temporal scheme        :        Runge-kutta 3
 ===========================================================
 FreeStream (BC)           :        Off
 ===========================================================
 Theta dot Model           :       Biau
 ===========================================================
 Blowing                   :        Off
 ===========================================================
 Adverse Pressure Gradient :        Off
 ===========================================================
 Probe for Spectra         :        Off
 ===========================================================
 ifirst                 :                 1
 ilast                  :            100000
 ===========================================================
 Lx                     :       16.00000000
 Ly                     :        4.00000000
 Lz                     :        8.00000000
 nx                     :               256
 ny                     :               129
 nz                     :               128
 ===========================================================

Ending portion of output log file:

 ===========================================================
 Time step =  19550/ 100000, Time unit =     97.7500
 UT  0.666665271859267      -1.394807399246467E-006
 DIV U* max mean=  0.180556375198097       1.729217924563013E-003
 DIV U  max mean=  1.512123759539463E-013  1.001891682304998E-014
 U,V,W min= -1.5105979E-02 -0.2130847     -0.3430957
 U,V,W max=  0.9455600      0.2297405      0.2926305
 CFL_x                  :        0.07564480
 CFL_y                  :        0.12230480
 CFL_z                  :        0.02744766
 UT  0.666664434333844      -2.232332822260652E-006
 DIV U* max mean=  4.568163201393492E-002  4.334133518986912E-004
 DIV U  max mean=  1.044615782763714E-013  7.568030694144696E-015
 U,V,W min= -1.5032710E-02 -0.2130928     -0.3431139
 U,V,W max=  0.9455069      0.2297603      0.2925532
 CFL_x                  :        0.07564055
 CFL_y                  :        0.12230098
 CFL_z                  :        0.02744911
 UT  0.666666105751826      -5.609148405749664E-007
 DIV U* max mean=  0.113512511569459       1.080209187881530E-003
 DIV U  max mean=  1.310063169057685E-013  9.125334272581926E-015
 U,V,W min= -1.4857966E-02 -0.2131504     -0.3431620
 U,V,W max=  0.9454154      0.2298810      0.2923798
 CFL_x                  :        0.07563324
 CFL_y                  :        0.12229683
 CFL_z                  :        0.02745296
 Time for this time step (s):   1.145685
 Remaining time:      20 h 28 min
 Elapsed time:         4 h 58 min

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 3 PID 2805614 RUNNING AT cn0001
=   KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 19 PID 2805630 RUNNING AT cn0001
=   KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 23 PID 2805634 RUNNING AT cn0001
=   KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 31 PID 2805642 RUNNING AT cn0001
=   KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

[mathrack]

Are you running the simulation using a job manager such as slurm ? How many CPUs are you using ?

I think we can not reproduce this error because many time steps were performed without issue.

You can try to build a debug version of 2decomp (v2.0.4) and compile a debug version of xcompact3d linked against the debug version of 2decomp to obtain more information in case of segmentation fault.

[tambeyash]

I am using the pbs. I will try your suggestion, and run the simulation using different parameters, and see if I get the same result/error.