Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
H
hacc_fft_decomp
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Adrian Pope
hacc_fft_decomp
Commits
6afc008d
Commit
6afc008d
authored
7 years ago
by
Adrian Pope
Browse files
Options
Downloads
Patches
Plain Diff
adding command line utilities and README
parent
171cc1e6
No related branches found
No related tags found
No related merge requests found
Changes
4
Hide whitespace changes
Inline
Side-by-side
Showing
4 changed files
README.txt
+93
-0
93 additions, 0 deletions
README.txt
cl_decomp3d.py
+18
-0
18 additions, 0 deletions
cl_decomp3d.py
cl_ng.py
+19
-0
19 additions, 0 deletions
cl_ng.py
cl_pencils.py
+37
-0
37 additions, 0 deletions
cl_pencils.py
with
167 additions
and
0 deletions
README.txt
0 → 100644
+
93
−
0
View file @
6afc008d
Requirements: python with numpy and sympy (eg. anaconda)
Suppose we want to find problem sizes and configurations for a HACC run on
all of Sequoia with 1 MPI rank per processor:
nranks = 16*1024*96 = 1572864
1) Find ng^3 problem sizes for which co-commensurate decompositions exist.
$ python cl_ng.py
USAGE: cl_ng.py <nranks> <ng0> <ng1> [maxAspectRatio]
nranks: total number of MPI ranks
ng0: min ng to check
ng1: max ng to check
maxAspectRatio: the (optional) maximum ratio between the largest and
smallest number of MPI ranks along a dimension in the 3D decomposition.
$ python cl_ng.py 1572864 10000 30000 5.0
12288
15360
18432
21504
24576
27648
2) List potential 3D decompositions for a particular problem size for which
co-commensurate pencil decompositions exist.
$ python cl_decomp3d.py
USAGE: cl_decomp3d.py <nranks> <ng> [maxAspectRatio]
nranks: total number of MPI ranks
ng: grid points along one side of 3D FFT
maxAspectRatio: the (optional) maximum ratio between the largest and
smallest number of MPI ranks along a dimension in the 3D decomposition.
$ python cl_decomp3d.py 1572864 15360 5.0
(96, 128, 128) 1.333333
(64, 128, 192) 3.000000
(64, 96, 256) 4.000000
3) Enumerate pencil decompositions that are co-commensurate with a particular
3D decomposition for a particular problem size.
$ python cl_pencils.py
USAGE: cl_pencils.py <ng> <nrankx> <nranky> <nrankz>
ng: grid points along one side of 3D FFT
nrankx: number of MPI ranks in the x dimension
nranky: number of MPI ranks in the y dimension
nrankz: number of MPI ranks in the z dimension
$ python cl_pencils.py 15360 96 128 128
ng = 15360
3D
(96, 128, 128)
X-PENCILS
(1, 1024, 1536)
(1, 1536, 1024)
(1, 512, 3072)
(1, 3072, 512)
Y-PENCILS
(1536, 1, 1024)
(3072, 1, 512)
Z-PENCILS
(1536, 1024, 1)
(3072, 512, 1)
4) This can be compared with the output of the CheckDecompsition utility
available in HACC and SWFFT that is based on the actual decomposition.c code.
$ ./CheckDecomposition 15360 15360 15360 1572864 96 128 128
distribution 1D: [1572864:1:1]
distribution 3D: [96:128:128]
2d_z: 1536, 1024, 1.
distribution 2z: [1536:1024:1]
2d_x: 1, 1536, 1024.
distribution 2x: [1:1536:1024]
2d_y: 1536, 1, 1024.
distribution 2y: [1536:1:1024]
This diff is collapsed.
Click to expand it.
cl_decomp3d.py
0 → 100644
+
18
−
0
View file @
6afc008d
import
sys
import
decomp
as
D
if
len
(
sys
.
argv
)
<
3
:
print
(
'
USAGE: %s <nranks> <ng> [maxAspectRatio]
'
%
sys
.
argv
[
0
])
sys
.
exit
(
-
1
)
nranks
=
int
(
sys
.
argv
[
1
])
ng
=
int
(
sys
.
argv
[
2
])
if
len
(
sys
.
argv
)
>
3
:
maxAspectRatio
=
float
(
sys
.
argv
[
3
])
else
:
maxAspectRation
=
0.0
decompList
=
D
.
filterNgNranks
(
ng
,
nranks
,
maxAspectRatio
)
for
decomp
in
decompList
:
print
(
'
%s %f
'
%
(
decomp
[
1
],
decomp
[
0
]))
This diff is collapsed.
Click to expand it.
cl_ng.py
0 → 100644
+
19
−
0
View file @
6afc008d
import
sys
import
decomp
as
D
if
len
(
sys
.
argv
)
<
4
:
print
(
'
USAGE: %s <nranks> <ng0> <ng1> [maxAspectRatio]
'
%
sys
.
argv
[
0
])
sys
.
exit
(
-
1
)
nranks
=
int
(
sys
.
argv
[
1
])
ng0
=
int
(
sys
.
argv
[
2
])
ng1
=
int
(
sys
.
argv
[
3
])
if
len
(
sys
.
argv
)
>
4
:
maxAspectRatio
=
float
(
sys
.
argv
[
4
])
else
:
maxAspectRatio
=
0.0
ngList
=
D
.
ngCandidatesNranks
(
nranks
,
ng0
,
ng1
,
maxAspectRatio
)
for
ng
in
ngList
:
print
(
'
%d
'
%
ng
)
This diff is collapsed.
Click to expand it.
cl_pencils.py
0 → 100644
+
37
−
0
View file @
6afc008d
import
sys
import
decomp
as
D
if
len
(
sys
.
argv
)
<
5
:
print
(
'
USAGE: %s <ng> <nrankx> <nranky> <nrankz>
'
%
sys
.
argv
[
0
])
sys
.
exit
(
-
1
)
ng
=
int
(
sys
.
argv
[
1
])
nrankx
=
int
(
sys
.
argv
[
2
])
nranky
=
int
(
sys
.
argv
[
3
])
nrankz
=
int
(
sys
.
argv
[
4
])
t3d
=
(
nrankx
,
nranky
,
nrankz
)
pencils
=
D
.
enumeratePencilsNg3d
(
ng
,
t3d
)
print
(
'
ng = %d
\n
'
%
ng
)
print
(
'
3D
'
)
print
(
'
%s
'
%
str
(
t3d
))
print
(
''
)
print
(
'
X-PENCILS
'
)
for
pencil
in
pencils
[
0
]:
print
(
'
%s
'
%
str
(
pencil
))
print
(
''
)
print
(
'
Y-PENCILS
'
)
for
pencil
in
pencils
[
1
]:
print
(
'
%s
'
%
str
(
pencil
))
print
(
''
)
print
(
'
Z-PENCILS
'
)
for
pencil
in
pencils
[
2
]:
print
(
'
%s
'
%
str
(
pencil
))
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment