Poor Performance on Small Files
From Tom:
Reading from Bebop filesystem
[turam@bdw-0319 turam]$ python time_gio.py 03_31_2018.OR.401.coreproperties x
Columns of data to read:
['x']
Runtime to read in 1 properties of 7.54e+08 halos = 33.084 seconds
Runtime to load the same data from a Numpy binary = 1.462 seconds
For fname = `03_31_2018.OR.401.coreproperties`,
gio reads 22.63x slower than numpy
Reading from /dev/shm:
[turam@bdw-0319 shm]$ python time_gio.py 03_31_2018.OR.401.coreproperties x
Runtime to read in 1 properties of 7.54e+08 halos = 4.730 seconds
Runtime to load the same data from a Numpy binary = 2.116 seconds
For fname = `03_31_2018.OR.401.coreproperties`,
gio reads 2.24x slower than numpy
Using a simple c++ genericio code:
mpiexec -n 64 /home/turam/dev/genericio//mpi/gioread 03_31_2018.OR.401.coreproperties x
Read 1 variables from 03_31_2018.OR.401.coreproperties (3016146524 bytes) in 0.682227s: 4216.22 MB/s [excluding header read]
Potential Solutions
- Copy small files into /dev/shm before reading.
- As an optimization, GenericIO can do one large read if reading the whole file.
- Use GenericIORewrite to rewrite them for a small number of ranks (maybe do this as a standard "cleanup" after jobs for small fies. Needs to be careful about correlations between rank blocks between files.
- Implement a GenericIO interface that uses readv to avoid as many file-system syscalls.
- We can also change the subfiles used for smaller files.
- Note: The python interface opens/closes each file a lot. That can be cleaned up.