diff --git a/README.md b/README.md index 9db28ee694fda879ad80b1638d64954968e687ea..c48a0647800641d332225755a651bc1c40aa1aaa 100644 --- a/README.md +++ b/README.md @@ -72,46 +72,35 @@ The `pygio` library is pip-installable and works with `mpi4py`. **Requirements** -Currently, a **CMake version >= 3.11.0** is required to fetch dependencies -during configuration. The ``pygio`` module also requires MPI libraries to be -findable by CMake's FindMPI. The compiler needs to support **C++17** (make sure -that ``CC`` and ``CXX`` point to the correct compiler) +- Currently, a **CMake version >= 3.11.0** is required to fetch dependencies +during configuration. If the system does not provide a suitable `cmake` version, +`pip` should (theoretically) download `cmake` from the PyPI repository. +- The ``pygio`` module also requires MPI libraries to be detectable by CMake's +FindMPI. The compiler needs to support **C++17** (make sure that ``CC`` and +``CXX`` point to the correct compiler). **Install** The python library can be installed by running pip in the **main folder**: ```bash -pip install . +python -m pip install . +``` +Alternatively, the library can also directly be installed from the git URL +without having to clone the repository first: +```bash +python -m pip install git+https://git.cels.anl.gov/hacc/genericio.git ``` It will use the compiler referred by the ``CC`` and ``CXX`` environment -variable. If the compiler supports OpenMP, the library will be threaded. Make -sure to set ``OMP_NUM_THREADS`` to an appropriate variable, in particluar when -using multiple MPI ranks per node. - ------ - -## Output file partitions (subfiles) - -If you're running on an IBM BG/Q supercomputer, then the number of subfiles -(partitions) chosen is based on the I/O nodes in an automatic way. Otherwise, by -default, the GenericIO library picks the number of subfiles based on a -fairly-naive hostname-based hashing scheme. This works reasonably-well on small -clusters, but not on larger systems. On a larger system, you might want to set -these environmental variables: +variable. In case the automatically detected compiler is incorrect, specify the +compiler path as ```bash -GENERICIO_PARTITIONS_USE_NAME=0 -GENERICIO_RANK_PARTITIONS=256 +CC=/path/to/gcc CXX=/path/to/g++ python -m pip install . ``` -Where the number of partitions (256 above) determines the number of subfiles -used. If you're using a Lustre file system, for example, an optimal number of -files is: -``` -# of files * stripe count ~ # OSTs -``` +If the compiler supports OpenMP, the library will be threaded. Make +sure to set ``OMP_NUM_THREADS`` to an appropriate variable, in particluar when +using multiple MPI ranks per node. -On Titan, for example, there are 1008 OSTs, and a default stripe count of 4, so -we use approximately 256 files. diff --git a/docs/environment.rst b/docs/environment.rst index 7dc96caf4bbf3047d542271511c97d91eecff868..4e03e35f9bea8ab26fa08838e3c642bb62717c5a 100644 --- a/docs/environment.rst +++ b/docs/environment.rst @@ -1,8 +1,35 @@ Runtime Options =============== -Library / Executables ---------------------- +Output file partitions (subfiles) +--------------------------------- + +If you're running on an IBM BG/Q supercomputer, then the number of subfiles +(partitions) chosen is based on the I/O nodes in an automatic way. Otherwise, by +default, the GenericIO library picks the number of subfiles based on a +fairly-naive hostname-based hashing scheme. This works reasonably-well on small +clusters, but not on larger systems. On a larger system, you might want to set +these environmental variables: + +.. code-block:: bash + + GENERICIO_PARTITIONS_USE_NAME=0 + GENERICIO_RANK_PARTITIONS=256 + +Where the number of partitions (256 above) determines the number of subfiles +used. If you're using a Lustre file system, for example, an optimal number of +files is: + +.. code-block:: + + # of files * stripe count ~ # OSTs + +On Titan, for example, there are 1008 OSTs, and a default stripe count of 4, so +we use approximately 256 files. + + +All GenericIO Flags +------------------- ``GENERICIO_RANK0_CREATE_ALL`` @@ -20,13 +47,10 @@ Library / Executables ``GENERICIO_RANK_PARTITIONS`` -Executables ------------ - ``GENERICIO_USE_MPIIO`` -BLOSC ------ +BLOSC Flags +----------- ``BLOSC_CLEVEL`` @@ -40,4 +64,4 @@ BLOSC ``BLOSC_NTHREADS`` -``BLOSC_NOLOCK`` \ No newline at end of file +``BLOSC_NOLOCK``