PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports
- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.
- HDFS client and libhdfs for HDFS access
- CPython >= 3.6
Installation
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .Documentation
$ cd pfio/docs
$ make html
$ open build/html/index.htmlTest
$ cd pfio
$ pip install .[test]
$ pytest tests/Please refer to the official document for more information about the usage.
Check the official document for latest release procedure.
Run tests locally:
$ pip install --user -e .[test]
$ pytestBump version numbers in pfio/version.py .
Push and open a pull request to invoke CI. Once CI passed and the pull request merged, tag a release:
$ git tag -s X.Y.Z
$ git push --tagsBuild:
$ rm -rf dist
$ pip3 install --user build
$ python3 -m buildRelease to PyPI:
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*