Petastorm

Petastorm is a library enabling the use of Parquet storage from Tensorflow, Pytorch, and other Python-based ML training frameworks.

Github project page

• User guide
  • Installation
  • Generating a dataset
  • Plain Python API
  • Tensorflow API
  • Pytorch API
  • Spark Dataset Converter API
  • Analyzing petastorm datasets using PySpark and SQL
  • Non Petastorm Parquet Stores
  • Troubleshooting
  • See also
• How to Contribute
• API
  • Unischema
  • Reader
  • Tensorflow
  • PyTorch
  • PySpark Dataset Converter
  • PySpark & SQL
  • TransformSpec
  • Row queries
  • Local cache
  • Codecs
  • Dataset generation
  • Row-group selectors
  • Benchmarks
  • HDFS
• Developer guide
  • Setting up development environment (recommendation)
  • Unit tests
  • Documentation
  • Release procedure
• Benchmarks
• Troubleshooting
  • Trouble running the pytorch example
• Release notes
  • Release 0.11.3 (unreleased)
  • Release 0.11.2
  • Release 0.11.1
  • Release 0.11.0
  • Release 0.10.0
  • Release 0.9.8
  • Release 0.9.7
  • Release 0.9.6
  • Release 0.9.5
  • Release 0.9.4
  • Release 0.9.3
  • Release 0.9.2
  • Release 0.9.1
  • Release 0.9.0
  • Release 0.8.2
  • Release 0.8.1
  • Release 0.8.0
  • Release 0.7.7
  • Release 0.7.6
  • Release 0.7.5
  • Release 0.7.4
  • Release 0.7.3
  • Release 0.7.2
  • Release 0.7.1
  • Release 0.7.0
  • Release 0.6.0
  • Release 0.5.1
  • Release 0.5.0
  • Release 0.4.3
  • Release 0.4.2
  • Release 0.4.0, 0.4.1
• Migration notes
  • To petastorm 0.5.0

Indices and tables

• Index
• Module Index
• Search Page