API === Unischema --------- .. automodule:: petastorm.unischema Reader ------ .. automodule:: petastorm.reader .. automodule:: petastorm.weighted_sampling_reader .. automodule:: petastorm.ngram Tensorflow ---------- .. automodule:: petastorm.tf_utils PyTorch ------- .. automodule:: petastorm.pytorch PySpark Dataset Converter ------------------------- .. automodule:: petastorm.spark.spark_dataset_converter PySpark & SQL ------------- .. automodule:: petastorm.spark_utils TransformSpec ------------- .. automodule:: petastorm.transform Row queries ----------- .. automodule:: petastorm.predicates Local cache ----------- .. automodule:: petastorm.cache .. automodule:: petastorm.local_disk_cache Codecs ------ .. automodule:: petastorm.codecs Dataset generation ------------------ .. automodule:: petastorm.etl .. automodule:: petastorm.etl.dataset_metadata :exclude-members: load_row_groups .. automodule:: petastorm.etl.petastorm_generate_metadata Row-group selectors ------------------- .. automodule:: petastorm.selectors .. automodule:: petastorm.etl.rowgroup_indexers .. automodule:: petastorm.etl.rowgroup_indexing Benchmarks ---------- .. automodule:: petastorm.benchmark.throughput HDFS ---- .. automodule:: petastorm.hdfs.namenode