Safe Haskell	Safe-Inferred
Language	Haskell2010

Torch.Data.Pipeline

Contents

Defining a Dataset
Dataset
Dataloading

Synopsis

class Ord k => Dataset m dataset k sample | dataset -> m, dataset -> sample, dataset -> k where
- getItem :: dataset -> k -> m sample
- keys :: dataset -> Set k
data DatasetOptions = DatasetOptions {
- dataBufferSize :: Int
- numWorkers :: Int
- shuffle :: Sample
}
datasetOpts :: Int -> DatasetOptions
data Sample where
- Sequential :: Sample
- Shuffle :: RandomGen g => g -> Sample
streamFromMap :: forall m dataset k sample r. (Dataset m dataset k sample, MonadIO m, MonadBaseControl IO m) => DatasetOptions -> dataset -> ContT r m (ListT m sample, Sample)

Defining a Dataset

See the Vision module which implements the MNIST dataset for a good example of how to define a dataset.

Dataset

class Ord k => Dataset m dataset k sample | dataset -> m, dataset -> sample, dataset -> k where Source #

The base dataset class. A dataset is capable of returning a sample for a given key, and every Dataset has a known set of keys.

Methods

getItem :: dataset -> k -> m sample Source #

keys :: dataset -> Set k Source #

Instances

Instances details

Applicative m => Dataset (m :: Type -> Type) (MNIST m) Int (Tensor, Tensor) Source #
Instance details Defined in Torch.Vision Methods getItem :: MNIST m -> Int -> m (Tensor, Tensor) Source # keys :: MNIST m -> Set Int Source #
Applicative m => Dataset (m :: Type -> Type) (CachedDataset m sample) Int (sample :: Type) Source #
Instance details Defined in Torch.Data.Utils Methods getItem :: CachedDataset m sample -> Int -> m sample Source # keys :: CachedDataset m sample -> Set Int Source #
(KnownNat batchSize, KnownDevice device, Applicative m) => Dataset (m :: Type -> Type) (MNIST m device batchSize) Int ((Tensor device 'Float '[batchSize, 784], Tensor device 'Int64 '[batchSize]) :: Type) Source #
Instance details Defined in Torch.Typed.Vision Methods getItem :: MNIST m device batchSize -> Int -> m (Tensor device 'Float '[batchSize, 784], Tensor device 'Int64 '[batchSize]) Source # keys :: MNIST m device batchSize -> Set Int Source #

data DatasetOptions Source #

Dataset options used when loading datasets. Specify shuffling behavior, the number of threads to use, and the buffer size used to store retrieved samples in each thread.

Constructors

DatasetOptions
Fields dataBufferSize :: Int Max number of samples stored in each buffer at a given time. numWorkers :: Int Number of threads retrieving samples. shuffle :: Sample The ordering of samples streamed.

datasetOpts :: Int -> DatasetOptions Source #

Default DatasetOptions. The Int parameter specifies the number of workers, and sets the buffer size equal to the number of workers. Sampling is sequential.

data Sample where Source #

A Sample determines the ordering of samples streamed out of a dataset. You can either order sequentially, or supply a random generator to shuffle samples.

Constructors

Sequential :: Sample
Shuffle :: RandomGen g => g -> Sample

Dataloading

streamFromMap :: forall m dataset k sample r. (Dataset m dataset k sample, MonadIO m, MonadBaseControl IO m) => DatasetOptions -> dataset -> ContT r m (ListT m sample, Sample) Source #

Return a stream of samples from the given dataset, along with a new Sample value. The returned stream contains every sample returned by getItem for every key in the set of keys associated with the given dataset. The returned Sample value returns an updated Sample value, this will be identical to the original Sample value if sampling is Sequential but will return a new random number generator if sampling is Shuffle.