datasetops.loaders

Module Contents

Classes

Loader

Contains information on how to access the raw data, and performs

Functions

from_pytorch(pytorch_dataset)	Create dataset from a Pytorch dataset
from_tensorflow(tf_dataset)	Create dataset from a Tensorflow dataset
from_folder_data(→ datasetops.dataset.Dataset)	Load data from a folder with the data structure:
from_folder_class_data(→ datasetops.dataset.Dataset)	Load data from a folder with the data structure:
from_folder_dataset_class_data(...)	Load data from a folder with the data structure:
_dataset_from_np_dict(→ datasetops.dataset.Dataset)
from_mat_single_mult_data(...)	Load data from .mat file consisting of multiple data.

class datasetops.loaders.Loader(getdata: datasetops.types.Callable[[datasetops.types.Any], datasetops.types.Any], name: str = None)

Bases: datasetops.dataset.Dataset

Contains information on how to access the raw data, and performs sampling and splitting related operations.

append(identifier: datasetops.types.Data)

extend(ids: datasetops.types.Union[datasetops.types.List[datasetops.types.Data], numpy.ndarray])

datasetops.loaders.from_pytorch(pytorch_dataset)

Create dataset from a Pytorch dataset

Arguments:

tf_dataset {torch.utils.data.Dataset} – A Pytorch dataset to load from

Returns:

[Dataset] – A datasetops.Dataset

datasetops.loaders.from_tensorflow(tf_dataset)

Create dataset from a Tensorflow dataset

Arguments:

tf_dataset {tf.data.Dataset} – A Tensorflow dataset to load from

Raises:

AssertionError: Raises error if Tensorflow is not executing eagerly

Returns:

[Dataset] – A datasetops.Dataset

datasetops.loaders.from_folder_data(path: datasetops.types.AnyPath) → datasetops.dataset.Dataset

Load data from a folder with the data structure:

folder |- sample1.jpg |- sample2.jpg

Arguments:

path {AnyPath} – path to folder

Returns:

Dataset – A dataset of data paths,

e.g. (‘nested_folder/class1/sample1.jpg’)

datasetops.loaders.from_folder_class_data(path: datasetops.types.AnyPath) → datasetops.dataset.Dataset

Load data from a folder with the data structure:

nested_folder |- class1

|- sample1.jpg |- sample2.jpg

|- class2

|- sample3.jpg

Arguments:

path {AnyPath} – path to nested folder

Returns:

Dataset – A labelled dataset of data paths and corresponding class labels,

e.g. (‘nested_folder/class1/sample1.jpg’, ‘class1’)

datasetops.loaders.from_folder_dataset_class_data(path: datasetops.types.AnyPath) → datasetops.types.List[datasetops.dataset.Dataset]

Load data from a folder with the data structure:

nested_folder |- dataset1

|- class1

|- sample1.jpg |- sample2.jpg

|- class2

|- sample3.jpg

|- dataset2

|- …

Arguments:

path {AnyPath} – path to nested folder

Returns:

List[Dataset] – A list of labelled datasets, each with data paths and corresponding class labels,

e.g. (‘nested_folder/class1/sample1.jpg’, ‘class1’)

datasetops.loaders._dataset_from_np_dict(data: datasetops.types.Dict[str, numpy.ndarray], data_keys: datasetops.types.List[str], label_key: str = None, name: str = None) → datasetops.dataset.Dataset

datasetops.loaders.from_mat_single_mult_data(path: datasetops.types.AnyPath) → datasetops.types.List[datasetops.dataset.Dataset]

Load data from .mat file consisting of multiple data.

E.g. a .mat file with keys [‘X_src’, ‘Y_src’, ‘X_tgt’, ‘Y_tgt’]

Arguments:

path {AnyPath} – path to .mat file

Returns:

List[Dataset] – A list of datasets, where a dataset was created for each suffix

e.g. a dataset with data from the keys (‘X_src’, ‘Y_src’) and from (‘X_tgt’, ‘Y_tgt’)