NAME
AI::MXNet::IO - Data loading interface of MXNet
DESCRIPTION
This document summarizes supported data formats and iterator APIs to read the data including
mx->io Data iterators for common data formats.
mx->recordio Data iterators for the RecordIO data format.
mx->image Image Iterators and image augmentation functions.
NAME
AI::MXNet::DataDesc - A container class for describing the data layout.
get_batch_axis
Get the dimension that corresponds to the batch size.
Parameters
----------
layout : str
layout string. For example, "NCHW".
Returns
-------
An axis indicating the batch_size dimension. When data-parallelism is
used, the data will be automatically split and concatenate along the batch_size
dimension. Axis can be -1, which means the whole array will be copied for each
data-parallelism device.
get_list
Coverts the input to an array ref AI::MXNet::DataDesc objects.
Parameters
----------
$shapes : HashRef[Shape]
$types= : Maybe[HashRef[Dtype]]
NAME
AI::MXNet::DataBatch - A container for a mini-batch of the data and related information.
DESCRIPTION
Default object for holding a mini-batch of data and related information.
NAME
AI::MXNet::DataIter - A parent class for MXNet data iterators.
reset
Reset the iterator.
list
Returns remaining iterator items as an array ref.
next
Returns the next data batch from the iterator.
Returns
-------
$data : AI::MXNet::DataBatch
The data of next batch.
iter_next
Iterate to next batch.
Returns
-------
$has_next : Bool
get_data
The data of current batch.
Returns
-------
data : AI::MXNet::NDArray
getlabel
The label of the current batch.
Returns
-------
label : AI::MXNet::NDArray
getindex
The index of the current batch.
Returns
-------
$index : PDL
getpad
The number of padding examples in the current batch.
Returns
-------
$pad : Int
NAME
AI::MXNet::ResizeIter
DESCRIPTION
Resize a DataIter to a given number of batches per epoch.
May produce incomplete batch in the middle of an epoch due
to the padding from internal iterator.
Parameters
----------
data_iter : DataIter
Internal data iterator.
size : number of batches per epoch to resize to.
reset_internal : whether to reset internal iterator on ResizeIter.reset
NAME
AI::MXNet::NDArrayIter - Predefined NDArray iterator.
DESCRIPTION
Predefined NDArray iterator. Accepts PDL or AI::MXNet::NDArray object as an input.
Parameters
----------
data: Maybe[AcceptableInput|HashRef[AcceptableInput]|ArrayRef[AcceptableInput]].
NDArrayIter supports single or multiple data and label.
label: Maybe[AcceptableInput|HashRef[AcceptableInput]|ArrayRef[AcceptableInput]].
Same as data, but is not given to the model during testing.
batch_size=1: Int
Batch Size
shuffle=0: Bool
Whether to shuffle the data
last_batch_handle='pad': 'pad', 'discard' or 'roll_over'
How to handle the last batch
Note
----
This iterator will pad, discard or roll over the last batch if
the size of data does not match batch_size. Roll over is intended
for training and can cause problems if used for prediction.
NAME
AI::MXNet::MXDataIter - A data iterator pre-built in C++ layer of MXNet.
DESCRIPTION
Here are the list of currently available predefined iterators, for more custom iterators
please check out the examples directory.
Also please refer to the L<Python docs|https://mxnet.apache.org/api/python/io/io.html>
mx->io->CSVIter Returns the CSV file iterator.
mx->io->LibSVMIter Returns the LibSVM iterator which returns data with csr storage type.
mx->io->ImageRecordIter Iterates on image RecordIO files
mx->io->ImageRecordInt8Iter Iterating on image RecordIO files
mx->io->ImageRecordUInt8Iter Iterating on image RecordIO files
mx->io->MNISTIter Iterating on the MNIST dataset.
mx->recordio->MXRecordIO Reads/writes RecordIO data format, supporting sequential read and write.
mx->recordio->MXIndexedRecordIO Reads/writes RecordIO data format, supporting random access.
mx->image->ImageIter Image data iterator with a large number of augmentation choices.
debug_skip_load
Set the iterator to simply return always first batch.
Notes
-----
This can be used to test the speed of network without taking
the loading delay into account.