torch_opt

Source code: sensai/torch/torch_opt.py

---

class Optimiser(value)

Bases: Enum

An enumeration.

SGD = ('sgd', torch.optim.SGD)

ASGD = ('asgd', torch.optim.ASGD)

ADAGRAD = ('adagrad', torch.optim.Adagrad)

ADADELTA = ('adadelta', torch.optim.Adadelta)

ADAM = ('adam', torch.optim.Adam)

ADAMW = ('adamw', torch.optim.AdamW)

ADAMAX = ('adamax', torch.optim.Adamax)

RMSPROP = ('rmsprop', torch.optim.RMSprop)

RPROP = ('rprop', torch.optim.Rprop)

LBFGS = ('lbfgs', torch.optim.LBFGS)

classmethod from_name(name: str) → Optimiser

classmethod from_name_or_instance(name_or_instance: Union[str, Optimiser]) → Optimiser

class NNLossEvaluator

Bases: ABC

Base class defining the interface for training and validation loss evaluation.

class Evaluation

Bases: ABC

abstract start_epoch() → None

Starts a new epoch, resetting any aggregated values required to ultimately return the epoch’s overall training loss (via getEpochTrainLoss) and validation metrics (via getValidationMetrics)

abstract compute_train_batch_loss(model_output, ground_truth, x, y) → torch.Tensor

Computes the loss for the given model outputs and ground truth values for a batch and aggregates the computed loss values such that :meth:getEpochTrainLoss can return an appropriate result for the entire epoch. The original batch tensors X and Y are provided as meta-information only.

Parameters:

• model_output – the model output

• ground_truth – the ground truth values

• x – the original batch input tensor

• y – the original batch output (ground truth) tensor

Returns:

the loss (scalar tensor)

abstract get_epoch_train_loss() → float

Returns:

the epoch’s overall training loss (as obtained by collecting data from individual training batch data passed to computeTrainBatchLoss)

abstract process_validation_batch(model_output, ground_truth, x, y) → None

Processes the given model outputs and ground truth values in order to compute sufficient statistics for velidation metrics, which at the end of the epoch, shall be retrievable via method getValidationMetrics

Parameters:

• model_output – the model output

• ground_truth – the ground truth values

• x – the original batch input tensor

• y – the original batch output (ground truth) tensor

Returns:

the loss (scalar tensor)

abstract get_validation_metrics() → Dict[str, float]

abstract start_evaluation(cuda: bool) → Evaluation

Begins the evaluation of a model, returning a (stateful) object which is to perform the necessary computations.

Parameters:

cuda – whether CUDA is being applied (all tensors/models on the GPU)

Returns:

the evaluation object

abstract get_validation_metric_name() → str

Returns:

the name of the validation metric which is to be used to determine the best model (key for the ordered dictionary returned by method Evaluation.getValidationMetrics)

class NNLossEvaluatorFixedDim

Bases: NNLossEvaluator, ABC

Base class defining the interface for training and validation loss evaluation, which uses fixed-dimension outputs and aggregates individual training batch losses that are summed losses per batch (averaging appropriately internally).

class Evaluation(criterion, validation_loss_evaluator: ValidationLossEvaluator, output_dim_weights: Optional[torch.Tensor] = None)

Bases: Evaluation

start_epoch()

Starts a new epoch, resetting any aggregated values required to ultimately return the epoch’s overall training loss (via getEpochTrainLoss) and validation metrics (via getValidationMetrics)

compute_train_batch_loss(model_output, ground_truth, x, y) → torch.Tensor

Computes the loss for the given model outputs and ground truth values for a batch and aggregates the computed loss values such that :meth:getEpochTrainLoss can return an appropriate result for the entire epoch. The original batch tensors X and Y are provided as meta-information only.

Parameters:

• model_output – the model output

• ground_truth – the ground truth values

• x – the original batch input tensor

• y – the original batch output (ground truth) tensor

Returns:

the loss (scalar tensor)

get_epoch_train_loss() → float

Returns:

the epoch’s overall training loss (as obtained by collecting data from individual training batch data passed to computeTrainBatchLoss)

process_validation_batch(model_output, ground_truth, x, y)

Processes the given model outputs and ground truth values in order to compute sufficient statistics for velidation metrics, which at the end of the epoch, shall be retrievable via method getValidationMetrics

Parameters:

• model_output – the model output

• ground_truth – the ground truth values

• x – the original batch input tensor

• y – the original batch output (ground truth) tensor

Returns:

the loss (scalar tensor)

get_validation_metrics() → Dict[str, float]

start_evaluation(cuda: bool) → Evaluation

Begins the evaluation of a model, returning a (stateful) object which is to perform the necessary computations.

Parameters:

cuda – whether CUDA is being applied (all tensors/models on the GPU)

Returns:

the evaluation object

abstract get_training_criterion() → torch.nn.Module

Gets the optimisation criterion (loss function) for training. Standard implementations are available in torch.nn (torch.nn.MSELoss, torch.nn.CrossEntropyLoss, etc.).

abstract get_output_dim_weights() → Optional[ndarray]

abstract create_validation_loss_evaluator(cuda: bool) → ValidationLossEvaluator

Parameters:

cuda – whether to use CUDA-based tensors

Returns:

the evaluator instance which is to be used to evaluate the model on validation data

get_validation_metric_name() → str

Gets the name of the metric (key of dictionary as returned by the validation loss evaluator’s endValidationCollection method), which is defining for the quality of the model and thus determines which epoch’s model is considered the best.

Returns:

the name of the metric

class ValidationLossEvaluator

Bases: ABC

abstract start_validation_collection(ground_truth_shape)

Initiates validation data collection for a new epoch, appropriately resetting this object’s internal state.

Parameters:

ground_truth_shape – the tensor shape of a single ground truth data point (not including the batch entry dimension)

abstract process_validation_result_batch(output, ground_truth)

Collects, for validation, the given output and ground truth data (tensors holding data on one batch, where the first dimension is the batch entry)

Parameters:

• output – the model’s output

• ground_truth – the corresponding ground truth

abstract end_validation_collection() → OrderedDict

Computes validation metrics based on the data previously processed.

Returns:

an ordered dictionary with validation metrics

class NNLossEvaluatorRegression(loss_fn: LossFunction = LossFunction.L2LOSS, validation_tensor_transformer: Optional[TensorTransformer] = None, output_dim_weights: Optional[Sequence[float]] = None, apply_output_dim_weights_in_validation=True, validation_metric_name: Optional[str] = None)

Bases: NNLossEvaluatorFixedDim, ToStringMixin

A loss evaluator for (multi-variate) regression.

Parameters:

• loss_fn – the loss function to use

• validation_tensor_transformer – a transformer which is to be applied to validation tensors (both model outputs and ground truth) prior to computing the validation metrics

• output_dim_weights – vector of weights to apply to then mean loss per output dimension, i.e. for the case where for each data point, the model produces n output dimensions, the mean loss for the i-th dimension is to be computed separately and be scaled with the weight, and the overall loss returned is the weighted average. The weights need not sum to 1 (normalisation is applied).

• apply_output_dim_weights_in_validation – whether output dimension weights are also to be applied to to the metrics computed for validation. Note that this may not be possible if a validationTensorTransformer which changes the output dimensions is used.

• validation_metric_name – the metric to use for model selection during validation; if None, use default depending on lossFn

class LossFunction(value)

Bases: Enum

An enumeration.

L1LOSS = 'L1Loss'

L2LOSS = 'L2Loss'

MSELOSS = 'MSELoss'

SMOOTHL1LOSS = 'SmoothL1Loss'

create_validation_loss_evaluator(cuda)

Parameters:

cuda – whether to use CUDA-based tensors

Returns:

the evaluator instance which is to be used to evaluate the model on validation data

get_training_criterion()

Gets the optimisation criterion (loss function) for training. Standard implementations are available in torch.nn (torch.nn.MSELoss, torch.nn.CrossEntropyLoss, etc.).

get_output_dim_weights() → Optional[ndarray]

class ValidationLossEvaluator(cuda: bool, validation_tensor_transformer: Optional[TensorTransformer], output_dim_weights: ndarray, apply_output_dim_weights: bool)

Bases: ValidationLossEvaluator

start_validation_collection(ground_truth_shape)

Initiates validation data collection for a new epoch, appropriately resetting this object’s internal state.

Parameters:

ground_truth_shape – the tensor shape of a single ground truth data point (not including the batch entry dimension)

process_validation_result_batch(output, ground_truth)

Collects, for validation, the given output and ground truth data (tensors holding data on one batch, where the first dimension is the batch entry)

Parameters:

• output – the model’s output

• ground_truth – the corresponding ground truth

end_validation_collection()

Computes validation metrics based on the data previously processed.

Returns:

an ordered dictionary with validation metrics

get_validation_metric_name()

Gets the name of the metric (key of dictionary as returned by the validation loss evaluator’s endValidationCollection method), which is defining for the quality of the model and thus determines which epoch’s model is considered the best.

Returns:

the name of the metric

class NNLossEvaluatorClassification(loss_fn: LossFunction)

Bases: NNLossEvaluatorFixedDim

A loss evaluator for classification

class LossFunction(value)

Bases: Enum

An enumeration.

CROSSENTROPY = 'CrossEntropy'

NLL = 'NegativeLogLikelihood'

create_criterion() → Callable

get_validation_metric_key() → str

classmethod default_for_output_mode(output_mode: ClassificationOutputMode)

create_validation_loss_evaluator(cuda)

Parameters:

cuda – whether to use CUDA-based tensors

Returns:

the evaluator instance which is to be used to evaluate the model on validation data

get_training_criterion()

Gets the optimisation criterion (loss function) for training. Standard implementations are available in torch.nn (torch.nn.MSELoss, torch.nn.CrossEntropyLoss, etc.).

get_output_dim_weights() → Optional[ndarray]

class ValidationLossEvaluator(cuda: bool, loss_fn: LossFunction)

Bases: ValidationLossEvaluator

start_validation_collection(ground_truth_shape)

Initiates validation data collection for a new epoch, appropriately resetting this object’s internal state.

Parameters:

ground_truth_shape – the tensor shape of a single ground truth data point (not including the batch entry dimension)

process_validation_result_batch(output, ground_truth)

Collects, for validation, the given output and ground truth data (tensors holding data on one batch, where the first dimension is the batch entry)

Parameters:

• output – the model’s output

• ground_truth – the corresponding ground truth

end_validation_collection()

Computes validation metrics based on the data previously processed.

Returns:

an ordered dictionary with validation metrics

get_validation_metric_name()

Gets the name of the metric (key of dictionary as returned by the validation loss evaluator’s endValidationCollection method), which is defining for the quality of the model and thus determines which epoch’s model is considered the best.

Returns:

the name of the metric

class NNOptimiserParams(loss_evaluator: Optional[NNLossEvaluator] = None, gpu: Optional[int] = None, optimiser: Union[str, Optimiser] = 'adam', optimiser_lr=0.001, early_stopping_epochs=None, batch_size=None, epochs=1000, train_fraction=0.75, scaled_outputs=False, use_shrinkage=True, shrinkage_clip=10.0, shuffle=True, optimiser_args: Optional[Dict[str, Any]] = None)

Bases: ToStringMixin

Parameters:

• loss_evaluator – the loss evaluator to use

• gpu – the index of the GPU to be used (if CUDA is enabled for the model to be trained); if None, default to first GPU

• optimiser – the optimiser to use

• optimiser_lr – the optimiser’s learning rate

• early_stopping_epochs – the number of epochs without validation score improvement after which to abort training and use the best epoch’s model (early stopping); if None, never abort training before all epochs are completed

• batch_size – the batch size to use; for algorithms L-BFGS (optimiser=’lbfgs’), which do not use batches, leave this at None. If the algorithm uses batches and None is specified, batch size 64 will be used by default.

• train_fraction – the fraction of the data used for training (with the remainder being used for validation). If no validation is to be performed, pass 1.0.

• scaled_outputs – whether to scale all outputs, resulting in computations of the loss function based on scaled values rather than normalised values. Enabling scaling may not be appropriate in cases where there are multiple outputs on different scales/with completely different units.

• use_shrinkage – whether to apply shrinkage to gradients whose norm exceeds shrinkageClip, scaling the gradient down to shrinkageClip

• shrinkage_clip – the maximum gradient norm beyond which to apply shrinkage (if useShrinkage is True)

• shuffle – whether to shuffle the training data

• optimiser_args – keyword arguments to be passed on to the actual torch optimiser

REMOVED_PARAMS = {'cuda'}

RENAMED_PARAMS = {'batchSize': 'batch_size', 'earlyStoppingEpochs': 'early_stopping_epochs', 'lossEvaluator': 'loss_evaluator', 'optimiserClip': 'optimiser_clip', 'optimiserLR': 'optimiser_lr', 'scaledOutputs': 'scaled_outputs', 'shrinkageClip': 'shrinkage_clip', 'trainFraction': 'train_fraction', 'useShrinkage': 'use_shrinkage'}

classmethod from_dict_or_instance(nn_optimiser_params: Union[dict, NNOptimiserParams]) → NNOptimiserParams

classmethod from_dict(params: dict) → NNOptimiserParams

classmethod from_either_dict_or_instance(nn_optimiser_dict_params: dict, nn_optimiser_params: Optional[NNOptimiserParams])

class NNOptimiser(params: NNOptimiserParams)

Bases: object

Parameters:

params – parameters

log = <Logger sensai.torch.torch_opt.NNOptimiser (WARNING)>

fit(model: TorchModel, data: Union[DataUtil, List[DataUtil], TorchDataSetProvider, List[TorchDataSetProvider], TorchDataSet, List[TorchDataSet], Tuple[TorchDataSet, TorchDataSet], List[Tuple[TorchDataSet, TorchDataSet]]], create_torch_module=True) → TrainingInfo

Fits the parameters of the given model to the given data, which can be a list of or single instance of one of the following:

• a DataUtil or TorchDataSetProvider (from which a training set and validation set will be obtained according to the trainFraction parameter of this object)

• a TorchDataSet which shall be used as the training set (for the case where no validation set shall be used)

• a tuple with two TorchDataSet instances, where the first shall be used as the training set and the second as the validation set

Parameters:

• model – the model to be fitted

• data – the data to use (see variants above)

• create_torch_module – whether to newly create the torch module that is to be trained from the model’s factory. If False, (re-)train the existing module.

class TrainingInfo(best_epoch: Optional[int] = None, log: Optional[List[str]] = None, training_loss_sequence: Optional[Sequence[float]] = None, validation_metric_sequence: Optional[Sequence[float]] = None, total_epochs=None)

Bases: object

get_training_loss_series() → Series

get_validation_metric_series() → Series

plot_all() → Figure

Plots both the sequence of training loss values and the sequence of validation metric values