Package Reference

ctlearn_optimizer.optimizer

class ctlearn_optimizer.optimizer.Optimizer(opt_config)

Basic class for an optimizer.

Currently, only tree parzen estimators, random search, gaussian processes and genetic algorithm based optimization using Ray Tune is supported.

__init__(self, opt_config)

Initialize the optimizer:

• Set optimizer attributes.

• Set logger writing to both optimization.log and stdout.

• Load trials file from working_directory/trials.pkl if required, thus allowing to resume a past optimization run.

• Load optimization results file from working_directory/optimization_results.csv or create one at the same path as required. The results of the optimization run (loss, iteration, metrics, hyperparameters, time) are logged to this file for further analysis.

Parameters

opt_config (dict) – loaded optimization configuration file.

Raises

NotImplementedError – if self.optimization_type is genetic_algorithm and self.reload_trials is True.

create_optimization_algorithm(self, hyperparameter_space)

Create optimization algorithm for Ray Tune.

Currently, only tree parzen estimators, random search, gaussian processes and genetic algorithm based optimization using Ray Tune is supported.

Parameters

space (dict, list or ray.tune.automl.search_space.SearchSpace) – space of hyperparameters following the syntax required by the optimization algorithm.

Returns

Optimization algorithm for Ray Tune.

Raises

NotImplementedError – if self.optimization_type is other than tree_parzen_estimators, random_search, gaussian_processes or genetic_algorithm.

create_space_hyperparams(self)

Create space of hyperparameters following required syntax.

Currently, only tree parzen estimators and random search spaces based on hyperopt, gaussian processes space based on skopt and genetic algorithm space based on ray.tune.automl are supported.

Returns

space of hyperparameters following the syntax required by the optimization algorithm.

Raises

NotImplementedError – if self.optimization_type is other than tree_parzen_estimators, random_search, gaussian_processes or genetic_algorithm.

get_ctlearn_metric_to_optimize(self, hyperparams)

Evaluate a CTLearn model and return metric to optimize.

Parameters

hyperparams (dict) – set of hyperparameters to evaluate provided by the optimizer.

Returns

float – metric to optimize.

optimize(self, objective_function)

Start the optimization of objective_function using Ray Tune.

Currently, only tree parzen estimators, random search, gaussian processes and genetic algorithm based optimization using Ray Tune is supported.

Parameters

objective_function – function to optimize following the syntax:

def(hyperparams, reporter):
    ...
    # compute loss to optimize
    ...
    reporter(loss=loss)

Returns

ExperimentAnalysis – object used for analyzing results from Tune run().

optimize_ctlearn_model(self)

Start the optimization of a CTLearn model using Ray Tune.

Currently, only tree parzen estimators, random search, gaussian processes and genetic algorithm based optimization using Ray Tune is supported.

Returns

ExperimentAnalysis – object used for analyzing results from Tune run().

set_basic_config(self)

Set basic config and fixed hyperparameters in CTLearn config file.

ctlearn_optimizer.common

ctlearn_optimizer.common.auxiliar_modify_params(self, hyperparams)

Modify the values of the hyperparameters in CTLearn configuration file.

This function also modifies the logging model_directory of CTLearn and the prediction_file_path.

Parameters

• self – ctlearn_optimizer.optimizer.Optimizer instance.

• hyperparams (dict) – dictionary containing values of the hyperparameters.

ctlearn_optimizer.common.create_nested_item(dictionary, *keys)

Create an empty item with specific keys and positions in a dictionary.

The dictionary can be a nested dictionary containing lists, these lists can also contain nested dictionaries, and so on. The keys list can contain string (which refer dictionary keys) and integers (which refer list indices). The dictionary may or may be not empty.

Parameters

• dictionary (dict) – dictionary to modify.

• keys (list) – list of keys containing strings and integers.

Returns

dict – modified dictionary.

Raises

TypeError – if type(dictionary) is not dict.

Example:

dictionary = {}
keys = ['a', 'b', 1, 'c', 2 , 'd']
create_nested_item(dictionary, *keys) =
    {'a': {'b': [0, {'c': [0, 0, {'d': {}}]}]}}

ctlearn_optimizer.common.ctlearn_objective(self, hyperparams)

Evaluate a CTLearn model and return metric to optimize.

Train a CTLearn model and predict if necessary, get the metrics and log them to the optimization_results.csv file. Also save trials file for resuming training if it has been interrupted.

Parameters

• self – ctlearn_optimizer.optimizer.Optimizer instance.

• hyperparams (dict) – values of the hyperparameters to evaluate suggested by the optimizer.

Returns

float – metric to optimize.

ctlearn_optimizer.common.get_pred_metrics(self)

Get CTLearn prediction metrics from the current CTLearn logging folder.

Parameters

self – ctlearn_optimizer.optimizer.Optimizer instance.

Returns

dict – dictionary containing prediction set metrics to log to the optimization_results.csv file.

ctlearn_optimizer.common.get_val_metrics(self)

Get CTLearn validation metrics from the current CTLearn logging folder.

Parameters

self – ctlearn_optimizer.optimizer.Optimizer instance.

Returns

dict – dictionary containing validation set metrics to log to the optimization_results.csv file.

ctlearn_optimizer.common.modify_optimizable_params(self, hyperparams)

Update CTLearn config file with new hyperparameters at each iteration.

This function takes the dictionary containing the values of the hyperparameters to optimize suggested by the optimizer, flattens and corrects the dictionary if required, then add the values of the dependent hyperparameters to the dictionary. Finally calls auxiliar_modify_params() to modify the hyperparameters.

Parameters

• self – ctlearn_optimizer.optimizer.Optimizer instance.

• hyperparams (dict) – flat or nested dictionary containing the values of the hyperparameters to optimize suggested by the optimizer.

Returns

dict – flat dictionary containing the values of the dependent hyperparameters and hyperparameters to optimize.

ctlearn_optimizer.common.optimization_results_logger(self, loss, hyperparams_dict, metrics_val, metrics_pred, run_time)

Write loss, hyperparameters, metrics and run_time to the results file.

This function log the data to the optimization results file stored as optimization_results.csv at working_directory.

Parameters

• self – ctlearn_optimizer.optimizer.Optimizer instance.

• loss (float) – value to optimize.

• hyperparams_dict (dict) – values of the hyperparameters the user wishes to store.

• metrics_val (dict) – values of the validation set metrics the user wishes to store.

• metrics_pred (dict) – values of the prediction set metrics the user wishes to store.

• run_time (float) – execution time the user wishes to store.

ctlearn_optimizer.common.predict(self)

Predict using a trained CTLearn model.

Debug is set to False and log_to_file is set to True.

Parameters

self – ctlearn_optimizer.optimizer.Optimizer instance.

ctlearn_optimizer.common.restore(self)

Load trials.pkl of a previous run from the working_directory.

Currently, trial loading for only tree parzen_estimators, random search or gaussian processes based optimization using Ray Tune is supported.

Returns

skopt.optimizer.optimizer.Optimizer – optimizer provided from Skopt (only if self.optimization.type is gaussian_processes).

Raises

NotImplementedError – if self.optimization_type is genetic_algorithm.

ctlearn_optimizer.common.save(self)

Save trials of the current run at the working folder as trials.pkl.

Currently, trial saving for only tree parzen estimators, random search or gaussian processes based optimization using Ray Tune is supported.

Raises

NotImplementedError – if self.optimization_type is genetic_algorithm.

ctlearn_optimizer.common.set_basic_config(self)

Set basic config and fixed hyperparameters in CTLearn config file.

Parameters

self – ctlearn_optimizer.optimizer.Optimizer instance.

ctlearn_optimizer.common.set_logger(log_path)

Set up new logger writing to both log_path and stdout.

Ray Tune optimizator runs the objective function on a different Python process, so new loggers writing to the same file have to be created when necessary.

Parameters

log_path (str) – path to log file.

ctlearn_optimizer.common.set_value(dictionary, value, *keys)

Modify the value the keys point to in a nested dictionary.

The dictionary can be a nested dictionary containing lists, these lists can also contain nested dictionaries, and so on. The keys list can contain strings (which refer dictionary keys) and integers (which refer list indices). Dictionary cannot be empty.

Parameters

• dictionary (dict) – dictionary that contais the key-value pair the user wishes to modify.

• value (int, float, string) – value to set.

• keys (list) – list of keys containing strings and integers.

Returns

dict – modified dictionary.

Raises

TypeError – if type(dictionary) is not dict.

Example:

dictionary = {'a':[0,{'b':1},0]}
value = 2
keys = ['a', 1, 'b']
set_value(dictionary, value, *keys) = {'a':[0,{'b':2},0]}

ctlearn_optimizer.common.train(self)

Run a CTlearn model training.

Debug is set to False and log_to_file is set to True.

Parameters

self – ctlearn_optimizer.optimizer.Optimizer instance.

ctlearn_optimizer.bayesian_tpe

ctlearn_optimizer.bayesian_tpe.hyperopt_space(hyper_to_opt)

Create space of hyperparameters for the tree parzen estimators optimizer.

This function creates the space of hyperparameter following hyperopt syntax.

Parameters

hyper_to_opt (dict) – dictionary containing the configuration of the hyperparameters to optimize. This dictionary must follow the next syntax:

hyper_to_opt = {'hyperparam_1': {'type': ...,
                                 'range: ...,
                                 'step': ...},
                'hyperparam_2': {'type': ...,
                                 'range: ...,
                                 'step': ...},
                ...
                }

See the oficial documentation for more details.

Returns

dict – space of hyperparameters following the syntax required by the tree parzen estimators optimization algorithm.

Example:

hyper_top_opt = {
    'cnn_rnn_dropout':{
        'type': 'uniform',
        'range': [0,1]},
    'optimizer_type': {
        'type': 'choice',
        'range': ['Adadelta', 'Adam', 'RMSProp', 'SGD']},
    'number_of_layers':{
        'type': 'conditional',
        'range': {
            'value': 1,
            'cond_params':{
                'layer1_kernel':{
                    'type': 'quniform',
                    'range': [2, 10],
                    'step': 1},
                'base_learning_rate':{
                    'type': 'loguniform',
                    'range': [-5, 0]} }}}}

ctlearn_optimizer.bayesian_gp

ctlearn_optimizer.bayesian_gp.skopt_space(hyper_to_opt)

Create space of hyperparameters for the gaussian processes optimizer.

This function creates the space of hyperparameter following skopt syntax.

Parameters

hyper_to_opt (dict) – dictionary containing the configuration of the hyperparameters to optimize. This dictionary must follow the next syntax:

hyper_to_opt = {'hyperparam_1': {'type': ...,
                                 'range: ...,
                                 'step': ...},
                'hyperparam_2': {'type': ...,
                                 'range: ...,
                                 'step': ...},
                ...
                }

See the oficial documentation for more details.

Returns

list – space of hyperparameters following the syntax required by the gaussian processes optimization algorithm.

Example:

hyper_top_opt = {
    'cnn_rnn_dropout':{
        'type': 'uniform',
        'range': [0,1]},
    'optimizer_type':{
        'type': 'choice',,
        'range': ['Adadelta', 'Adam', 'RMSProp', 'SGD']},
    'base_learning_rate':{
        'type': 'loguniform',
        'range': [-5, 0]},
    'layer1_filters':{
        'type': 'quniform',
        'range': [16, 64],
        'step': 1}}

Raises

KeyError – if type is other than uniform, quniform, loguniform or choice.

ctlearn_optimizer.genetic_algorithm

ctlearn_optimizer.genetic_algorithm.gen_al_space(self)

Create space of hyperparameters for the genetic algorithm optimizer.

This function creates the space of hyperparameter following ray.tune.automl syntax.

Parameters

hyper_to_opt (dict) – dictionary containing the configuration of the hyperparameters to optimize. This dictionary must follow the next syntax:

hyper_to_opt = {'hyperparam_1': {'type': ...,
                                 'range: ...,
                                 'step': ...},
                'hyperparam_2': {'type': ...,
                                 'range: ...,
                                 'step': ...},
                ...
                }

See the oficial documentation for more details.

Returns

ray.tune.automl.search_space.SearchSpace – space of hyperparameters following the syntax required by the genetic algorithm optimizer.

Example:

hyper_top_opt = {
    'cnn_rnn_dropout':{
        'type': 'uniform',
        'range': [0,1]},
    'optimizer_type':{
        'type': 'choice',,
        'range': ['Adadelta', 'Adam', 'RMSProp', 'SGD']},
    'layer1_filters':{
        'type': 'quniform',
        'range': [16, 64],
        'step': 1}}

Raises

KeyError – if type is other than uniform, quniform or choice.

ctlearn_optimizer.auxiliar_functions

ctlearn_optimizer.auxiliar_functions.df2result(df, metric_col, param_cols, param_types=None)

Convert df with metrics and hyperparams to the OptimizeResults format.

It is a helper function that lets you use all the tools that expect OptimizeResult object like for example scikit-optimize plot_evaluations function.

Parameters

• df (pandas.DataFrame) – Dataframe containing metric and hyperparameters.

• metric_col (str) – Name of the metric column.

• param_cols (list) – Names of the hyperparameter columns.

• param_types (list or None) – Optional list of hyperparameter column types. By default it will treat all the columns as float but you can also pass str for categorical channels. Example: param_types=[float, str, float, float]

Returns

scipy.optimize.OptimizeResult – Results object that contains the hyperparameter and metric information.

ctlearn_optimizer.auxiliar_functions.plot_convergence(*args, **kwargs)

Plot one or several convergence traces.

Parameters

• args[i] (OptimizeResult, list of OptimizeResult, or tuple) – The result(s) for which to plot the convergence trace.

  • if OptimizeResult, then draw the corresponding single trace;

  • if list of OptimizeResult, then draw the corresponding convergence traces in transparency, along with the average convergence trace;

  • if tuple, then args[i][0] should be a string label and args[i][1] an OptimizeResult or a list of OptimizeResult.

• ax (Axes, optional) – The matplotlib axes on which to draw the plot, or None to create a new one.

• yscale (None or string, optional) – The scale for the y-axis.

Returns

Axes – The matplotlib axes.