# Tasks

# execute

machinable.execute(task, storage=None, seed=None, driver=None, engine=None)

Executes a machinable task

Schedules the task for execution. If a Ray server is available the execution will be scheduled and distributed on the Ray server.

# Arguments

task: machinable.Task, specifies the execution task. For convenience, it can also be a string or tuple that defines the node argument, e.g. ml.execute('my_model') or ml.execute((('my_model', {'param': 1}),))
storage: String, URL of the storage location for the results. If unspecified, the result storage will be non-persistent. Remote locations like SSH or S3 are supported via pyFilesystem URLs
seed: Integer|String|None, determines the global random seed. If None, a random seed will be generated. To re-use the same random seed of a previous execution, you can pass in its task ID
driver: Dict|String|None, driver type and options that determine the mode of execution, e.g. 'ray' or 'multiprocessing' etc.
engine: Optional machinable.engine.Engine to use for this execution

# Example

import machinable as ml
ml.execute(ml.Task().component('iris', 'random_forest'), 's3://bucket', seed=42)

# Functional API

You can use this method as a function decorator to implement a custom execution. The decorated function is invoked with the node configuration as well as an observer object, for example:

@ml.execute
def custom_execute(component, children, observer):
    observer.log.info('Custom training with learning_rate=' + str(component.config.lr))

custom_execute(task, storage, seed) # invokes the decorated function

# Task

machinable.Task(component=None)

Defines an execution schedule for available components. The task interface is fluent, methods can be chained in arbitrary order.

# Arguments

component: Optional String or tuple component definition to add to the task

# Example

import machinable as ml
linear_regression = ml.Task().component('mnist').repeat(3)
#### or using the shorthand where component arguments are passed into the constructor
linear_regression = ml.Task('mnist').repeat(3)

# component

component(self, node, children=None, resources=None)

Adds a component to the task

# Arguments

node: machinable.TaskComponent specifying the node component
children: optional list of machinable.Component, supplementary components for this node
resources: dict, specifies the Ray resources that are available to the actor if used. Has no effect in local execution mode.

# Examples

import machinable as ml
task = ml.Task().component(('mnist', version='~shuffled', resources={'num_gpus': 2}))

# confirm

confirm(self, each=False, timeout=None)

Adds a confirmation step before execution

WARNING

This feature is currently under development and not ready for wider use

# Arguments

each: Boolean, if True each sub-execution has to be confirmed individually
timeout: Optional timeout in seconds after which the execution will be confirmed

# copy

copy(self)

Returns a copy of the current task object

# dry

dry(self, verbosity=1)

Marks the execution as simulation without actual execution

# Arguments

verbosity: Integer determining the level of output detail

# export

export(self, path=None, overwrite=False)

Marks the task as export.

When executed, machinable converts the task into a plain Python project that can be executed without machinable.

WARNING

This feature may not work reliably in all circumstances and project use cases

# Arguments

path: String, directory where exported task will be stored. If None defaults to 'exports' in the current working directory
overwrite: Boolean, whether to overwrite an existing export.

# name

name(self, name)

Specifies a name of the current task that may help to manage its results later

# Arguments

name: String, the name of the task

# repeat

repeat(self, k, name='REPEAT', mode='independent')

Repeats the task k times

Schedules the current task multiple times and injects the flags REPEAT_NUMBER and REPEAT_SEED to the node instances.

# Arguments

k: Integer, the number of repetitions
name: String, flag prefix, e.g. '{name}_SEED' etc. Defaults to REPEAT

# Examples

import machinable as ml
#### five independent runs of the same node
ml.Task().component('regression').repeat(5)

# split

split(self, k, name='SPLIT', mode='independent')

Splits the task k times

Schedules the current task k times and injects appropriate flags SPLIT_NUMBER and SPLIT_SEED to the node instances. Note that machinable does not split the nodes automatically. The user has to implement the algorithmic splitting based on the flag information. For example, to implement a cross-validation algorithm the node should split the data using the split seed in the flag SPLIT_SEED and use the split that is specified in the flag SPLIT_NUMBER.

# Arguments

k: Integer, the number of splits
name: String, flag prefix, e.g. '{name}_SEED' etc. Defaults to SPLIT

# Examples

import machinable as ml
#### five independent runs of the same node
ml.Task().component('regression').repeat(5)

# tune

tune(self, stop=None, config=None, num_samples=1, **kwargs)

Schedules for hyperparameter tuning

Uses Ray tune. Components need to implement on_execute_iteration event that becomes tune trainable. The record writer can be used as usual. In particular, record fields may be used in stop conditions.

WARNING

Tuning integration is experimental. Please report any issues that you encounter.

# Arguments

stop: dict, the stopping criteria. The keys may be any field in self.records of the node, whichever is reached first. Defaults to empty dict.
config: dict, algorithm-specific configuration for Tune variant generation (e.g. env, hyperparams). Defaults to empty dict. Custom search algorithms may ignore this.
num_samples: Integer, Number of times to sample from the hyperparameter space. Defaults to 1. If grid_search is provided as an argument, the grid will be repeated num_samples of times. **kwargs: Additional options passed to ray.tune.run

Please refer to the documentation of Ray to learn more.

# version

version(self, config=None, **kwargs)

Applies a configuration update

Selected components use the default configuration as specified in the machinable.yaml. Versioning allows for configuration overrides using dictionaries or defined version patterns. The versions here are applied globally to all components in this task. If you want to apply the version locally, use the version parameter in the local method.

# Arguments

config: Configuration update represented as dictionary

# Examples

import machinable as ml
task = ml.Task().component('evolution').component('sgd')
#### use a dictionary to override configuration values
task.version({'data': {'shuffle': True}, 'children': {'lr': 0.01}})
#### or kwargs
task.version(data={'shuffle': True}, children={'lr': 0.01})
#### use a specified version of the machinable.yaml
task.version(data='~shuffled')
#### use a mixin configuration
task.version(data='_mnist_')

← Component Observations →