# Component

machinable.Component(config: dict = None, flags: dict = None, node=None)

Component base class. All machinable components must inherit from this class.

TIP

All components are Mixins by default. However, if you want to explicitly mark your components for Mixin-only use consider inheriting from the Mixin base class machinable.Mixin.

# components property

List of sub components or None

# config property

Component configuration

# events property

Event interface

# flags property

Component flags

# log property

Log writer instance

# node property

Node component or None

# record property

Record writer instance

# storage property

Storage interface

# submission property

Submission of this component

# bind

bind(self, mixin, attribute)

Binds a mixin to the component

# Arguments

  • mixin: Mixin module or class
  • attribute: Attribute name, e.g. my_mixin

# create

create(self, components=None)

Creates the components

TIP

This method is triggered automatically. However, sub-component creation can be suppressed in the on_create event of the node component. See on_create for details.

Triggers create events of the lifecycle

# Arguments

  • components: Optional list of sub components instances that are used by the components

# destroy

destroy(self)

Destroys the components

Triggers destroy events

TIP

This method is triggered automatically.

# execute

execute(self)

Executes the components

Triggers execution events and writes execution meta-data

TIP

Execution is triggered automatically for node components only.

# on_after_create

on_after_create(self)

Lifecycle event triggered after components creation

# on_after_destroy

on_after_destroy(self)

Lifecycle event triggered after components destruction

# on_after_execute

on_after_execute(self)

Lifecycle event triggered after execution

# on_after_execute_iteration

on_after_execute_iteration(self, iteration: int)

Lifecycle event triggered after execution iteration

# on_after_init

on_after_init(self)

Lifecycle event triggered after components initialisation

# on_after_init_components

on_after_init_components(self, components, index: Union[int, NoneType] = None)

Lifecycle event triggered after components initialisation

# on_after_init_storage

on_after_init_storage(self, storage_config: Dict)

Lifecycle event triggered when storage is available

# on_before_create

on_before_create(self)

Lifecycle event triggered before components creation

# on_before_destroy

on_before_destroy(self)

Lifecycle event triggered before components destruction

# on_before_execute

on_before_execute(self)

Lifecycle event triggered before components execution

# on_before_execute_iteration

on_before_execute_iteration(self, iteration: int)

Lifecycle event triggered before an execution iteration

# on_before_init

on_before_init(self, config=None, flags=None, node=None)

Lifecycle event triggered before components initialisation

# on_create

on_create(self)

Lifecycle event triggered during components creation

The method can declare arguments to handle components explicitly. For example, the signature on_create(self, node, alias_of_child_1, alias_of_child_2=None) declares that components accepts two sub components with alias alias_of_child_1 and alias_of_child_2 where the latter is declared optional. If the alias starts with an underscore _ the components lifecycle will not be triggered automatically.

The function signature may also be used to annotate expected types, for example:

on_create(self, node: DistributedExperiment, model: DistributedModel = None):
    self.experiment = node
    self.model = model

Note, however, that the type annotation will not be enforced.

# on_destroy

on_destroy(self)

Lifecycle event triggered at components destruction

# on_execute

on_execute(self)

Lifecycle event triggered at components execution

TIP

This event and all other execution events are triggered in node components only

# on_execute_iteration

on_execute_iteration(self, iteration: int)

Lifecycle event triggered at execution iteration

Allows to implement repeating iterations

The method is repeatedly called until StopIteration is returned or raised; iteration is increased on each call The iteration number is also available class-wide under self.flags.ITERATION

# Arguments

  • iteration: Integer, iteration number that automatically increases in every iteration starting from 0

# on_failure

on_failure(self, result)

Lifecycle event triggered iff execution finished with an exception

# Arguments

  • result: Execution exception

# on_finish

on_finish(self, result, success)

Lifecycle event triggered right before the end of the component execution

# Arguments

  • success: Whether the execution finished sucessfully

# on_init

on_init(self, config=None, flags=None, node=None)

Lifecycle event triggered during components initialisation

Return False to prevent the default configuration and flag instantiation

# on_init_components

on_init_components(self, components_config, index: Union[int, NoneType] = None)

Lifecycle event triggered during components initialisation

Return False to prevent the components instantiation

# on_init_storage

on_init_storage(self, storage_config: Dict)

Lifecycle event triggered at write initialisation

# on_restore

on_restore(self, checkpoint)

Implements the restore checkpoint functionality of the components

# Arguments

  • checkpoint: Checkpoint specification
  • timestep: Integer, counting number of checkpoint

# Returns

Return False to indicate that checkpoint has not been restored.

# on_save

on_save(self, checkpoint_path: str, timestep: int) -> Union[bool, str]

Implements the checkpoint functionality of the components

# Arguments

  • checkpoint_path: String, target directory
  • timestep: Integer, counting number of checkpoint

# Returns

Returns the name of the checkpoint file or path. Return False to indicate that checkpoint has not been saved.

# on_seeding

on_seeding(self)

Lifecycle event to implement custom seeding

Return False to prevent the default seeding procedure

# on_start

on_start(self)

Lifecycle event triggered at the very beginning of the component dispatch

# on_success

on_success(self, result)

Lifecycle event triggered iff execution finishes successfully

# Arguments

  • result: Return value of on_execute event

# refresh_status

refresh_status(self, log_errors=False)

Updates the status.json file with a heartbeat at the current time

# restore_checkpoint

restore_checkpoint(self, checkpoint)

Restores a checkpoint

# Arguments

  • filepath: Checkpoint filepath

# save_checkpoint

save_checkpoint(self, path: str = None, timestep=None) -> Union[bool, str]

Saves components to a checkpoint

# Arguments

  • path: Path where checkpoints should be saved
  • timestep: Optional timestep of checkpoint. If None, timestep will count up automatically

# Returns

Filepath of the saved checkpoint or False if checkpoint has not been saved

# set_seed

set_seed(self, seed=None) -> bool

Applies a global random seed

# Arguments

  • seed: Integer, random seed. If None, self.flags.SEED will be used

To prevent the automatic seeding, you can overwrite the on_seeding event and return False

# Storage

machinable.storage.Storage(url: Union[str, NoneType] = None, directory: Union[str, NoneType] = None, submission: Union[str, NoneType] = None, component: Union[str, NoneType] = None)

# log property

Log interface

# record property

Record interface

# data

data(self, name=None, default=<object object at 0x7f152a427f90>)

Retrieves a data object from the storage

# Arguments

  • name: Name of the data object. If None, a list of available objects is returned

# file

file(self, filepath, default=<object object at 0x7f152a427f90>)

Returns the content of a file in the storage

# Arguments

  • filepath: Relative filepath
  • default: Optional default if file does not exist

# get_local_directory

get_local_directory(self, append='', create=False)

Returns the local storage filesystem path, or False if non-local

# Arguments

  • create: Boolean, if True path is being created if not existing

# Returns

Local filesystem path, or False if non-local

# get_path

get_path(self, append='', create=False) -> str

Returns the storage's relative path

# Arguments

  • append: String, optional postfix that is appended to the path
  • create: Boolean, if True path is being created if not existing

# get_records

get_records(self, scope=None, events=None, created_at=None) -> Union[dict, machinable.storage.record.Record]

Creates or returns an instance of a record writer

# Arguments

  • scope: Name of the record writer. If None, a dict of all registered writers will be returned
  • events: Event callback

# get_stream

get_stream(self, path, mode='r', *args, **kwargs)

Returns a file stream on the storage

# Arguments

  • path: Relative file path
  • mode: Python file mode *args, **kwargs: Optional arguments passed into stream open()

# get_submission

get_submission(self, component=False, or_fail=False)

Returns the Submission

# Arguments

  • component: Component to be returned if the URL is a submission containing multiple components. For example, set to 0 or -1 to retrieve first or last in the collection respectively
  • or_fail: Boolean, by default None is returned if the submission does not exist. If True, an Exception will be raised instead

# get_url

get_url(self, append='', create=False) -> str

Returns the storage's url path

# Arguments

  • append: String, optional postfix that is appended to the URL
  • create: Boolean, if True path is being created if not existing

# has_file

has_file(self, filepath) -> bool

Checks if a file exists

# Arguments

  • filepath: String, file name

# has_log

has_log(self) -> bool

Determines whether log has been written

# has_records

has_records(self, scope='default') -> bool

Determines whether records have been written

# Arguments

  • scope: String, name of the record writer. Defaults to 'default'

# save_data

save_data(self, name, data, overwrite=True)

Stores a data object

# Arguments

  • name: String, name identifier. Supported formats are .json (JSON), .npy (numpy), .p (pickle), .txt (txt)
  • data: The data object
  • overwrite: Boolean, if False write will be appended to file if existing

# save_file

save_file(self, filepath, data, overwrite=True)

Writes data to file

# Arguments

  • filepath: String, filepath with extension. Supported formats are .json (JSON), .npy (numpy), .p (pickle), .txt (txt)
  • data: The data object
  • overwrite: Boolean, if False write will be appended instead of overwritten

# Record

machinable.storage.record.Record(storage, scope='default', events=None, created_at=None)

Tabular record writer

TIP

Within components, the default record writer becomes available as self.record

Exposes the standard interface of Python loggers, e.g. info, debug etc.

# Arguments

  • storage: machinable.Storage
  • scope: String, optional scope name

# filepath property

Returns the record filepath relative to the storage

# empty

empty(self)

Whether the record writer is empty (len(self.record) == 0)

# save

save(self, echo=False, force=False)

Save the current row

# Arguments

  • echo: Boolean, if True the written row will be printed to the terminal
  • force: Boolean, if True the row will be written even if it empty

# Returns

The row data

# timing

timing(self, mode='iteration', end=None, return_type='seconds')

Get timing statistics about the records

# Arguments

  • mode: String, 'iteration', 'avg' or 'total' that determines whether the last iteration, the average iteration or the total time is collected
  • end: Mixed, end of the timespan; if None, datetime.now() is used
  • return_type: String, specifying the return format 'seconds', 'words', or 'period' (pendulum.Period object)

Returns: See return_type

# update

update(self, dict_like=None, **kwargs)

Update record values using a dictionary. Equivalent to dict's update method.

# Arguments

  • dict_like: dict, update values

# write

write(self, key, value)

Writes a cell value

self.record.write('loss', 0.1)
#### is equivalent to
self.record['loss'] = 0.1

# Arguments

  • key: String, the column name
  • value: Value to write

# Log

machinable.storage.log.Log(storage)

A simple logging interface

TIP

Within components, the log becomes available as self.log

Exposes the standard interface of Python loggers, e.g. info, debug etc.

# Arguments

  • storage: machinable.Storage

# logger property

The underlying logging interface

# debug

debug(self, msg, *args, **kwargs)

Logs a message with level DEBUG on this logger.

# error

error(self, msg, *args, **kwargs)

Logs a message with level ERROR on this logger.

# info

info(self, msg, *args, **kwargs)

Logs a message with level INFO on this logger.

# warning

warning(self, msg, *args, **kwargs)

Logs a message with level WARNING on this logger.