# Experiments

The experiment interface allows for dynamic configuration adjustments in a clear and systematic way. It eliminates global for-loops or manual re-runs of python scripts with different command-line parameters.

# Defining experiments

An experiment entails an arbitrary amount of components that can be added using the component method which specifies the component name as defined in the machinable.yaml.

Experiment().component('A').component('B').component('C')

TIP

Note that all experiment methods can be chained, e.g. Experiment().component('B').repeat(5).

# Combining, repeating and splitting

The components can be proliferated using the repeat() functionality, for example:

Experiment().component('A').repeat(3)
# -> [A], [A], [A]

Note that a repeat includes every components of the task and that it can be used recursively:

Experiment().component('A').component('B').repeat(2)
# -> [A, B], [A, B]
Experiment().component('A').component('B').repeat(2).repeat(2)
# -> [[A, B], [A, B]], [[A, B], [A, B]]

machinable will inject the flags REPEAT_NUMBER and REPEAT_TOTAL into each of the components accordingly. By default, the repeats are independent meaning machinable will inject a different SEED flag for each of the repeated components.

Another form of repetition is induced by the split() method that injects SPLIT_SEED, SPLIT_NUMBER and SPLIT_TOTAL flags into the components. Using the flag information, you can implement customized splitting operations. For example, to implement a cross-validation algorithm the components can split the dataset using the SPLIT_SEED and use the split that is specified by the SPLIT_NUMBER for training. As a result, the split components will conduct a k-fold cross-validation.

# Adjusting configuration

A key feature of Experiments is the programmatic adjustment of configuration; you can use experiments to capture a specific execution of components with a particular configuration -- an experiment as-they-say.

In the simplest case, you can use a dictionary to override the default component configuration as defined in the machinable.yaml.

components:
  - optimization:
      learning_rate: 0.001
      data:
        name: cifar10
        augmentation: False
      ~mnist:
        data:
          name: mnist
      ~alexnet:
        learning_rate: 0.1
        data:
          name: imagenet

Experiment().component('optimization', {'dataset': 'mnist', 'learning_rate': 0.5})

# Versions

Since dictionaries can be cumbersome, it is possible to pass configuration patches directly as YAML:

Experiment().component("optimization", """
learning_rate: 0.1
data:
  name: mnist
""")

# is equivalent to:

Experiment().component("optimization", {
    "learning_rate": 0.1, 
    "data": {"name": "mnist"}
})

However, rather retrieving YAML from variables, it is more suitable to define the versions directly in the machinable.yaml. To define a version, specify the configuration difference under a key that starts with ~, for instance:

~alexnet:
  learning_rate: 0.1
  data:
    name: imagenet

The version can then be accessed using its key name ~<version-name>, for example:

Experiment().component('optimization', '~alexnet')
# is equivalent to 
Experiment().component('optimization', {
  'learning_rate': 0.1, 
  'data': {'name': 'imagenet'}
})

It is also possible to reference mixins configuration using _<mixin-name>_:

Experiment().component('optimization', '_imagenet_')

You can merge and iterate over configuration adjustments using tuples and lists.

# Merging

Tuples are interpreted as a merge operators that merge the containing elements together. Consider the following example:

Experiment().component('optimization', ({'a': 1}, {'a': 2, 'b': 3}))
# is equivalent to                     ^ - merge operation ------ ^
Experiment().component('optimization', {'a': 2, 'b': 3})

# Iterating

To compare two different learning rates, you could declare the following experiment:

Experiment().component('optimization', {'learning_rate': 0.1})\
            .component('optimization', {'learning_rate': 0.5})

Since the experiment will execute every components with their adjusted configuration, the optimization will proceed with a learning rate of 0.1 and 0.5. To express these types of iterations more effectively, you can use lists to induce the same repetition as above:

Experiment().component('optimization', [{'learning_rate': lr} for lr in (0.1, 0.5)])
#                                      ^ -- list of patches induces a repeat ---- ^

# Combinations

Taking these concepts together, experiments allow you to manage complex configuration adjustments in a flexible way. Consider the following example:

Experiment().component('optimization', ('~alexnet', '~mnist', {'learning_rate': 0.5}))

This would result in the following components configuration:

learning_rate: 0.5
network: alexnet
data:
    name: mnist

Can you work out what the following experiment entails?

Experiment().component('optimization', [
  (
    '~mnist', 
    {'network': 'resnet', 'learning_rate': lr * 0.01 + 0.1}
  ) 
  for lr in range(10)
]).repeat(2)

# Other component options

In summary, the Experiment.component() method has the following signature:

(
  name,       # components name, see above
  version,    # configuration adjustment, see above
  checkpoint, # see below
  flags       # see below
)

# Checkpoints

If the checkpoint option is specified, machinable will trigger the components's on_restore event with the given filepath. This allows for restoring components from previously saved checkpoints

# Flags

In addition to the default execution flags, you can use the flags parameter to extend the flags dictionary of the components. You can use the ENVIRON flag to set environment variables that are necessary for the execution. The variables will be set before the component is imported. Consider the following example:

Experiment().component(
    "flag_example",
    flags=dict(
        example="test",  # regular flag
        ENVIRON={        # os.environ.update argument
            "KEY": "VALUE", 
            "EXAMPLE": "ENABLED"
        },
    ),
)

# Sub-components

You can organise components in a hierarchical way using the components() method which allows to add one or a list of many [sub-components]. You can specify the sub-components with the same arguments of the component() method, for example:

Experiment.components(('alexnet', {'lr': 0.1}), [('imagenet_component', {'augmentation': True})])

For a comprehensive description of the Experiment API, consult the reference.

# Hyperparameter tuning

While Experiments allow for simple configuration iteration, complex hyperparameter tuning is supported through Ray tune (opens new window) using the tune() method of the Experiment object:

import numpy as np

import machinable as ml

random_search = (
    ml.Experiment()
    .component("optimization")
    .tune(
        stop={"episodes_total": 50},
        num_samples=50,
        config={
            "stepsize": lambda spec: np.random.uniform(1e-6, 0.1),
            "noise_std": lambda spec: np.random.uniform(1e-6, 0.1),
            "l2coeff": lambda spec: np.random.uniform(1e-6, 0.1),
        },
    )
)

grid_search = (
    ml.Experiment()
    .component("optimization")
    .tune(
        stop={"accurary": 80},
        num_samples=100,
        config={"learning_rate": {"grid_search": [0.1, 0.05, 0.01]}},
    )
)

Please refer to Ray's project documentation (opens new window) to learn more about available options.

← Overview Engines →