# Engines
machinable provides different execution engines that enable seamless parallel and remote execution. You can also implement your own engine to customize the execution to your needs.
# Native
By default, the native engine is used. The engine supports parallel execution using Python's multiprocessing.
execute(Experiment().component('demo'), engine='native')
# use a maximum of 10 processes in parallel
execute(Experiment().component('demo'), engine='native:10')
# Ray
Ray (opens new window) is a powerful distributed framework that supports local as well as remote execution on clusters like AWS and Google Cloud instances (opens new window). Please refer to Ray's documentation (opens new window) to learn more.
To execute using the connected Ray backend use the Ray driver:
execute(Experiment().component('demo'), engine='ray')
# Slurm
The Slurm Workload Manager is a free and open-source job scheduler, used by many computing clusters. To execute schedule jobs on a Slurm cluster, specify the resources in the component argument.
execute(
Experiment().component(
'demo',
resources=[
"-p bulk",
"-c 8"
"--mem=10GB"
]
),
engine='slurm'
)
# Dry
The dry engine will simulate but not actually preform the execution.
# Detached, Remote
The detached and remote wrappers let you push execution into the background or onto remote machines.
Refer to the engine reference to learn more about all available options.
# Custom engines
You can implement a custom engine as a subclass of machinable.Engine
and pass them as an argument.
from machinable import Engine
class CustomEngine(Engine):
# implement abstract methods
execute(Experiment().component('demo'), engine=CustomEngine())