Workflow Decorators¶
Introduction¶
Here, we provide code snippets for several decorator-based workflow engines. For a comparison of the pros and cons of each approach, refer to the Workflow Engines Overview page. We describe the specifics of how to use each workflow engine in more detail later in the documentation.
Background¶
To help enable interoperability between workflow engines, quacc offers a unified set of decorators: @job, @flow, and @subflow.
@job¶
A @job in quacc is an individual compute task. In the context of an HPC environment, this can typically be thought of as a compute task you would normally submit as a single job to the scheduler.
@flow¶
A @flow in quacc is a collection of one or more jobs. It defines the logic of the workflow, connecting the output of one job to the inputs of one or more other jobs. As a general guideline, the flow should not contain computations beyond those of the underlying jobs or subflows.
@subflow¶
A @subflow in quacc is any workflow that returns a list of job outputs and where the number of jobs to be called is not necessarily known until runtime.
Take a moment to learn about the main Covalent Concepts, namely the @ct.electron and @ct.lattice decorators, which describe individual compute tasks and workflows, respectively.
| Quacc | Covalent |
|---|---|
@job |
@ct.electron |
@flow |
@ct.lattice |
@subflow |
@ct.electron@ct.lattice |
Take a moment to read the Dask Delayed documentation overview page to get a sense of how the Dask decorators works and the Dask Distributed quickstart page to understand how to submit tasks to a Dask cluster. Namely, you should understand the @delayed decorator and how to interface with the Client.
| Quacc | Dask |
|---|---|
@job |
@delayed |
@flow |
No effect |
@subflow |
@delayed |
Technically, there are some subtle differences between the @delayed decorator and the quacc equivalents, but for the purposes of this tutorial, you can think of them as similar.
Take a moment to read the Parsl documentation's "Quick Start" to get a sense of how Parsl works. Namely, you should understand the concept of a python_app and join_app, which describe individual compute tasks and dynamic job tasks, respectively.
| Quacc | Parsl |
|---|---|
@job |
@python_app |
@flow |
No effect |
@subflow |
@join_app |
Take a moment to read the Prefect documentation's Quickstart and Tutorials. Namely, you should understand the concept of a @task and a @flow, which describe individual compute tasks and workflows, respectively.
| Quacc | Prefect |
|---|---|
@job |
@task |
@flow |
@flow |
@subflow |
@flow |
Take a moment to read the Redun documentation's Design Overview page to get a sense of how Redun works. Namely, you should understand the Task decorator and how to interface with the Scheduler.
| Quacc | Redun |
|---|---|
@job |
@task |
@flow |
@task |
@subflow |
@task |
Take a moment to read the Jobflow documentation's Quick Start to get a sense of how Jobflow works. Namely, you should understand the Job and Flow definitions, which describe individual compute tasks and workflows, respectively.
| Quacc | Jobflow |
|---|---|
@job |
@job |
@flow |
N/A |
@subflow |
N/A |
Warning
Due to the difference in how Jobflow handles workflows compared to other supported workflow engines, any quacc recipes that have been pre-defined with a @flow or @subflow decorator (i.e. have _flow in the name) cannot be run directly with Jobflow.
The quacc descriptors are drop-in replacements for the specified workflow engine analogue, which we will use for the remainder of the tutorials. Based on the value for the WORKFLOW_ENGINE global variable in your quacc settings, the appropriate decorator will be automatically selected. If the WORKFLOW_ENGINE setting is set to None (i.e. quacc set WORKFLOW_ENGINE None), the decorators will have no effect on the underlying function.