Cluster reuse

When running traditional Cromwell or CromwellOnAzure, each task is required to configure a Docker setting. This means that when a task is executed, a Docker instance will be launched and it will be deleted when a task is finished. In Seqslab V3, we provide more flexibility to launch a single task or multiple tasks in one cluster so as to reduce the overhead costs of cloud resource provisioning and to reduce management complexity.

Workflow diagram

The following workflow diagram represents an example of a WE2 job which introduces how Seqslab V3 enables cluster reuse through different configurations. In this example, we have four levels of workflows and tasks. We name them based on their level and chronological order, which means that SubWorkflow11 represents the first subworkflow at the first level. In addition, both Main Workflow and SubWorkflow21 contain a scatter operation. We use Task22_i, Task23_i, and Task31_i to represent tasks generated by the scatter.

cluster reuse diagram

Cluster settings

Because of the varying needs in computing resources, SeqsLab allows you to provision Spark cluster resources at three different call levels. You can choose from a list of Spark cluster resources and each one contains a number of spot or dedicated virtual machines on a specific backend such as Azure. For example, the cluster acu-m4 has one dedicated Azure Standard_D12_v2.

  1. Main workflow level

    The cluster configured at the main workflow level is the default resource used to execute all tasks submitted to it. It is mandatory to have one default main workflow cluster setting inside every SeqsLab job.

  2. Subworkflow level

    A cluster configured at the subworkflow level serves as the default resource for all tasks under this subworkflow. For example, if you configure a cluster at SubWorkflow_11, then by default, Task_21, Task_31, and Task_32 will be submitted to that Spark cluster. If SubWorkflow_11 or SubWorkflow_21 contains a scatter operation, then all tasks, such as Task_21_1, …, Task_21_i, will be submitted to the SubWorkflow_11 cluster.

  3. Task level

    A cluster configured at the task level only serves that specific task and will be deleted once the task is finished. If the task is inside a scatter operation, then each task will launch a cluster for job execution.

Image settings

Instead of configuring a Docker image inside a task as is typically done in the traditional WDL syntax, you should associate an uploaded Docker image with a WDL file when registering your tools through TRS on SeqsLab. By default, all tasks and workflows inside the WDL will launch a cluster with a configured Docker image when executed.

Important

Cluster reuse would not work as expected when an image that was configured at the task level differs from the image in a target cluster, since a Spark cluster can only be initialized by one image. In this case, another Spark cluster, with the same call name as what an image is configured with (either as a subworkflow or task), will be provisioned and the task will be submitted to this.

The following are some examples that show how cluster management works on SeqsLab:

Example 1: Same cluster settings and different image settings at the subworkflow level

Only the Main Workflow is configured with ClusterA. In addition, Main Workflow, SubWorkflow11, SubWorkflow12_i, SubWorkflow_21 are configured using Image1, Image2, Image1,and Image3, respectively. The actual execution details are as follows:

Cluster name

Cluster setting

Image setting

Tasks executed

Subworkflow11-RunID

ClusterA

Image2

Task21

MainWorkflow-RunID

ClusterA

Image3

Task31_1, …, Task31_i

Subworkflow12_1-RunID

ClusterA

Image1

Task22_1, Task23_1

Subworkflow12_i-RunID

ClusterA

Image1

Task22_i, Task23_i

Example 2: Same cluster settings and different image settings at the task level

Only the Main Workflow is configured with ClusterA. In addition, Main Workflow, Task21, Task31_i, Task32, Task22_i, and Task23_i are configured using Image1, Image2, Image3, Image1, Image1, and Image4, respectively. The actual execution details are as follows:

Cluster name

Cluster setting

Image setting

Tasks executed

Task21-RunID

ClusterA

Image2

Task21

Task31_1-RunID

ClusterA

Image3

Task31_1

Task31_i-RunID

ClusterA

Image3

Task31_i

MainWorkflow-RunID

ClusterA

Image1

Task32, Task22_1, …, Task22_i

Task23_1-RunID

ClusterA

Image4

Task23_1

Task23_i-RunID

ClusterA

Image4

Task23_i

In this case, we can observe that MainWorkflow-RunID is reused between Task32 and Task22_i. However, this is not a good practice for cluster and image setting. It would be better to group Task32 and Task22 together in a subworkflow so that a single image set at the subworkflow level would be sufficient.