cloudsoft.io

Nested and Custom Workflow

The type: workflow can be used to create new step types and save them to the AMP type registry. New step types can be used in the same way as the standard step types. This allows common workflow logic to be saved and re-used as steps in other workflow. The usual properties of steps – inputs, outputs, error handling – are all supported, and it supports defining parameters and even custom shorthand templates.

This step type can also be used directly within a workflow to run a “nested workflow”, for instance to structure code and to isolate a sequence of steps or to apply on-error behavior to a group of steps.

Nested and custom workflows are not permitted to access data from their containing workflow; instead, they accept an input block like other steps.

This type permits all the common step properties and all the workflow settings properties, plus a few others, target, concurrency, parameters, and shorthand as described below.

Basic Usage in a Workflow

When used in a workflow, the nested workflow should be defined as a list in the steps key. An input section can also be defined to pass data from the outer workflow to the nested workflow.

For example:

- log This is about to run a nested workflow
- let x=1
- let y=2
- type: workflow
  input:
    x: ${x}
  steps:
    - log This is a nested workflow, able to see x=${x} from input but not y from the output workflow.
  on-error:
    # error handler which runs if the nested workflow fails (i.e. if any step therein fails and does not correct it)

Loops and Parallelization

The workflow type can also be used to run a sequence of steps on multiple targets. If given a target value, AMP will run the workflow against that target or targets, as follows:

  • If the target is a managed entity, e.g. $brooklyn:entity("some-child"), the nested workflow will run in the scope of that entity. It will be visible in the UI under that entity, and references to sensors and effectors will be aginst that entity.
  • If the target is any value which resolves to a list, it will be run against every entry in the list, with the variable expression ${target} available in the sub-workflow to refer to the relevant entry
  • If the target is children or members it will run against each entity in the relevant list
  • If the target is of the form M..N for integers M and N it will run for all integers in that range, inclusive (so the string 1..4 is equivalent to the list [1,2,3,4])

The scratch variables target and target_index are available referring to to the specific target and its 0-indexed position. These names can be overridden with the target_var_name and target_index_var_name keys.

Where a list is supplied, the result of the step is the list collecting the output of each sub-workflow.

If a condition is supplied when a list is being used, that condition is applied to each entry in the list. The workflow step itself will always appear to run, but might have 0 nested workflows, and in that case will return the empty list. If the condition is intended to block the list from being evaluated, and should be evaluated in the containing workflow’s scope, consider using a subworkflow with the outer condition containing the workflow. An example of this is included below, using service.isUp with children.

Associated Step Types

The foreach type is a simplified variant of workflow when recursing over a list, taking the same arguments as workflow but supporting a shorthand syntax where the target_var_name and target can be specified as in foreach VAR_NAME in TARGET. A single step can be supplied as do STEP at the end; either that or steps is required (but not both), and that step or those steps are run for all items in the target list.

The subworkflow type is another simplified variant that does not allow any target (so no recursing over a list and no concurrency), and which – unlike the other workflow step types – shares variables with the calling workflow. All variables from the immediate containing workflow can be accessed and updated in a subworkflow, and new variables defined in the subworkflow are subsequently available in the outer workflow. The steps are required/ A name is frequently given for readability, as are condition or on-error entries to apply a condition or error-handler to the block of steps. If no step type is defined, but steps are defined, this subworkflow step type is assumed.

The if type acts like subworkflow but allows specifying a condition in shorthand.

Example

- foreach x in 1..3 do return ${x}

The above loop will return [1,2,3] and proceed to the next step.

# foreach, nested workflow
- step: foreach x in 1..3
  steps:
  - let x = ${x} + 1
  - return ${x}

The above loop will return [2,3,4] and proceed to the next step. It is the same as the previous except it is supplying multiple steps to run on each iteration. Note that the assignment to ${x} does not side-effect the loop, and nor is it available to the outer workflow, when using foreach or workflow. Use ${output} to access the result, and see the reducing section below.

# subworkflow
- let x = 1
- steps:
  - let x = ${x} + 1
  - return ${x}
- let x = Not run, but if it were it could access ${x}

The above implicit lightweight subworkflow does run in the parent workflow’s context, so it will access and update the same instance of ${x}, and it will return from the workflow, not running the next step.

# if
- let x = 1
- if ${x} == 1 then let x = ${x} + 1
- return ${x}

The above if step can access and update outer variables, like subworkflow, providing a convenient shorthand for simple conditions and steps. It is functionally identical to this:

- let x = 1
- condition:
    target: ${x}
    equals: 1
  steps:
  - let x = ${x} + 1
- return ${x}

Reducing

Each nested workflow runs in its own scope and does not share workflow variables with the parent, apart from values specified as input, or with other iterations of a loop. Where it is desired to share variables across iterations, the key reducing can be supplied, giving a map of variable names to be shared and their initial values.

When reducing, the indicated variables are available as scratch workflow variables in the calling workflow.

This cannot be used with subworkflow, as all variables are considered reduced in a subworkflow, and it cannot be used with concurrency as that risks race conditions in updating the reduced variables. (You can use inputs with both to provide new local variables.)

Example

- step: foreach x in 1..3 do let sum = ${sum} + ${x}
  reducing:
    sum: 0

The above loop will return 6.

Concurrency

By default nested workflows with list targets run sequentially over the entries, but this can be varied by setting concurrency. The following values are allowed:

  • a number to indicate the maximum number of simultaneous executions (with 1 being the default, for no concurrency)
  • the string unlimited to allow all to run in parallel
  • a negative number to indicate all but a certain number
  • a percentage to indicate a percentage of the targets
  • the string min(...) or max(...), where ... is a comma separated list of valid values

This concisely allows complicated – but important in the real world – logic such as max(1, min(50%, -10)) to express running concurrently over up to half if more than twenty, otherwise all but 10, and always allowing 1. This might be used for example to upgrade a cluster in situ, leaving the larger of 10 instances or half the cluster alone, if possible.
If the concurrency expression evaluates to 0, or to a negative number whose absolute value is larger than the number of values, the step will fail before executing, to ensure that if e.g. “-10” is specified when there are fewer than 10 items in the target list, the workflow does not run. (Use “max(1, -10)” to allow it to run 1 at a time if there are 10 or fewer.)

Note: Concurrency cannot be specified when reducing.

Example

This example invokes an effector on all children which are service.isUp, running in batches of up to 5 but not more than a third of the children at once:

- type: workflow
  target: children
  concurrency: max(1, min(33%, 5))
  condition:
    sensor: service.isUp
    equals: true
  steps:
    - invoke-effector effector-on-children

As noted above, the condition here is evaluated on each child. To have it evaluated on the parent context instead, you can write:

- condition:
    sensor: service.isUp
    equals: true
  steps:
  - type: workflow
    target: children
    concurrency: max(1, min(33%, 5))
    steps:
    - invoke-effector effector-on-children

Defining Custom Workflow Steps

This type can be used to define new step types and add them as new types in the type registry. The definition must specify the steps, and may in addition specify:

  • parameters: a map of parameters accepted by the workflow, with the key the parameter name, and the value map possibly empty or providing optional type (default string), defaultValue (default none), required (default false), description (default none), and/or constraints
  • shorthand: a template, as described below
  • output: an output map or value to be returned by the step, evaluated in the context of the nested workflow

When this type is used to define a new workflow step, the newly defined step does not allow the steps or any of the parameters listed above to be overridden. Instead it accepts the parameters defined in the parameters key of the definition. It also accepts the standard step keys such as input, timeout on on-error. A user of the defined step type can also supply output which, as per other steps, is evaluated in the context of the outer workflow, with visibility of the output from the current step.

When supplying a workflow in contexts where a workflow is already expected, such as in a config key that takes a workflow (a Java CustomWorkflowStep), it is not necessary to specify the type: workflow, and additionally, if the only things being set is steps, those steps can be provided as a list without the steps keyword. Internally a list will coerce to a workflow by interpreting the list as the steps.

Shorthand Template Syntax

A custom workflow step can define a shorthand template which permits a user to use the workflow step as a string rather than a map, even with parameters. The shorthand template syntax consists of a sequence of the following tokens:

  • ${VAR} - to set VAR, which should be of the regex [A-Za-z0-9-]+(.[A-Za-z0-9-]+)*, with dot separation used to set nested maps
  • ${VAR...} - as ${VAR} but allowing it to match multiple words
  • "LITERAL" - to expect the user to supply the exact token LITERAL; this should include spaces if spaces are required
  • [ <TOKENS> ] - to indicate that a sequence of <TOKENS> is optional; parsing is attempted first with this block, then without it

Example

A simple example to say hello is as follows:

id: greet
type: workflow
shorthand: ${name...} [ " with " ${greeting} ]
parameters:
  name:
    required: true
  greeting:
    defaultValue: Hello
steps:
- log ${greeting} ${name}

With this added as a registered type, workflows can write:

- type: greet
  input:
    name: Angela

The result will be the same as log Hello Angela. The shorthand template spec also allows greet Angela for the same, or exercising the optional block, greet Zachary Jones with Howdy to log Howdy Zachary Jones.

This is a trivial single-step example but shows the power of creating custom workflows, especially with parameters and shorthand templates. The examples and the workflow settings include more realistic illustrations of custom workflow steps.

Writing Workflow Steps in Java

The most common way to define custom workflow types is as workflow, using the primitives defined here, and delegating to custom containers where the logic is best done in a higher-level programming language. This avoids any language bias and the need to learn AMP interfaces. However it is supported to provide custom workflow step types as Java classes in a bundle.

To write a Java workflow step type, provide a class extending WorkflowStepDefinition, providing implementations for the following methods:

Object doTaskBody(WorkflowStepInstanceExecutionContext context);
void populateFromShorthand(String value);
boolean isDefaultIdempotent();

The first of these does the work of the step, resolving inputs and accessing context as needed via context. The second handles providing a custom shorthand, as described above; it can call to a superclass method populateFromShorthandTemplate(TEMPLATE, value) with the TEMPLATE for the class, if shorthand is to be supported. Finally, the third returns whether the step is idempotent, that is if the custom step is interrupted, can AMP safely recover simply by rerunning it with the same inputs. As described here, it is recommended to write the step so that it is idempotent if possible.

Once written, the class should be added to the AMP Catalog, e.g. for a custom java step called com.acme.YourJavaWorkflowStep with shorthand name your-step, create a catalog.bom such as the following, and br catalog add catalog.bom:

brooklyn.catalog:
  bundle: your-step-bundle
  version: "1.0.0-SNAPSHOT"
  items:
  - id: your-step
    format: java-type-name
    itemType: bean
    item:
      type: com.acme.YourJavaWorkflowStep

AMP and Maeztro are developed and supported by Cloudsoft Corporation.