23.23. flexiflow - FlexiFlow

The following documentation is for FlexiFlow (flexiflow) content package at version v4.6.0-beta01.97+g1e33864277dc9bbb839f71f1eff994a0c4f05c23.

The FlexiFlow system allows an operator to dynamically inject new tasks in to an existing workflow based on a Parameter. This allows for other content to build up Tasks to add an existing workflow flexibly. Some examples may include adding specific tasks based on classification results from the Classify system, or other paths.

Stage Based Manipulation:

The Tasks that are defined in the parameter (specified by the flexiflow/list-parameter) will be composed and added to the task list. These will be dynamically inserted between the flexiflow-start and flexiflow-stop tasks.

Composed means the value of that parameter is the aggregation of all occurances of that parameter at all levels of the system, (e.g. From the parameters on the machine, then profiles on machines, then parameters on the stage, …). This allows multiple resources to provide the FlexiFlow tasks that will eventually be run on the system. Typically, Parameter orders of precedence would override earlier occurrences of the Parameter values. FlexiFlow does not follow that standard path.

The Param that is referenced by flexiflow/list-parameter should be a simple array type Param. It lists each Task that should be executed in the listed sequence by the FlexiFlow process at that Stage in the Workflow.

Workflow Based Manipulation

The Workflow that is defined in the Param named flexiflow-workflow will be run when the stage of the same name is run. This allows the operator to jump in to another Workflow in an existing workflow.

23.23.1. Operating FlexiFlow Stage Based Manipulation

Here is a brief outline of example usage of the FlexiFlow system utilizing the RackN defined stage. In this example, we will define tasks to run in the flexiflow-stage. Using the Stage and Task Injection

In this example we will create a “my-flexiflow-discovery” workflow.

  color: grey
  icon: magic
Name: my-flexiflow-discovery
Description: My flexiFlow Discovery Workflow
  - discover
  - ipmi-inventory
  - raid-inventory
  - network-lldp
  - inventory
  - classify
  - flexiflow-stage
  - sledgehammer-wait

Note that the flexiflow-stage is run after the classify stage. A FlexiFlow stage can be added anywhere that conceptually makes sense. Using the FlexiFlow Stage

The FlexiFlow system is comprised of a single stage named flexiflow-stage. The stage must contain ONLY the flexiflow-start and flexiflow-stop Tasks. The flexiflow-start stage will dynamically inject the desired tasks in to the Workflow for the operator. DO NOT add any tasks between these start and stop tasks.

You can choose to use the provided flexiflow-stage, and customize it’s use by adding Params to the Machine that define the Tasks that will run, or you can create a custom Stage. The provided FlexiFlow stage is used the same way that a custom named stage would be used.

To define the custom list of tasks to execute, simply follow the below steps:

  • Add the flexiflow/list-parameter param to the Machine
  • The value of which is another Parameter that is an Array of Strings that lists tasks to add
  • An example is superflex (which would be the Array of Strings)
  • Insure you create a Param with Array of Strings as the type definition (eg superflex, defined as Array of Strings)
  • Add the Param (eg superflex) to the Machine with a list of tasks in the array

If you create a custom stage, you can optionally embed the Params that control defining the list of Tasks to insert and/or the actual task list to insert.

Here is an example Stage in YAML that defines which specific Param will contain the list of tasks to be executed. Note that we are not defining the actual list of task in this example. Presumably a previous Stage (possibly Classify) would build up the value of superflex tasks to execute dynamically.

Name: "my-superflex-stage"
Description: "Perform tasks defined by the 'superflex' Param."
Documentation: |
  The Param 'superflex' is an array that will contain
  the list of flexiflow tasks to run.
  flexiflow/list-parameter: "superflex"
  - "flexiflow-start"
  - "flexiflow-stop"
  color: "orange"
  icon: "magic"
  title: "RackN Content"

Note that as the Documentation field says; the flexiflow/list-parameter for this Stage is defined as superflex. This Parameter must be defined as an array; which is a list of the Tasks to execute during this Stage. The Param can be defined on the Machine in all of the normal ways (directly as a Param, as part of a Profile, or as a Param in the Global Profile, dynamically by other content, etc).

Here is an example stage that defines the custom FlexiFlow stage, which sets the control Parameter name to superflex, and also defines the Tasks that superflex will reference and run.

Name: "my-superflex-stage-v2"
Description: "Perform tasks to run that are defined by the 'superflex' Param."
Documentation: |
  The Param 'superflex' is an array that will contain
  the list of additional tasks to run.  The actual tasks that
  are run are also listed in this Stage.
  flexiflow/list-parameter: superflex
    - task1
    - task2
    - task3
  - "flexiflow-start"
  - "flexiflow-stop"
  color: "orange"
  icon: "magic"
  title: "RackN Content"
..note:: Warning!! The Params that are used in FlexiFlow MUST BE TYPE DEFINED correctly
and added to the system before they are used. This use case pattern does NOT support using adhoc Params. Create Parameter Definition

It is critical that you correctly type define the Parameter that defines the list of tasks to run in advance of using the Parameter. If you do not, you will receive a warning message and the task injection will fail.

An example type definition for the Parameter (using our superflex example above), would look like:

Name: "superflex"
Description: "Defines the flexiflow list parameter."
Documentation: |
  This param defines the list of Tasks to execute for flexiflow
  during the stage run.  If left empty, no tasks will be run.
  type: "array"
    type: "string"
  default: []
  color: "blue"
  icon: "magic"
  title: "RackN Content" Possible Errors

Here is a list of some of the possible errors you may encounter.

Failed to render actions: : template: :35:30: executing "flexiflow_start.sh" at <.ComposeParam>: error calling ComposeParam: Cannot compose freeform param superflex
Updated job for flexiflow-stage:flexiflow:flexiflow-start to incomplete

In this case, the superflex list parameter must be defined in advance of using it, the operator has attempted to use the Parameter as an adhoc param. Create a Param type definition as specified in the Create Parameter Definition section above.

23.23.2. Operating FlexiFlow Workflow Based Manipulation

The FlexiFlow content also allows you to add a Stage to a given workflow. At that Stage, the system will reference the Param named flexiflow/workflow. The value of the param should be the name of an existing Workflow. That workflow will then be executed on the system.

Please review the test-flexiflow-workflow-main Workflow which will dynamically add the test workflow named test-flexiflow-workflow-include. This provides a working example of using the include workflow capability.

23.23.3. Object Specific Documentation params

The content package provides the following params. flexiflow/list-parameter

This defines the parameter that the flexiflow-stage stage will use to reference the list of Tasks to dynamically execute. It essentially operates as a “control pointer” to the real list, allowing for flexible usage of a single defined Param to point to a variable list of tasks.

The Param that it points to must be type defined in the system, and it must be an Array of Strings. flexiflow/maintenance-mode

Indicate that we should go back to local disk. test-flexiflow/parameter

Param for testing flexiflow system. This Param contains an Array of Strings that will define which tasks to dynamically add to the flexiflow-stage. stages

The content package provides the following stages. test-flexiflow-start

Just provides a marker (decorator) that the flexiflow main stage has started. test-flexiflow-stop

Just provides a marker (decorator) that the flexiflow main stage has stopped. test-flexiflow-workflow

Sets the flexiflow/workflow Parameter to test-flexiflow-workflow-include which should then be dynamically included in the test-flexiflow-workflow-main. flexiflow-stage

This Stage implements the FlexiFlow capability. When the task flexiflow-start is run, it will reference the paramtger defined in the flexiflow/list-parameter Param. The Param that is referenced must be defined as an Array of Strings. Each array element must be an existing Task on the system. The list of tasks will be dynamically injected in to this stage and run.

Do not insert any tasks between the start and stop tasks.

This task requires that the Param flexiflow/list-parameter exists on the machine to run successfully. flexiflow-workflow

Allows the operator to set the flexiflow/workflow Parameter to the name of an existing workflow. This will then be added dynamically to the system to be executed immediately at this position in the original workflow. test-flexiflow-stage

This Stage tests the FlexiFlow capability. It defines that the flexiflow/list-parameter is set to test-flexiflow/parameter, and subsequently sets that Paramter to use the test tasks named test-flexiflow/task1 and test-flexiflow/task2. tasks

The content package provides the following tasks. test-flexiflow-start

Just markes the beginning of the test flexiflow workflow. test-flexiflow-stop

Just markes the ending of the test flexiflow workflow. test-flexiflow-task1

This is an example test task that just prints a message and exits successfully. test-flexiflow-task2

This is an example test task that just prints a message and exits successfully. flexiflow-start

This task begins the FlexiFlow stage. No other tasks should preceed it on the stage, and the only other task that should follow is the flexiflow-stop task.

Tasks will be dynamically injected in to the task list after this task, if they have been specified by the appropriate control Params. flexiflow-stop

This task ends the FlexiFLow stage. No other tasks should follow it on the stage, and the only other task that should preceed is the flexiflow-start task.

Tasks will be dynamically injected in to the workflow if they have been specified by the appropriate control Params prior to this task.