Skip to content

Latest commit

 

History

History
316 lines (248 loc) · 12.4 KB

runbook-development.md

File metadata and controls

316 lines (248 loc) · 12.4 KB

Runbook Development

Introduction

This document outlines the syntax and capabilities of runbooks.

Prerequisites

Users developing runbooks should have familiarity with:

  1. How resources are Created-Read-Updated-Delete with the EPCC CLI.
  2. How aliases work in the EPCC CLI.
  3. Familiarity with Go Templates may help for more advanced cases.
  • The EPCC CLI uses the Sprig template functions which is the same library used by Helm templates

Tip

JSON Schema

There is a JSON Schema for Runbooks available here, if you are using IntelliJ or GoLand you can use IntelliJ's built-in support for JSON Templates.

RunbookJSONSchema

Qualified Aliases

You might find runbooks more readable if you prefix an aliases with the JSON API Type (this is the type reported by the API) (i.e., an alias like last_read=entity could be user_authentication_info/last_read=entity)

Syntax

Each runbook consists of a Yaml document that defines a set of actions that can be run:

A Simple Example

# The name of the runbook is used in command lines for the 4th argument (e.g., epcc runbooks run hello-world) 
name: hello-world
description:
  short: "A hello world runbook"
actions:
  # The action name is used in command lines for the 4th argument (e.g., epcc runbook run hello-world create-customer)
  create-customer:
   commands:
   # Commands use a shell like syntax with word splitting
   - epcc create customer name "Hello World" email "hello@world.example"
   - epcc update customer name=Hello_World name "Goodbye"
  reset:
   commands:
   - epcc delete -s customer email=hello@world.example --log-on-success "Successfully Reset Runbook" --log-on-failure "Could not Reset Runbook, maybe the customer does not exist"

A few interesting points to note about the above example:

  1. The runbook defines two actions create-customer and reset.
  2. Each element in the list is run in sequence, one at a time.
  3. Aliases are used to reference previously existing arguments.
  4. Conventionally a lot of runbooks use the action reset, to reset the store to the initial state.
  5. Each line can be one of epcc create, epcc get, epcc update, epcc delete.
    • There is also sleep <N> which can be used to delay execution for a bite, this is a coarse mechanism to handle asynchronous processes.
  6. The --log-on-success and --log-on-failure flags are used to log messages when the command succeeds or fails. This can be useful to provide more context or abstract away problems in the runbook.

Parallel Operations

Runbooks can also process things in parallel, this is done by using multiple commands in a single list element (or line). The syntax here is subtle, and relies on Yaml HereDoc syntax where newlines are preserved.

name: hello-world
description:
  short: "A hello world runbook"
actions:
  create-10-customers:
   commands:
   - |
    epcc create customer name "Stephen Sloan" email "Stephen.Sloan@test.example"
    epcc create customer name "Antwan Griffin" email "Antwan.Griffin@test.example"
    epcc create customer name "Yurem Thornton" email "Yurem.Thornton@test.example"
    epcc create customer name "Lydia Rice" email "Lydia.Rice@test.example"
    epcc create customer name "Jenny Lopez" email "Jenny.Lopez@test.example"
    epcc create customer name "Javier Green" email "Javier.Green@test.example"
    epcc create customer name "Lennon Mills" email "Lennon.Mills@test.example"
    epcc create customer name "Kelly Burns" email "Kelly.Burns@test.example"
    epcc create customer name "Mohamed Love" email "Mohamed.Love@test.example"
    epcc create customer name "Lindsey Sexton" email "Lindsey.Sexton@test.example"

Some important observations are:

  1. All requests to EPCC will be subject to the rate limit passed in via --rate-limit, so you don't need to worry about lots of parallel requests in flight.
  2. Each command (i.e., line starting with a -) is processed at a time, if you are creating lots of things in parallel it might help to split them in multiple commands anyway (though this won't be an issue unless you are using templates).

Variables & Templates

Runbooks support variables and Go Templates with Sprig template functions. These can be used to create more dynamic runbooks.

name: hello-world
description:
  short: "A hello world runbook"
actions:
  create-some-customer-addresses:
  variables:
    customer-id:
      type: RESOURCE_ID:customer
      default: 00000000-feed-dada-iced-c0ffee000000
      required: true
      description:
        short: "Customer with which to create the address"
    country:
      type: STRING
      default: "US"
      description:
        short: "The country the address should be in"
    number_of_addresses:
      type: INT
      default: 10
      description:
        short: "The number of addresses"
  description:
    short: "Create some addresses"
  commands:
    #language=gotemplate
    - |
      {{- range untilStep 0 .number_of_addresses 1 }}
      epcc create customer-address "{{ index $ "customer-id" }}" name "address_{{.}}" first_name "John" \
      last_name "Smith" line_1 "1234 Main Street" county "XX" "postcode" "H0H 0H0" country "{{$.country}}"
      {{- end -}}

The user can then supply the variables on the command line, each type can be one of:

  1. INT - An integer
  2. STRING - A string
  3. RESOURCE_ID: - A resource type (retrieved from epcc resource-list). This is preferable for auto complete purposes.

A few additional notes:

  1. You can use line continuation characters \ to extend one call to the next line
  2. If you would like to use a dashed argument name you need to use a different syntax in the chart, as shown above. {{ index . "dashed-argument-name" }}
  3. The syntax #language=gotemplate is specific for IntelliJ Language Injections, and is optional.

When users type the command, they will see the supplied variables:

#epcc runbooks show hello-world create-some-customer-addresses --help
Create some addresses

Usage:
  epcc runbooks show hello-world create-some-customer-addresses [flags]

Flags:
    --country string              The country the address should be in (default "US")
    --customer-id string          Customer with which to create the address 
  -h, --help                      help for create-some-customer-addresses
    --number_of_addresses string  The number of addresses (default "10")
Templated Arguments

Occasionally it can be used to have a default argument that is dynamic, for instance you might want it to be based on a the current directory, or a random number:

The values supplied in a runbook can be templated using the Go Template (e.g., helm syntax) , the list of functions is avaliable here: https://masterminds.github.io/sprig/

actions:
  create-a-customer:
   variables:
    name:
      type: STRING   
      default: |
         {{ env "USER" -}}

Showing the output

You can use the show command to see what a rendered script looks like (although you will lose any information about concurrency):

#epcc runbooks show hello-world create-some-customer-addresses --number_of_addresses 2 --customer_id "Hello World" --country DE
epcc create customer-address  "Hello World" name "address_0" first_name "John" last_name "Smith" line_1 "1234 Main Street" county "XX" "postcode" "H0H 0H0" country "DE"
epcc create customer-address  "Hello World" name "address_1" first_name "John" last_name "Smith" line_1 "1234 Main Street" county "XX" "postcode" "H0H 0H0" country "DE"

Dynamic Steps

Templates are only renderable for a particular step, but you can instead of generating a command, actually generate a Yaml array, that will be interpreted as steps. This can give you more control over control flow, where-as otherwise you are stuck with all rendered commands being in the same sequence.

name: hello-world
description:
  short: "A hello world runbook"
actions:
  sequential-sleeps:
    variables:
      count:
        type: INT
        default: 2
        description:
          short: "The number of sleeps"
    commands:
      - |2
        {{- range untilStep 0 .count 1}}
          - sleep 1
        {{- end -}}
  concurrent-sleeps:
    variables:
      count:
        type: INT
        default: 2
        description:
          short: "The number of sleeps"
    commands:
      - |
        {{- range untilStep 0 .count 1}}
        sleep 1
        {{- end -}}

It's important for all commands to be indended the same amount, and start with a - this will cause the entire string to be valid as a Yaml array.

You can see the results of executing these runbooks below, without the - the commands all execute concurrently. With the - the execute one after another.

$time epcc runbooks run hello-world concurrent-sleeps  --count 4
INFO[0000] Executing> {{- range untilStep 0 .count 1}}
sleep 1
{{- end -}} 
INFO[0000] Sleeping for 1 seconds                       
INFO[0000] Sleeping for 1 seconds                       
INFO[0000] Sleeping for 1 seconds                       
INFO[0000] Sleeping for 1 seconds                       
real    0m1.302s
user    0m0.441s
sys     0m0.059s
$time epcc runbooks run hello-world sequential-sleeps --count 4
INFO[0000] Executing> sleep 1                           
INFO[0000] Sleeping for 1 seconds                       
INFO[0001] Executing> sleep 1                           
INFO[0001] Sleeping for 1 seconds                       
INFO[0002] Executing> sleep 1                           
INFO[0002] Sleeping for 1 seconds                       
INFO[0003] Executing> sleep 1                           
INFO[0003] Sleeping for 1 seconds                       
real    0m4.289s
user    0m0.405s
sys     0m0.050s

Error Handling

By default, an error in a command will stop execution, when operating concurrently all commands will finish in that block, and then abort. In some cases, it may be the case that errors are unavoidable, in which case the ignore_errors block can be used. In the future more granular error handling could be implemented based on need.

name: hello-world
description:
  short: "A hello world runbook"
actions:
  reset:
   ignore_errors: true
   commands:
    - epcc delete customer email=hello@world.example
    - |
      epcc delete customer email=stephen.sloan@test.example
      epcc delete customer email=antwan.griffin@test.example
      epcc delete customer email=yurem.thornton@test.example
      epcc delete customer email=lydia.rice@test.example
      epcc delete customer email=jenny.lopez@test.example
      epcc delete customer email=javier.green@test.example
      epcc delete customer email=lennon.mills@test.example
      epcc delete customer email=kelly.burns@test.example
      epcc delete customer email=mohamed.love@test.example
      epcc delete customer email=lindsey.sexton@test.example

Standalone Script Files

If you want to run a set of epcc commands without defining a full runbook, you can use exec-script to execute a standalone YAML file:

epcc runbooks exec-script my-script.yml

The YAML file is simply a list of commands, using the same syntax as the commands block in a runbook action. Sequential execution uses one command per list element:

- epcc create -s account name Account1 --auto-fill
- epcc create -s account name Account2 --auto-fill
- epcc create -s account-address name=Account1 --auto-fill
- epcc create -s account-address name=Account2 --auto-fill

Parallel execution uses YAML block scalars (|) so multiple commands share a single list element:

- |
  epcc create -s account name Account1 --auto-fill
  epcc create -s account name Account2 --auto-fill

- |
  epcc create -s account-address name=Account1 --auto-fill
  epcc create -s account-address name=Account2 --auto-fill

The --execution-timeout and --max-concurrency flags are supported, just like runbooks run.

Example script files can be found in external/runbooks/scripts/.

Output JQ

You will need to escape \ when using the --output-jq argument.