21.10. cloud-wrappers - Cloud Wrappers

The following documentation is for Cloud Wrappers (cloud-wrappers) content package at version v4.8.0-alpha00.25+g8e224a5bec085c80968ec27e0332c339a65d0659.

This library contains items that help run Digital Rebar manage machines on public clouds. It uses Terraform tasks to create/delete machines and Ansible tasks join the machine to install the Digital Rebar runner. Once the runners starts, it will collect cloud specific data if a Metadata API is available.

TL;DR: `cloud-provision` will create and attach a machine to a cloud backed instance. `cloud-decommission` will destroy the backing machine but lean the Digital Rebar entry.

21.10.1. Design

Since most cloud operations are highly standardized at tha machine level, the reference code in this wrapper is intented to be a generic workflow that can be used on a wide range of clouds.

The design uses the idea that Digital Rebar can manage a machine instance …

The approach:

  1. Uses Terraform to provide the “give me a machine”
  2. Uses Ansible to SSH into a machine and start the DRP runner
  3. Controls the machine with the DRP runner in a standard way
  4. Uses Terraform to provide the “destroy a machine” controls

The default Terraform plans include only create the most basic machines with SSH access. We anticipate that advanced operators will create or extend these tasks for based on their specific requirements.

Since the approach relies on SSH, the workflows include generating a per machine RSA key pair for each machine.

21.10.2. Prerequistes

21.10.2.1. Inbound Access

The Digital Rebar Server must be at a location that is accessible to the machines being provisioned. This is required because the machines must be able to download the join-up script from the server.

21.10.2.2. Outbound Access

The Digital Rebar Server must be able to SSH into the newly created machines so that it can run the join-up script. Once the script is run, the runner will be able to pull instructions from the DRP and the SSH port could be closed.

21.10.2.3. Catalog Items

The Cloud Wrapper requires Contexts because it uses Runner, Terraform and Ansible Contexts in Docker Containers. This feature also requires that you have enabled a RackN licensed feature.

21.10.3. Setting Up Cloud Wrapper

You must set the cloud/provider to the correct cloud. Beyond that, each cloud will have cloud specific values needed to set security, location, and instance type. In general, safe defaults have been provided where possible.

You must set Security credentials for each cloud! In many cases, you must also provide the name of the SSH user.

21.10.3.1. Required Values

The Cloud Wrapper components require at least two parameters without a default be set:

  1. cloud/provider determine which cloud is to be used.
  2. cloud/token enables the API

21.10.3.1.1. AWS

  • aws/access-secret
  • aws/access-key-id
  • rsa/key-user - typically ec2-user

Additional values, e.g. region, image and instance type, have safe defaults but should be reviewed.

21.10.3.1.2. Google

  • google/credential - this is a copy of contents from the JSON file Google provides
  • rsa/key-user - varies by user, make sure you have the correct default

Additional values, e.g. region, image and instance type, have safe defaults but should be reviewed.

21.10.3.1.3. Linode

  • linode/token

Additional values, e.g. region, image and instance type, have safe defaults but should be reviewed.

21.10.3.2. Optional Values

When possble, the machine on the cloud provider is given the name of the machine in Digital Rebar.

The reference terraform plan will create tags on the cloud provider based on the assigned profiles. It also creates one called “digitalrebar.” This can be handy to find or manage the machines on the cloud provider.

21.10.4. Creating / Destroying VMs

The Cloud Wrapper uses the terraform-apply tasks to create or destroy machines using a reference Terraform template defined in terraform/plan-automation. It then hands off to the ansible-join-up task to install the DRP agent.

These methods are provided as a reference. In practice, operators will likely want to use more customized Terraform plans.

The cloud-inspect task will use the Metadata API to collect additional information and set the cloud/* params where possible. These values will be removed by the cloud-cleanup task on decommission.

21.10.4.1. Create

To create a cloud backed machine:

  1. Starting from a Digital Rebar machine with an active runner.
  2. Run the ‘cloud-provision’ workflow.

If you have created a linode profile with the correct tokens, then this example will create and provision a machine in Linode:

drpcli machines create '{"Name":"foo",
    "Profiles":["linode"],
    "Workflow":"cloud-provision",
    "Meta":{"BaseContext":"runner"}
}'

If the machine is already created, then just the workflow change is required. For example, decommissioned machines are left in a ready to reprovision state.

Note that the machine will have several cloud/* and terraform/* parameters defined after the process completes. These are important to operation and can be used by downstream workflows.

21.10.4.2. Destroy

From a cloud backed machine:

  1. Starting from a Digital Rebar machine with an active runner.
  2. Run the ‘cloud-decommission’ workflow

Note that the machine will have removed the cloud/* parameters. If these are missing, the machine can be assumed to no longer have a backing cloud instance.

21.10.5. Extending the Model

This approach uses terraform/plan-automation to determine which Terraform plan to run for provisioning. The included cloud-provision-reference.tf.tmpl template has basic safe provisioning operations for all clouds defined in the cloud/provider parameter.

By default, the cloud-verify task will add this value to the provisioned machine if it has not been defined by the operator. The terraform-apply task will remove the parameter (if defined at the machine level) on destroy.

Operators who wish to override this behavior need to define terraform/plan-automation at the profile or global level. This will prevent the system from adding the reference template.

The behavior is provided to eliminate the need to clone or copy the cloud-provision workflow unless very different operations are required.

21.10.6. Future Work

At some point, the cloud-provision could include an automatic “flexiflow” operation to start another workflow.

21.10.7. Object Specific Documentation

21.10.7.1. params

The content package provides the following params.

21.10.7.1.1. google/zone

Provisioning to Zone for Google Cloud

21.10.7.1.2. pnap/client-id

The ID required by cloud provider to act aginst the API

21.10.7.1.3. pnap/client-secret

The token required by cloud provider to act aginst the API

21.10.7.1.4. aws/access-key-id

The ID needed to use the AWS secrete

Known types: aws

21.10.7.1.5. azure/image

Image Information for Azure including * publisher * offer * sku * version

21.10.7.1.6. azure/password

API Password

21.10.7.1.7. cloud/provider

The cloud provider detected by join-up script in cloud-provision

Custom types are supported by adding Terraform plan template ‘cloud-provision-[provider].tf.tmpl’

Implemented types:

  • aws (Amazon Web Services)
  • google (Google Compute Engine)
  • linode
  • azure (Microsoft Cloud)
  • digitalocean
  • pnap (Phoenix NAP)

Expand this list as new types are added!

21.10.7.1.8. google/credential

The token required by cloud provider to act aginst the API

21.10.7.1.9. terraform-var/instance_id

Storage the Cloud Istance ID generated when Terraform output

21.10.7.1.10. azure/tenant

API Tenant

21.10.7.1.11. digitalocean/image

Provision Digital Ocean O/S Image

Retrieve list of images: curl -X GET -H “Content-Type: application/json” -H “Authorization: Bearer $DO_TOKEN” “https://api.digitalocean.com/v2/images” | jq .images[].slug

21.10.7.1.12. google/instance-type

The type of resource assigned by the cloud provider

21.10.7.1.13. google/project-id

NO DEFAULT! You must supply a project name.

Provisioning to Project for Google Cloud

21.10.7.1.14. linode/token

The token required by cloud provider to act aginst the API

21.10.7.1.15. cloud/dr-install

Internal operations flag used to identify if cloud provision is used This is set inside on the user’s behalf in the cloud-site-* stages

21.10.7.1.16. cloud/instance-id

The ID reference from cloud provider

21.10.7.1.17. cloud/instance-type

The type of resource assigned by the cloud provider

21.10.7.1.18. cloud/placement/availability-zone

The location of resource assigned by the cloud provider

21.10.7.1.19. pnap/os

Provision Phoenix NSP O/S Image from available list

21.10.7.1.20. azure/appId

App ID

21.10.7.1.21. digitalocean/token

The token required by cloud provider to act aginst the API

21.10.7.1.22. google/region

Provisioning to Region for Google Cloud

21.10.7.1.23. aws/ami-id

Provision AWS O/S Image

21.10.7.1.24. aws/instance-type

The type of resource assigned by the cloud provider

21.10.7.1.25. aws/security-groups

These should alreday be defined. If omitted, will create a basic security group with SSH, HTTP and HTTPS inbound

21.10.7.1.26. azure/region

Region

21.10.7.1.27. linode/root-password

Password for Nodes

21.10.7.1.28. mist/api-token

The ID needed to use the Mist.io API

21.10.7.1.29. aws/region

Provisioning to Region for AWS

21.10.7.1.30. cloud/provider-icons

List of icons to use for Cloud Providers

Implemented clouds: aws, google, linode, azure, default

Expand this list as new types are added!

21.10.7.1.31. cloud/public-ipv4

Address assigned by the Cloud Provider

21.10.7.1.32. digitalocean/size

Provision Digital Ocean Droplet Size

Retrieve list of sizes: curl -X GET -H “curl -X GET -H “Content-Type: application/json” -H “Authorization: Bearer $DO_TOKEN” “https://api.digitalocean.com/v2/sizes” | jq .sizes[].slug

21.10.7.1.33. google/boot-disk-image

Provision AWS O/S Image

21.10.7.1.34. linode/region

Provisioning to Region for Linode

21.10.7.1.35. pnap/location

Provision PNAP location

21.10.7.1.36. pnap/type

Provision PNAP allocation size

21.10.7.1.37. aws/secret-key

The token required by cloud provider to act aginst the API

21.10.7.1.38. azure/subscription_id

Subscriber ID

21.10.7.1.39. digitalocean/region

Provisioning to Region for Digital Ocean

List of regions: curl -X GET -H “Content-Type: application/json” -H “Authorization: Bearer $DO_KEY” “https://api.digitalocean.com/v2/regions” | jq .regions[].slug

21.10.7.1.40. linode/instance-image

Provision Linode O/S Image

21.10.7.1.41. linode/instance-type

Provision Linode allocation size

retrieve with curl https://api.linode.com/v4/linode/types | jq '.data.[].id'

21.10.7.1.42. azure/size

Size

21.10.7.1.43. cloud/public-hostname

Hostname assigned by the Cloud Provider

21.10.7.2. stages

The content package provides the following stages.

21.10.7.2.1. cloud-site-create

Use cloud-wrapper workflow, but installs DRP server instead

  • Run Cloud-Provision Workflow Components
  • Install New Server for DRP
NOTE: Does NOT start runner on machine. Stays in DRP Contexts!
The DRP server becomes the self-runner!

21.10.7.2.2. cloud-site-destroy

  • Deprovision machines on Endpoint
  • Delete Endpoint Entry

21.10.7.2.3. cloud-inspect

Collect information from cloud metadata API

If a mist/api-token is defined, then attempt to synchronize with Mist.io

21.10.7.3. tasks

The content package provides the following tasks.

21.10.7.3.1. cloud-cleanup

Remove collected internal API information about a cloud instance

21.10.7.3.2. cloud-inspect

Collect internal API information about a cloud instance

21.10.7.3.3. cloud-site-destroy

Relies on Cloud-Decomission Checks to ensure no machines are orphaned before decommissioning

21.10.7.3.4. cloud-validate

Used as a validate that the right params have been set for cloud scenarios and provide operator friendly feedback in the Machine.Description

NOTE: The required cloud/dr-install value is NOT user settable. It is managed by the calling stage.

Developer Notes: * Please add synchronize with the cloud/provider enum! * Sets the terraform/plan-automation value on the machine if missing

21.10.7.3.5. mist-io-sync

Make sure instance is registered with Mist.io

21.10.7.4. workflows

The content package provides the following workflows.

21.10.7.4.1. cloud-decommission

Requires that operator has created a Contexts for “runner”, “terraform” and “ansible” that can run DRP Agent and Terraform then Connect via Ansible

Leaves the machines in runner Context

21.10.7.4.2. cloud-provision

Requires that operator has created a Contexts for “runner”, “terraform” and “ansible” that can run DRP Agent and Terraform then Connect via Ansible

Leaves the machines in machine Context

21.10.7.4.3. cloud-site-create

First, runs the cloud-provision stages to create machine target Then, runs the dr-server-install process to install & create multi-site DRP. Locks machine to prevent accidential operations

NOTE: DOES NOT INSTALL AGENT! DR Server becomes the agent on tbe system!

21.10.7.4.4. cloud-site-destroy

Make sure that all the provisioned machines are decommissioned Then remove the endpoint entry Then cloud-decommission First, runs the cloud-provision stages to create machine target Then, runs the dr-server-install process to install & create multi-site DRP.