24.10. cloud-wrappers - Cloud Wrappers

The following documentation is for Cloud Wrappers (cloud-wrappers) content package at version v4.6.0-alpha00.289+g5c46fa5acc1eb1e07540a21a03a3035140571967.

This library contains items that help run Digital Rebar manage machines on public clouds. It uses Terraform tasks to create/delete machines and Ansible tasks join the machine to install the Digital Rebar runner. Once the runners starts, it will collect cloud specific data if a Metadata API is available.

TL;DR: `cloud-provision` will create and attach a machine to a cloud backed instance. `cloud-decommission` will destroy the backing machine but lean the Digital Rebar entry.

24.10.1. Design

Since most cloud operations are highly standardized at tha machine level, the reference code in this wrapper is intented to be a generic workflow that can be used on a wide range of clouds.

The design uses the idea that Digital Rebar can manage a machine instance …

The approach:

1. Uses Terraform to provide the “give me a machine” 1. Uses Ansible to SSH into a machine and start the DRP runner 1. Controls the machine with the DRP runner in a standard way 1. Uses Terraform to provide the “destroy a machine” controls

The default Terraform plans include only create the most basic machines with SSH access. We anticipate that advanced operators will create or extend these tasks for based on their specific requirements.

Since the approach relies on SSH, the workflows include generating a per machine RSA key pair for each machine.

24.10.2. Prerequistes

24.10.2.1. Inbound Access

The Digital Rebar Server must be at a location that is accessible to the machines being provisioned. This is required because the machines must be able to download the join-up script from the server.

24.10.2.2. Outbound Access

The Digital Rebar Server must be able to SSH into the newly created machines so that it can run the join-up script. Once the script is run, the runner will be able to pull instructions from the DRP and the SSH port could be closed.

24.10.2.3. Catalog Items

The Cloud Wrapper requires Contexts because it uses Runner, Terraform and Ansible Contexts in Docker Containers. This feature also requires that you have enabled a RackN licensed feature.

24.10.3. Setting Up Cloud Wrapper

You must set the cloud/provider to the correct cloud. Beyond that, each cloud will have cloud specific values needed to set security, location, and instance type. In general, safe defaults have been provided where possible.

You must set Security credentials for each cloud! In many cases, you must also provide the name of the SSH user.

24.10.3.1. Required Values

The Cloud Wrapper components require at least two parameters without a default be set:

1. cloud/provider determine which cloud is to be used. 1. cloud/token enables the API

24.10.3.1.1. AWS

  • aws/access-secret
  • aws/access-key-id
  • rsa/key-user - typically ec2-user

Additional values, e.g. region, image and instance type, have safe defaults but should be reviewed.

24.10.3.1.2. Google

  • google/credential - this is a copy of contents from the JSON file Google provides
  • rsa/key-user - varies by user, make sure you have the correct default

Additional values, e.g. region, image and instance type, have safe defaults but should be reviewed.

24.10.3.1.3. Linode

  • linode/token

Additional values, e.g. region, image and instance type, have safe defaults but should be reviewed.

24.10.3.2. Optional Values

When possble, the machine on the cloud provider is given the name of the machine in Digital Rebar.

The reference terraform plan will create tags on the cloud provider based on the assigned profiles. It also creates one called “digitalrebar.” This can be handy to find or manage the machines on the cloud provider.

24.10.4. Creating / Destroying VMs

The Cloud Wrapper uses the terraform-apply tasks to create or destroy machines using a reference Terraform template defined in terraform/plan-automation. It then hands off to the ansible-join-up task to install the DRP agent.

These methods are provided as a reference. In practice, operators will likely want to use more customized Terraform plans.

The cloud-inspect task will use the Metadata API to collect additional information and set the cloud/* params where possible. These values will be removed by the cloud-cleanup task on decommission.

24.10.4.1. Create

To create a cloud backed machine:

1. Starting from a Digital Rebar machine with an active runner. 1. Run the ‘cloud-provision’ workflow.

If you have created a linode profile with the correct tokens, then this example will create and provision a machine in Linode:

drpcli machines create '{"Name":"foo",
    "Profiles":["linode"],
    "Workflow":"cloud-provision",
    "Meta":{"BaseContext":"runner"}
}'

If the machine is already created, then just the workflow change is required. For example, decommissioned machines are left in a ready to reprovision state.

Note that the machine will have several cloud/* and terraform/* parameters defined after the process completes. These are important to operation and can be used by downstream workflows.

24.10.4.2. Destroy

From a cloud backed machine:

1. Starting from a Digital Rebar machine with an active runner. 1. Run the ‘cloud-decommission’ workflow

Note that the machine will have removed the cloud/* parameters. If these are missing, the machine can be assumed to no longer have a backing cloud instance.

24.10.5. Extending the Model

This approach uses terraform/plan-automation to determine which Terraform plan to run for provisioning. The included cloud-provision-reference.tf.tmpl template has basic safe provisioning operations for all clouds defined in the cloud/provider parameter.

By default, the cloud-verify task will add this value to the provisioned machine if it has not been defined by the operator. The terraform-apply task will remove the parameter (if defined at the machine level) on destroy.

Operators who wish to override this behavior need to define terraform/plan-automation at the profile or global level. This will prevent the system from adding the reference template.

The behavior is provided to eliminate the need to clone or copy the cloud-provision workflow unless very different operations are required.

24.10.6. Future Work

At some point, the cloud-provision could include an automatic “flexiflow” operation to start another workflow.

24.10.7. Object Specific Documentation

24.10.7.1. workflows

The content package provides the following workflows.

24.10.7.1.1. cloud-decommission

Requires that operator has created a Contexts for “runner”, “terraform” and “ansible” that can run DRP Agent and Terraform then Connect via Ansible

Leaves the machines in runner Context

24.10.7.1.2. cloud-provision

Requires that operator has created a Contexts for “runner”, “terraform” and “ansible” that can run DRP Agent and Terraform then Connect via Ansible

Leaves the machines in machine Context

24.10.7.2. params

The content package provides the following params.

24.10.7.2.1. digitalocean/region

Provisioning to Region for Digital Ocean

List of regions: curl -X GET -H “Content-Type: application/json” -H “Authorization: Bearer $DO_KEY” “https://api.digitalocean.com/v2/regions” | jq .regions[].slug

24.10.7.2.2. google/region

Provisioning to Region for Google Cloud

24.10.7.2.3. aws/ami-id

Provision AWS O/S Image

24.10.7.2.4. aws/security-groups

These should alreday be defined. If omitted, will create a basic security group with SSH, HTTP and HTTPS inbound

24.10.7.2.5. azure/appId

App ID

24.10.7.2.6. cloud/public-ipv4

Address assigned by the Cloud Provider

24.10.7.2.7. azure/size

Size

24.10.7.2.8. azure/subscription_id

Subscriber ID

24.10.7.2.9. cloud/provider-icons

List of icons to use for Cloud Providers

Implemented clouds: aws, google, linode, azure, default

Expand this list as new types are added!

24.10.7.2.10. digitalocean/image

Provision Digital Ocean O/S Image

Retrieve list of images: curl -X GET -H “Content-Type: application/json” -H “Authorization: Bearer $DO_TOKEN” “https://api.digitalocean.com/v2/images” | jq .images[].slug

24.10.7.2.11. azure/password

API Password

24.10.7.2.12. google/credential

The token required by cloud provider to act aginst the API

24.10.7.2.13. linode/instance-image

Provision Linode O/S Image

24.10.7.2.14. linode/root-password

Password for Nodes

24.10.7.2.15. google/project-id

NO DEFAULT! You must supply a project name.

Provisioning to Project for Google Cloud

24.10.7.2.16. aws/instance-type

The type of resource assigned by the cloud provider

24.10.7.2.17. azure/image

Image Information for Azure including * publisher * offer * sku * version

24.10.7.2.18. cloud/instance-type

The type of resource assigned by the cloud provider

24.10.7.2.19. digitalocean/size

Provision Digital Ocean Droplet Size

Retrieve list of sizes: curl -X GET -H “curl -X GET -H “Content-Type: application/json” -H “Authorization: Bearer $DO_TOKEN” “https://api.digitalocean.com/v2/sizes” | jq .sizes[].slug

24.10.7.2.20. cloud/dmi-hypervisor

The DMI.Hypervisor value returned by Gohai

24.10.7.2.21. cloud/instance-id

The ID reference from cloud provider

24.10.7.2.22. linode/token

The token required by cloud provider to act aginst the API

24.10.7.2.23. google/zone

Provisioning to Zone for Google Cloud

24.10.7.2.24. aws/access-key-id

The ID needed to use the AWS secrete

Known types: aws

24.10.7.2.25. cloud/placement/availability-zone

The location of resource assigned by the cloud provider

24.10.7.2.26. cloud/provider

The cloud provider detected by join-up script in discovery

Implemented types: aws, google, linode, azure

Expand this list as new types are added!

24.10.7.2.27. google/boot-disk-image

Provision AWS O/S Image

24.10.7.2.28. aws/region

Provisioning to Region for AWS

24.10.7.2.29. linode/instance-type

Provision Linode allocation size

24.10.7.2.30. terraform-var/instance_id

Storage the Cloud Istance ID generated when Terraform output

24.10.7.2.31. digitalocean/token

The token required by cloud provider to act aginst the API

24.10.7.2.32. google/instance-type

The type of resource assigned by the cloud provider

24.10.7.2.33. linode/region

Provisioning to Region for Linode

24.10.7.2.34. aws/secret-key

The token required by cloud provider to act aginst the API

24.10.7.2.35. azure/region

Region

24.10.7.2.36. azure/tenant

API Tenant

24.10.7.2.37. cloud/public-hostname

Hostname assigned by the Cloud Provider

24.10.7.3. stages

The content package provides the following stages.

24.10.7.3.1. cloud-inspect

Collect information from cloud metadata API

24.10.7.4. tasks

The content package provides the following tasks.

24.10.7.4.1. cloud-cleanup

Remove collected internal API information about a cloud instance

24.10.7.4.2. cloud-inspect

Collect internal API information about a cloud instance

24.10.7.4.3. cloud-validate

Used as a validate that the right params have been set for cloud scenarios and provide operator friendly feedback in the Machine.Description

Developer Notes: * Please add synchronize with the cloud/provider enum! * Sets the terraform/plan-automation value on the machine if missing