Migrate your Consul datacenter to HCP Consul Dedicated

With the release of HashiCorp Cloud Platform (HCP) Consul, it is now possible to delegate the management of your Consul servers to HashiCorp's SREs, allowing your team to focus on application development and deployment. However, in some cases you might already have an existing Consul datacenter deployed.

In this tutorial, you will migrate your existing datacenter's KVs and ACLs to a new HCP Consul Dedicated cluster.

The steps you'll perform to achieve the migration are the following.

1. Check Consul version compatibility
2. Take a backup of the data in both your datacenters
3. Build the migration tool
4. Export the data from the source datacenter
5. Import the data into your HCP Consul Dedicated cluster
6. Verify data is imported correctly
7. Migrate client agents

Prerequisites

• A Consul datacenter (1.4.0+) that will be the source of your data and represents the datacenter you want to migrate to HCP.

• An HCP Consul Dedicated cluster that will be the destination of your data and will be your new datacenter. If you don't have an existing HCP Consul Dedicated cluster deployed, you can deploy one with the Deploy HCP Consul Dedicated tutorial.

• The consul-migrate tool that helps you export and import ACLs and namespaces.
• Golang 1.15+ installed and configured to build the consul-migrate tool.

Notes on older Consul versions

The tutorial and the tools used for the migration assume that the Consul version you are running in the source datacenter you want to migrate is 1.4.0 or higher. In previous versions of Consul, the ACL system was radically different and the configuration is not directly compatible with the Consul version running on HCP Consul.

For this reason, if you are running a Consul version older than 1.4.0 in your source datacenter, you must first upgrade your Consul version and ACL system.

1. Upgrade Consul to 1.4.x or later
2. Migrate Legacy ACL Tokens

Take a snapshot of your existing data

The steps provided by this tutorial will alter the configuration and the data of your HCP Consul Dedicated cluster.

To make sure you can recover from any possible outage and that you can rollback to the initial state, it is recommended that you take a snapshot of both your source datacenter and HCP Consul Dedicated cluster.

1. Take a snapshot of your existing datacenter
2. Take a snapshot of your HCP Consul Dedicated cluster in the HCP portal

Build the ACL migration tool

To migrate your ACL data from your source datacenter, you will use the consul-migrate tool. It permits you to backup and restore your ACL configuration from one Consul datacenter into another, while keeping the tokens compatible.

First, clone the repository to your local machine.

$ git clone https://github.com/mkeeler/consul-migrate

Change directory to the cloned folder.

$ cd consul-migrate

Build the tool from source.

$ go build ./cmd/consul-migrate

If the build succeeds, you will find the binary in the working directory under the name of consul-migrate.

Verify the build.

$ ./consul-migrate --help

Usage: consul-migrate [--version] [--help] <command> [<args>]

Available commands are:
    export    Export Consul data
    import    Import Consul data

Once you create the binary you can either copy it to a Consul agent node in your datacenter, or use it on an independent node that has network access to your source datacenter.

Export the KV and ACL data

This section will guide you through the steps necessary to export your KV store content, ACLs, and namespace configuration (for Consul Enterprise).

Setup your environment

To communicate with Consul, you must configure your environment to communicate with your source datacenter.

If you are using a non-local deployment, you can configure a local Consul binary to interact with the deployment. Set the CONSUL_HTTP_ADDR variable on your local machine or jump host to the IP address of a server.

$ export CONSUL_HTTP_ADDR=<consul_server_ip>:<consul_server_port>

Note, this jump host will need to use an ACL token to access Consul data. You can export the token with the CONSUL_HTTP_TOKEN variable.

Additionally, if you have TLS encryption configured, you will need to use valid certificates.

Export Consul KV data

Consul provides a CLI command to export the KV store data into a JSON file.

$ consul kv export > kv_export.json

This command produces a file, named kv_export.json containing the KV content.

Export Consul ACL and namespace data

Use the consul-migrate tool to export your ACL and namespaces data.

$ consul-migrate export -output acl_export.json

[INFO]  starting data export
[INFO]  data written to file: file=acl_export.json

You can verify the ACLs and namespaces data has been exported to the file.

$ cat acl_export.json

{
  "enterprise": true,
  "namespaces": {
    "default": {
      "definition": {
        "Name": "default",
        "Description": "Builtin Default Namespace",
        "CreateIndex": 6,
        "ModifyIndex": 6
      },
      "acl_policies": {
        "18cbb4b1-6455-9fdf-511d-66372b84ddbe": {
          "ID": "18cbb4b1-6455-9fdf-511d-66372b84ddbe",
          "Name": "acl-policy-server-node",
          "Description": "Policy for Server nodes",
          "Rules": "## consul-server-one-policy.hcl\nnode_prefix \"server-\" {\n  policy = \"write\"\n}",
          "Datacenters": null,
          "Hash": "uNtCCCTGCj0wXCDivi8FuvUaQf3QRkcMHbuILquhSMk=",
          "CreateIndex": 20,
          "ModifyIndex": 20,
          "Namespace": "default"
        },
        "ac5a5ccc-8634-2794-1778-373e828b3eea": {
          "ID": "ac5a5ccc-8634-2794-1778-373e828b3eea",
          "Name": "acl-policy-dns",
          "Description": "Policy for DNS endpoints",
          "Rules": "## dns-request-policy.hcl\nnode_prefix \"\" {\n  policy = \"read\"\n}\nservice_prefix \"\" {\n  policy = \"read\"\n}\n# only needed if using prepared queries\nquery_prefix \"\" {\n  policy = \"read\"\n}",
          "Datacenters": null,
          "Hash": "reh08PMSQaibxYQ53q75fA9uWXyc9QdlfZDMrsL3jmU=",
          "CreateIndex": 19,
          "ModifyIndex": 19,
          "Namespace": "default"
        }
      },
      "acl_tokens": {
        "00000000-0000-0000-0000-000000000002": {
          "CreateIndex": 5,
          "ModifyIndex": 5,
          "AccessorID": "00000000-0000-0000-0000-000000000002",
          "SecretID": "anonymous",
          "Description": "Anonymous Token",
          "Local": false,
          "CreateTime": "2021-04-15T17:30:27.320483034Z",
          "Hash": "B023BjtZpWba+/EeQuv+0K2yc+cT+xPszCr8gDqouuc=",
          "Namespace": "default"
        },
        "0f0bdcc0-93a8-ee5f-af03-b334a8a00e15": {
          "CreateIndex": 21,
          "ModifyIndex": 21,
          "AccessorID": "0f0bdcc0-93a8-ee5f-af03-b334a8a00e15",
          "SecretID": "64ff96ab-5518-db53-08e5-edcb2336384f",
          "Description": "DNS - Default token",
          "Policies": [
            {
              "ID": "ac5a5ccc-8634-2794-1778-373e828b3eea",
              "Name": "acl-policy-dns"
            }
          ],
          "Local": false,
          "CreateTime": "2021-04-15T17:30:31.370845849Z",
          "Hash": "gCfGNbNjl7pW+2TkXp0xElbLtCBC1kChAZSP3GHMRKw=",
          "Namespace": "default"
        },
        "0fb34527-ab75-ce61-ec92-fd042b0e77f5": {
          "CreateIndex": 22,
          "ModifyIndex": 22,
          "AccessorID": "0fb34527-ab75-ce61-ec92-fd042b0e77f5",
          "SecretID": "01b18b5f-da51-aa34-6d6a-3fe58d3d5047",
          "Description": "server-1 agent token",
          "Policies": [
            {
              "ID": "18cbb4b1-6455-9fdf-511d-66372b84ddbe",
              "Name": "acl-policy-server-node"
            }
          ],
          "Local": false,
          "CreateTime": "2021-04-15T17:30:31.592941157Z",
          "Hash": "1C0NLd7PcAmsvv0rmJn6kIIsIx702SC8fXqQOJByMnY=",
          "Namespace": "default"
        },
        "347e18b9-1690-dc7a-1bc4-95cdfdc30e65": {
          "CreateIndex": 23,
          "ModifyIndex": 23,
          "AccessorID": "347e18b9-1690-dc7a-1bc4-95cdfdc30e65",
          "SecretID": "5a1d9d0e-01af-3de0-2b08-fd09c9a8a365",
          "Description": "server-2 agent token",
          "Policies": [
            {
              "ID": "18cbb4b1-6455-9fdf-511d-66372b84ddbe",
              "Name": "acl-policy-server-node"
            }
          ],
          "Local": false,
          "CreateTime": "2021-04-15T17:30:32.179037143Z",
          "Hash": "ThJBqybR+WsPKoEnWEhFjuaq73gjyCDgrZXLJFHbs+o=",
          "Namespace": "default"
        },
        "3d1a5ea3-1f5e-4185-40bf-f4b28d3ff611": {
          "CreateIndex": 24,
          "ModifyIndex": 24,
          "AccessorID": "3d1a5ea3-1f5e-4185-40bf-f4b28d3ff611",
          "SecretID": "1636718b-086b-387c-56c8-7661add2be13",
          "Description": "server-3 agent token",
          "Policies": [
            {
              "ID": "18cbb4b1-6455-9fdf-511d-66372b84ddbe",
              "Name": "acl-policy-server-node"
            }
          ],
          "Local": false,
          "CreateTime": "2021-04-15T17:30:32.771348435Z",
          "Hash": "yHcGIcpo+JGetxeUoT7V0YDDxbKrrdfpkb1+US8qIag=",
          "Namespace": "default"
        },
        "6bd15e91-3c8f-2db5-e818-24597ebeeef3": {
          "CreateIndex": 18,
          "ModifyIndex": 18,
          "AccessorID": "6bd15e91-3c8f-2db5-e818-24597ebeeef3",
          "SecretID": "b9fcdd1b-43d7-e8df-9b82-b45d0ab3687f",
          "Description": "Bootstrap Token (Global Management)",
          "Policies": [
            {
              "ID": "00000000-0000-0000-0000-000000000001",
              "Name": "global-management"
            }
          ],
          "Local": false,
          "CreateTime": "2021-04-15T17:30:30.780144686Z",
          "Hash": "b8qoe+c/T/IVHrL2bjjNHj3wgCy+LD5WMdwtLRW1740=",
          "Namespace": "default"
        }
      }
    }
  }
}

Configure access to your HCP Consul Dedicated cluster

To import the data into your HCP Consul Dedicated cluster, you will need a node with network access over the HCP Consul Dedicated address.

There are several ways to achieve this result.

1. (Recommended) Connect an Amazon Transit Gateway to your HashiCorp Virtual Network
2. Connect a Consul Client to HCP Consul Dedicated

Move the data to the import node

Once you have decided which approach to use to connect to your HCP Consul Dedicated cluster, you must migrate the data to the node you configured to access the cluster.

Install the migration tool

To import the ACL data, you will use the consul-migrate tool you built earlier during this tutorial.

Copy the consul-migrate tool to the node you configured to access the cluster.

Setup your environment

In the previous section, before exporting the data, you configured your node to be able to reach your Consul datacenter. In this section, before importing the data, you will need to perform the same operation using the HCP Consul Dedicated cluster info. Specifically, to be able to connect to your HCP Consul Dedicated cluster, you will need the following:

1. The datacenter URL
2. A valid token for the import operation

You can retrieve this information from the Overview tab of your HCP Consul Dedicated cluster page.

First, copy the Private link to get the URL of you datacenter. Then click on Generate token to get a token to use for the import.

You can use the datacenter URL and ACL token values to configure your environment variables.

export CONSUL_HTTP_ADDR=https://learn-hcp-consul-cluster.private.consul.11eaa743-0419-6632-8043-0242ac110006.aws.hashicorp.cloud
export CONSUL_HTTP_TOKEN=574fa16d-d993-b701-93e3-14e079a3d465

You can test your configuration using the Consul binary to check the content of ACLs and KV store.

$ consul kv get --recurse /

In a new cluster, the consul acl token list command should return an empty response.

$ consul acl token list

AccessorID:       218f48e3-0b42-05cb-18fc-a2f481cd4b53
Namespace:        default
Description:
Local:            false
Create Time:      2021-04-15 15:49:03.274698123 +0000 UTC
Legacy:           false
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

AccessorID:       00000000-0000-0000-0000-000000000002
Namespace:        default
Description:      Anonymous Token
Local:            false
Create Time:      2021-04-14 16:21:07.089208225 +0000 UTC
Legacy:           false

Import the data to HCP Consul Dedicated

Once you confirm that your node is able to communicate with your HCP Consul Dedicated cluster, restore your exported data to HCP Consul Dedicated.

Import KV data

First, import the KV data using the consul kv command.

$ consul kv import @kv_export.json

Imported: consul/data/key001
Imported: consul/data/key002
Imported: consul/data/key003
Imported: consul/data/key004
Imported: consul/data/key005
...

At the end of the process, you will be able to confirm the KV data is now imported using the consul kv command.

$ consul kv get --recurse /

consul/data/key001:value001
consul/data/key002:value002
consul/data/key003:value003
consul/data/key004:value004
consul/data/key005:value005

Run the migration tool to import the data

Use the consul-migrate tool to import the ACLs and namespaces information.

The tool will perform the import, and maintain each token's accessor and secrets ID across the two datacenters. This allows you to plan your migration without the need to update tokens in the configuration, or to redistribute tokens to the clients.

$ consul-migrate import -input acl_export.json

[INFO]  created Namespace: ns=default
[INFO]  created ACL Policy: ns=default id=ba987117-9b71-8865-5b43-31ba5ad32c3b from=18cbb4b1-6455-9fdf-511d-66372b84ddbe
[INFO]  created ACL Policy: ns=default id=77cbc228-345d-98d7-816d-d7143f469a22 from=ac5a5ccc-8634-2794-1778-373e828b3eea
[INFO]  updated anonymous ACL Token: ns=default accessor-id=00000000-0000-0000-0000-000000000002
[INFO]  created ACL Token: ns=default accessor-id=0f0bdcc0-93a8-ee5f-af03-b334a8a00e15
[INFO]  created ACL Token: ns=default accessor-id=0fb34527-ab75-ce61-ec92-fd042b0e77f5
[INFO]  created ACL Token: ns=default accessor-id=347e18b9-1690-dc7a-1bc4-95cdfdc30e65
[INFO]  created ACL Token: ns=default accessor-id=3d1a5ea3-1f5e-4185-40bf-f4b28d3ff611
[INFO]  created ACL Token: ns=default accessor-id=6bd15e91-3c8f-2db5-e818-24597ebeeef3
[INFO]  successfully imported data

Move clients over to HCP Consul Dedicated

To migrate your Consul clients to HCP Consul Dedicated, you will need to make sure they can reach the HCP Consul Dedicated servers. You can achieve that by peering your VPC network with the HashiCorp Virtual Network (HVN) you created for the cluster, or by connecting an Amazon Transit Gateway to your HashiCorp Virtual Network.

Once you enabled connectivity, and the data is migrated, you will be able to migrate over your clients using the same tokens you already have in your configuration.

To enable the clients to communicate with your HCP Consul Dedicated cluster, you will need to change the following parameters in your Consul client agent configuration.

1. Gossip encryption key
2. CA certificate for the HCP Consul Dedicated cluster
3. Join address information

To simplify the configuration operations you can download a pre-compiled configuration package directly from the Overview page of your HCP Consul Dedicated cluster by clicking on the Download client config button. The download is a zip file containing two files.

• ca.pem - contains the CA certificate for your HCP Consul Dedicated cluster
• client_config.json - contains a working configuration for a client that needs to join the cluster.

{
  "acl": {
    "enabled": true,
    "down_policy": "async-cache",
    "default_policy": "deny"
  },
  "ca_file": "./ca.pem",
  "verify_outgoing": true,
  "datacenter": "dc1",
  "encrypt": "XLgwI8IKQYzHa65iRT4fmw==",
  "encrypt_verify_incoming": true,
  "encrypt_verify_outgoing": true,
  "server": false,
  "log_level": "INFO",
  "ui": true,
  "retry_join": [
    "learn-hcp-consul-cluster.private.consul.11eaa743-0419-6632-8043-0242ac110006.aws.hashicorp.cloud"
  ],
  "auto_encrypt": {
    "tls": true
  }
}

With those files, you can configure your clients to connect to the new servers.

You can either edit the configuration file to remove unwanted options, or to add specific client parameters that are not present in the example to your existing configuration.

Consul catalog content

During the client migration process you must re-register the services in your new Consul datacenter. If you have your services defined in a configuration file inside your client configuration directory, they will be automatically registered to the new datacenter when the clients join.

Client downtime notes

When you plan a client migration, you should account for downtime in your applications. If your infrastructure permits it, you might want to spin up new instances of the clients, and their relative services, directly into the HCP Consul cluster and, once the client fleet is fully replicated, redirect the traffic into the new datacenter.

Network connectivity

This tutorial assumes you have your clients and services already on AWS either running inside VMs or in a EKS cluster. If your clients are running on a different platform, you can still use the steps listed in this tutorial to migrate ACLs, namespace, and KV store data into your HCP Consul Dedicated cluster, and then deploy your applications in AWS.

Next steps

In this tutorial, you learned how to migrate your existing Consul datacenter ACL, namespace, and KV data from a self hosted Consul datacenter into a HCP Consul Dedicated cluster.

If you want to learn more about how to deploy your clients in HCP Consul Dedicated, you can review our tutorials on how to deploy clients on a virtual machine or on Elastic Kubernetes Service (EKS).

If you encounter any issues, please contact the HCP team at support.hashicorp.com.