Data transformation with code examples

The Transform Secrets Engine tutorial introduced the basic working of the Transform secrets engine using Vault CLI, API, and UI. This tutorial takes it to the next step and go through code examples to show how an application may leverage the Transform secrets engine.

Prerequisites

This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, use a x86_64 based Linux virtual machine in your preferred cloud provider.

• Docker
• Shipyard

Scenario introduction

The example application is a simple RESTful payment service that receives and stores payment data for later processing. The API has a single route that accepts an HTTP POST request, and it uses PostgreSQL for data storage.

Security requirements

The security requirements for the API are:

• Credit card details must be stored in the database in an encrypted format at rest
• It must be possible to infer the bank and type of a card number without decrypting it

The first requirement is relatively trivial to solve using Vault and the Transit Secrets Engine. Transit secrets can be used as encryption as service to encrypt the credit card details before they are written to the database.

To satisfy the second requirement, you need to be able to query the type of the card and the bank which issued it. You can get this data from the card number as not all the data in a credit card number is unique. A credit card number is composed of three parts: the Issuer Number, the Account Number and the Checksum.

Issuer Number relates to the type of the card (first digit), and the issuer's code is the information that you would like to query to satisfy the second requirement.

Account Number is the unique identifier assigned to the holder of the card.

Checksum is not a secret part of the card number; instead, it is designed for quick error checking. The checksum is generated from the card number; before processing the checksum is regenerated using the Luhn algorithm, if the given and the computed checksums differ, then the card number has been entered incorrectly.

To be able to query the card issuer you realistically have two options:

1. Partially encrypt the card number in the database
2. Store metadata for the card along with the encrypted values

To implement this requirement in code, the developers have the responsibility for managing the complexity of partially encrypting the credit card data, and the information security needs to worry about the correct implementation.

Integrate with Transform secrets engine

Vault's Transform Secrets Engine can simplify the process while still satisfying the second requirement. Transform secrets engine allows you to encrypt data while preserving formatting or to partially encrypt data based on a user-configurable formula. The benefits of this are that the information security team can centrally manage the definition for the encryption process, and the developers don't need to worry about the implementation, they can use the Transform API to encrypt the card numbers.

There are three options to interact with the Vault API:

1. Use one of the Client libraries
2. Generate your client from the OpenAPI v3 specifications
3. Manually interact with the HTTP API

This tutorial follows the third option as this demonstrates the simplicity of interacting with Vault's API. In our scenario, the application only needs to encode data and not manage the configuration for Transform; to do this, it only needs to interact with the Encode API endpoint.

Clone the code example repository

1. Clone the demo assets from the demo-vault GitHub repository to perform the steps described in this tutorial.

   $ git clone https://github.com/nicholasjackson/demo-vault
   

2. Change the working directory to demo-vault/transform.

   $ cd demo-vault/transform
   

Review the code

The transform-engine-go directory contains the code example written in Go, and the transform-engine-java directory contains a Java code example.

$ tree -d
.
├── blueprint
│   ├── docs
│   │   └── images
│   ├── files
│   └── secrets
├── transform-engine-go
│   ├── bin
│   ├── data
│   ├── handlers
│   └── vault
└── transform-engine-java
    ├── gradle
    │   └── wrapper
    └── src
        └── main
            ├── java
            │   └── payments
            │       ├── model
            │       ├── repository
            │       └── vault
            └── resources

21 directories

To send a credit card number to Vault for encryption through the Transform secrets engine, the application must:

1. Create a request payload
2. Invoke the Vault API
3. Parse the response

Create a request payload

The first thing that needs to be done is to construct a byte array with a JSON formatted string for the payload. In both Go and Java common libraries easily handle this function.

• Go: See line 56-57 of transform-engine-go/vault/vault.go.
• Java: See line 43-48 of transform-engine-java/src/main/java/payments/vault/VaultClient.java.

Invoke the Vault API

Once the request body is set to the JSON data payload, you can make the request and retrieve the response from the Vault server.

• Go: See line 60-72 of transform-engine-go/vault/vault.go.
• Java: See line 51-62 of transform-engine-java/src/main/java/payments/vault/VaultClient.java.

Parse the response

To read the JSON response from the Vault API, you can parse the body from the HTTP client into a simple structure. The encoded data can now be passed to another function which will store it in the database.

• Go: See line 75-79 of transform-engine-go/vault/vault.go.
• Java: See line 65-67 of transform-engine-java/src/main/java/payments/vault/VaultClient.java.

package vault

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
)

type Client struct {
    token string
    uri   string
}

type TokenRequest struct {
    Value string `json:"value"`
}

type TokenResponse struct {
   Data TokenReponseData `json:"data"`
}

type TokenReponseData struct {
   EncodedValue string `json:"encoded_value"`
}

// NewClient creates a new Vault client
func NewClient(token, serverURI string) *Client {
   return &Client{token, serverURI}
}

// IsOK returns true if Vault is unsealed and can accept requests
func (c *Client) IsOK() bool {
   url := fmt.Sprintf("%s/v1/sys/health", c.uri)

   r, _ := http.NewRequest(http.MethodGet, url, nil)
   r.Header.Add("X-Vault-Token", c.token)

   resp, err := http.DefaultClient.Do(r)
   if err != nil {
      return false
   }

   defer resp.Body.Close()

   if resp.StatusCode != http.StatusOK {
      return false
   }

   return true
}

// TokenizeCCNumber uses the Vault API to tokenize the given string
func (c *Client) TokenizeCCNumber(cc string) (string, error) {
   // create the JSON request as a byte array
   req := TokenRequest{Value: cc}
   data, _ := json.Marshal(req)

   // call the api
   url := fmt.Sprintf("%s/v1/transform/encode/payments", c.uri)
   r, _ := http.NewRequest(http.MethodPost, url, bytes.NewReader(data))
   r.Header.Add("X-Vault-Token", c.token)

   resp, err := http.DefaultClient.Do(r)
   if err != nil {
      return "", err
   }
   defer resp.Body.Close()

   if resp.StatusCode != http.StatusOK {
      return "", fmt.Errorf("Vault returned reponse code %d, expected status code 200", resp.StatusCode)
   }

   // process the response
   tr := &TokenResponse{}
   err = json.NewDecoder(resp.Body).Decode(tr)
   if err != nil {
      return "", err
   }

   return tr.Data.EncodedValue, nil
}

package payments.vault;

import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.io.OutputStream;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;

public class VaultClient {
  private String token;
  private String serverURI;

  Logger logger = LoggerFactory.getLogger(VaultClient.class);

  public VaultClient(String token, String serverURI) {
    this.token = token;
    this.serverURI = serverURI;
  }

  public Boolean IsOK() throws IOException {
    URL url = new URL(this.serverURI+ "/v1/sys/health");
    HttpURLConnection con = (HttpURLConnection)url.openConnection();
    con.setRequestMethod("GET");
    con.setRequestProperty("X-Vault-Token", this.token);

    int status = con.getResponseCode();
    if (status != 200) {
      return false;
    }

    return true;
  }


  public String TokenizeCCNumber(String cardNumber) throws IOException, JsonProcessingException {
    // create the request
    TokenRequest req = new TokenRequest(cardNumber);

    // convert the POJO to a byte array
    ObjectMapper mapper = new ObjectMapper();
    mapper.enable(SerializationFeature.INDENT_OUTPUT);
    byte[] byteRequest = mapper.writeValueAsBytes(req);

    // make a call to vault to process the request
    URL url = new URL(this.serverURI+ "/v1/transform/encode/payments");
    HttpURLConnection con = (HttpURLConnection)url.openConnection();
    con.setDoOutput(true);
      con.setRequestMethod("POST");
      con.setRequestProperty("Content-Type", "application/json; utf-8");
    con.setRequestProperty("Accept", "application/json");
    con.setRequestProperty("X-Vault-Token", this.token);

    // write the body
    try(OutputStream os = con.getOutputStream()) {
      os.write(byteRequest, 0, byteRequest.length);
    }

    // read the response
    TokenResponse resp = new ObjectMapper()
      .readerFor(TokenResponse.class)
      .readValue(con.getInputStream());

    return resp.getTokenResponseData().getEncodedValue();
  }
}

Start the sample application

The demo uses Shipyard to start a Vault server and example application in Docker on your local machine.

Run the demo.

$ shipyard run ./blueprint

The output may look like:

Running configuration from:  ./blueprint

[INFO]  Creating Network: ref=local
[INFO]  Creating Documentation: ref=docs
[INFO]  Creating Container: ref=payments_java
[INFO]  Creating Container: ref=payments_go
[INFO]  Creating Container: ref=vault
[INFO]  Creating Container: ref=postgres

The Vault server, Postgres server, and example applications are also accessible from your terminal at the following locations:

Validation

1. You can verify that the Docker containers are running.

   $ docker ps -a
   
   CONTAINER ID   IMAGE                                          ...snip...   NAMES
   c60a35ee6601   registry.shipyard.run/terminal-server:v0.1.0   ...snip...   terminal.docs.shipyard.run
   b210301ddb17   postgres:11.6                                  ...snip...   postgres.container.shipyard.run
   44cc6051c6ac   nicholasjackson/transform-demo:java            ...snip...   payments-java.container.shipyard.run
   cc92ad0c2d26   registry.shipyard.run/docs:v0.1.0              ...snip...   docs.docs.shipyard.run
   2e37ff75de45   hashicorp/vault-enterprise:1.4.0-rc1_ent       ...snip...   vault.container.shipyard.run
   5af3a69ca41b   nicholasjackson/transform-demo:go              ...snip...   payments-go.container.shipyard.run
   

2. Set the VAULT_ADDR environment variable value to http://localhost:8200.

   $ export VAULT_ADDR="http://localhost:8200"
   

3. Check and see if the Transform secrets engine is configured.

   $ vault secrets list
   

4. If the output does NOT list transform/, execute the /blueprint/files/setup_transform.sh script.

   $ ./blueprint/files/setup_transform.sh
   
   Success! Enabled the transform secrets engine at: transform/
   Success! Data written to: transform/role/payments
   Success! Data written to: transform/transformation/ccn-fpe
   Success! Data written to: transform/template/ccn
   Success! Data written to: transform/alphabet/numerics
   

   This sets up the Transform secrets engine to take the credit card number and encrypt it based on the transformation rule defined by the cnn template.

Test the application

Both the Java and the Go code are running, so you can use curl to test either application.

$ curl --header "content-type: application/json" \
    --data '{"card_number": "1234-1234-1234-1234"}' \
    localhost:9091

$ curl --header "content-type: application/json" \
    --data '{"card_number": "1234-1234-1234-1234"}' \
    localhost:9092

Output:

{"transaction_id": 1}

You can send multiple requests.

To see the encrypted value for this transaction you can query the orders table on the database.

$ PGPASSWORD=password psql -h localhost -p 5432 -U root -d payments -c 'SELECT * from orders;'

 id |     card_number     |         created_at         |         updated_at         | deleted_at
----+---------------------+----------------------------+----------------------------+------------
  1 | 1234-1219-2537-6134 | 2021-12-22 00:18:28.566104 | 2021-12-22 00:18:28.566155 |
(1 rows)

The database contains the encrypted card number.

Real-world impact of partially encrypting data

You may wonder how it reduces the security of the encrypted card number by only encrypting the account number and CV2 data. In real terms, it probably does not make a difference.

A number containing 16 digits has a possibility of 16^16 combinations, including the CV2 number. This roughly equates to 10 quintillion different permutations.

If you only store 10 digits of the card number plus the CV2, this is 10^13, or about 10 trillion combinations. Since the first 6 digits of a card number are the issuer and card type, there are not 1 million different issuers. Let's say there are 10,000, storing the full 16 digits would give you roughly 100 quadrillion combinations. In both cases, we need to remove the checksum, so we get 10 quadrillion combinations if you encrypt the account number and 1 trillion if you do not.

Not encrypting the issuer means someone can make fewer guesses to determine the number; however, the attacker still need to make 1 trillion guesses. Assuming someone managed to obtain your database containing partially encrypted card numbers. If you had an average API request time of 100 milliseconds to accept or reject a payment, it would take about 190258 years for someone to brute force a payment. Even if the attacker was running parallel attacks, the odds are stacked heavily against them.

Clean up

1. When you are done exploring, stop and remove the Docker containers.

   $ shipyard destroy
   

2. Unset the VAULT_ADDR environment variable.

   $ unset VAULT_ADDR
   

Summary

In this tutorial, you have learned how the Transform secrets engine can partially encrypt credit card numbers at rest while preserving the formatting and ability to query the card issuer. This example only covers one of the possibilities for the Transform secrets engine.

Help and reference

• Vault Client Libraries
• Transform Secrets Engine
• Tokenize Data with Transform Secrets Engine
• Transform Secrets Engine (API)
• Transform Secrets Engine
• Encrypting Data while Preserving Formatting with the Vault Enterprise Transform Secrets Engine