How to Manage Go Application Secrets Using Vault

Time to read: 6 minutes

August 04, 2022
Written by
Matthew Setter
Twilion
Reviewed by
Hayden Powers
Twilion

How to Manage Go Application Secrets Using Vault

Because modern software is so complex it needs to use secrets and confidential information, such as API keys, tokens, and the older usernames and passwords for connecting to remote servers and databases.

While once it might have been seen as okay to store these alongside the code itself, these days — especially in light of the 12-factor app movement — that's no longer the case. It's considered bad security practice — with good reason — to keep any kind of secure information within your code.

Consequently, a range of approaches and tools have been developed to keep credentials out of code bases, keeping them secure and readily available to the code as and when required.

In this tutorial, you're going to learn how to manage Go application secrets with HashiCorp Vault.

If you're a PHP developer, check out the PHP version of this tutorial.

Prerequisites

To follow along with this tutorial, you will need the following:

What is Vault?

If this is your first time hearing about Vault, according to the Vault website, it is:

…an identity-based secrets and encryption management system. It allows you to secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.

It can store secrets, automate credential rotation, roll encryption keys, and provides API-driven encryption. Through a unified interface, secrets are persisted to an underlying secrets engine.

These engines can integrate with existing infrastructure such as Microsoft Azure, Google Cloud, a database such as PostgreSQL, or a queueing server such as RabbitMQ.

In this tutorial, the code will use the simplest engine, the KV Secrets Engine (kv).

This is a generic Key-Value store used to store arbitrary secrets within the configured physical storage for Vault.

Another great feature about this engine is that, when enabled (as in this tutorial), secrets can be versioned, allowing them to be rolled back, as and when required.

Start Vault

Before writing any code, to get you up and running quickly, start Vault in “Dev” Server mode, by running the following command.

vault server -dev

The Dev Server mode requires no complex setup, enables the Key/Value storage engine, and pre-generates an authentication token, which saves you a lot of time and effort.

Never use this mode in production as it is not secure.

The command will write output similar to the following to the terminal.

You may need to set the following environment variable:

    $ export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: OqoUeFDlv9PjqvJgBolLyevsj4y3gqPInNKvBubZTd0=
Root Token: hvs.2fWa1QeRWesGfjGeb2QqBYU4

Development mode should NOT be used in production installations!

From the output written to your terminal, copy the Root Token value written towards the end (e.g., hvs.2fWa1QeRWesGfjGeb2QqBYU4) and paste it in place of the placeholder <VAULT_TOKEN> in the second command below. Then, run the commands in a new terminal window.

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN=<VAULT_TOKEN>

Set up the project

Next, create a project directory, change to it, and enable dependency tracking by running the commands below.

cd
mkdir vault-go
cd vault-go
go mod init example/vault-go

Then, install the code's sole dependency (which saves a lot of time and effort interacting with Vault), HashiCorp's official Go library, by running the command below.

go get github.com/hashicorp/vault/api

Create the core application logic

Now, it's time to create the application's core code, which each successive section of the tutorial will draw on. To do that, create a new file, named main.go, in the project directory. Then, paste the code below into it.

package main

import (
    "context"
    "log"
    "os"

    vault "github.com/hashicorp/vault/api"
)

const password string = "<PASSWORD>"

func main() {
    config := vault.DefaultConfig()
    config.Address = os.Getenv("VAULT_ADDR")

    client, err := vault.NewClient(config)
    if err != nil {
        log.Fatalf("Unable to initialize a Vault client: %v", err)
    }

    client.SetToken(os.Getenv("VAULT_TOKEN"))

    secretData := map[string]interface{}{
        "password": password,
    }

    ctx := context.Background()
}

The code starts off by importing all of the required packages before declaring a constant, password, for the password that the code will store in the Key/Value engine. Replace the placeholder <PASSWORD> with a string of your choice.

Then, the main() function starts off by setting the address that the Go Vault client will use to connect to the Vault API. You stored this in the VAULT_ADDR environment variable earlier. After that, it initialises a new Vault client and prints an error, if one was returned while doing so.

If no error was returned, the authentication token to pass with the requests to the Vault API is set; retrieved from the VAULT_TOKEN environment variable. After that, it initialises the secret that will be stored in the Vault server and creates a Go context for interacting with the Vault server.

Store a secret

At this point the core of the code is ready, so it's time to add the functionality for storing secrets in the Key Value storage engine. To do that, at the bottom of the main() function add the code below.

_, err = client.KVv2("secret").Put(ctx, "my-secret-password", secretData)
if err != nil {
    log.Fatalf("Unable to write secret: %v to the vault", err)
}
log.Println("Super secret password written successfully to the vault.")

The code attempts to store the secret in secretData using the key my-secret-password. If the secret cannot be stored, the error returned is printed to the terminal. Otherwise, a confirmation message is printed out that the secret was stored successfully.

Execute the code, by running the command below and see if the secret password was successfully stored.

go run main.go

Now, run the code again. It should succeed as it did the first time. If a secret already exists its original value is not overwritten, rather, a new version of the secret is created.

Retrieve a secret

Now that a secret can be stored (and versioned) the next thing to do is to retrieve it. Paste the code below at the bottom of the main() function, after the code from the previous section.

secret, err := client.KVv2("secret").Get(ctx, "my-secret-password")
if err != nil {
    log.Fatalf(
        "Unable to read the super secret password from the vault: %v", 
        err,
    )
}

value, ok := secret.Data["password"].(string)
if !ok {
    log.Fatalf(
        "value type assertion failed: %T %#v", 
        secret.Data["password"], 
        secret.Data["password"],
    )
}

log.Printf("Super secret password [%s] was retrieved.\n", value)

The code attempts to retrieve the secret by calling the Get() function using the key "my-secret-password". It prints an error message if the secret could not be found or could not be retrieved.

If data was retrieved, the code checks if it contains the stored password. If not, then an error message is printed to the terminal. If it did contain the password, the password's value is cast to a string and printed to the terminal.

If you run the code, you should see Super secret password <YOUR PASSWORD> was retrieved. printed to the terminal, where the placeholder contains the password you set earlier.

Retrieve all versions of a secret

Let's say that you've created several versions of a secret and want to view them all. To do that, add the code below in place of the previous code, which retrieved a single version of a secret.

versions, err := client.KVv2("secret").GetVersionsAsList(ctx, "my-secret-password")
if err != nil {
    log.Fatalf(
        "Unable to retrieve all versions of the super secret password from the vault. Reason: %v", 
        err,
    )
}

for _, version := range versions {
    deleted := "Not deleted"
    if !version.DeletionTime.IsZero() {
        deleted = version.DeletionTime.Format(time.UnixDate)
    }

    secret, err := client.KVv2("secret").
        GetVersion(ctx, "my-secret-password", version.Version)
    if err != nil {
        log.Fatalf(
            "Unable to retrieve version %d of the super secret password from the vault. Reason: %v",
            err,
        )
    }
    value, ok := secret.Data["password"].(string)

    if ok {
        log.Printf(
            "Version: %d. Created at: %s. Deleted at: %s. Destroyed: %t. Value: '%s'.\n",
            version.Version,
            version.CreatedTime.Format(time.UnixDate),
            deleted,
            version.Destroyed,
            value,
       )
    }
}

The code calls the GetVersionsAsList() function to retrieve a slice of KVVersionMetadata objects, which contain the metadata for each version of a secret.

If an error was returned, it's printed to the terminal. If no error was returned it iterates over the returned KVVersionMetadata objects. For each one, it retrieves the version's value by calling GetVersion() before printing out the version number, creation time, deletion time (if it had been deleted), whether it had been destroyed or not, and its value.

If you run it, you should see output similar to the following.

Version: 1. Created at: Fri Jul 29 17:02:35 UTC 2022. Deleted at: Not deleted. Destroyed: false. Value: 'Hashi12345'.

Delete a single version of a secret

Let's say that you stored a version of a secret and wanted to delete it. To do that, paste the following code at the end of the main() function.

_, err = client.KVv2("secret").Delete(ctx, "my-secret-password")
if err != nil {
    log.Fatalf("Unable to delete the latest version of the secret from the vault. Reason: %v", err)
}
log.Println("Delete the latest version of the secret from the vault")

The code calls the Delete() function which deletes the most recent version of a secret, if it is available.

Delete all versions of a secret

The final feature to add is the ability to delete all versions of a secret. To do that, add the following code at the end of the main() function in place of the code that was added in the previous section.

err = client.KVv2("secret").DeleteMetadata(ctx, "my-secret-password")
if err != nil {
    log.Fatalf("Unable to entirely delete the super secret password from the vault. Reason: %v", err)
}
log.Println("Deleted the latest version of the super secret password from the vault")

The code calls the DeleteMetadata() function which deletes all versions and metadata of a secret, if the secret exists in the Key Value engine. If an error was returned, it's printed out. Otherwise it prints a confirmation message showing that the secret was returned to the terminal.

A parting word about security

Two things are worth bearing in mind.

  • Firstly: the code examples in this tutorial intentionally used HTTP to avoid setting up a self-signed SSL certificate and configuring a web server to use that certificate.
  • Secondly: while the application uses a token to interact with Vault, requests to the API itself are not secured. Consequently, anyone who can access the API can access your secrets

In a production application, only make requests over HTTPS, and use proper authentication and authorisation to ensure information is only available to valid users with appropriate access.

That's how to manage Go application secrets using Vault

While this hasn't been a deep dive into managing secrets with Vault, and has only covered the Key Value Engine, it's still been a good starting point for learning about Vault. Take a look at the documentation and play around with the code. I'd love to see what you build with Go and Vault.

As a parting word, a big thank you to the hashicorp-hello-vault-go repo. I based a number of the code samples in this tutorial on that project's work. Credit where credit's due.

Matthew Setter is a PHP Editor in the Twilio Voices team and (naturally) a PHP developer. He’s also the author of Deploy With Docker Compose, which shows the shortest path to deploying apps with Docker Compose. When he’s not writing PHP code, he’s editing great PHP articles here at Twilio. You can find him at msetter@twilio.com, and on Twitter, and GitHub.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.