Build custom Vault plugins

All Vault auth methods and secrets engines are considered plugins. This simple concept allows both built-in and external plugins to be treated like Legos, and enabled at multiple paths.

This tutorial walks through the basic steps to build, register, and enable external plugins.

Plugins utilize the plugin system to enable third-party secrets engines and auth methods.

It is worth noting that even though database secrets engines operate under the same underlying plugin mechanism, they are slightly different in design than plugins demonstrated in this tutorial. The database secrets engine manages multiple plugins under the same backend mount point, whereas plugins are key-value backends that function as either secret or auth methods.

Tutorial setup

For the demonstration, this tutorial walks through the mock plugins available in the hashicorp-education/learn-vault-plugins GitHub repository.

Clone the hashicorp-education/learn-vault-plugins repository.

$ git clone https://github.com/hashicorp-education/learn-vault-plugins

Or download the repository.

Deploy a custom secrets plugin

There are vault-plugin-auth-mock and vault-plugin-secrets-mock directories under learn-vault-plugins. First, walk through the vault-plugin-secrets-mock plugin to learn the workflow.

Compile vault-plugin-secrets-mock

1. Change the working directory to where the mock plugin is located.

   $ cd learn-vault-plugins/vault-plugin-secrets-mock
   

2. Compile the plugin.

   $ go build -o vault/plugins/vault-plugin-secrets-mock cmd/vault-plugin-secrets-mock/main.go
   

   This compiles the vault-plugin-secrets-mock plugin and outputs the generated plugin in the vault/plugins/vault-plugin-secrets-mock folder.

   $ tree
   .
   ├── Makefile
   ├── README.md
   ├── backend.go
   ├── cmd
   │   └── vault-plugin-secrets-mock
   │       └── main.go
   ├── go.mod
   ├── go.sum
   └── vault
      └── plugins
         └── vault-plugin-secrets-mock
   

Start Vault

Start a Vault server in -dev mode for demonstration. Also, set the -dev-plugin-dir to ./vault/plugins which is where the vault-plugin-secrets-mock plugin is generated.

$ vault server -dev -dev-root-token-id=root -dev-plugin-dir=./vault/plugins

==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.14.7
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: false, enabled: false
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.5.5
             Version Sha: 05fe30b1eeadd105312c529c495f107dc54b40c9

2020-10-26T13:37:16.280-0700 [WARN]  received Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol. attempting as db plugin, attempting as auth/secret plugin
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

    $ export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: 9fFg4N48+onMTculzevyvqXmEe5dcPcgVouSsbT5+gI=
Root Token: root

The following dev plugins are registered in the catalog:
    - vault-plugin-secrets-mock

Development mode should NOT be used in production installations!

==> Vault server started! Log data will stream in below:
## ...snip...

The output includes the following message:

The following dev plugins are registered in the catalog:
    - vault-plugin-secrets-mock

Test the vault-plugin-secrets-mock plugin

The name of the plugin is vault-plugin-secrets-mock and it is a custom secrets engine.

1. Open another terminal session.

2. Set the VAULT_ADDR environmental variable.

   $ export VAULT_ADDR="http://127.0.0.1:8200"
   

3. Login as root.

   $ vault login root
   

4. Enable the mock plugin.

   $ vault secrets enable -path=mock-secrets vault-plugin-secrets-mock
   Success! Enabled the vault-plugin-secrets-mock secrets engine at: mock-secrets/
   

5. Check that the vault-plugin-secrets-mock mock secrets engine is enabled at mock-secrets.

   $ vault secrets list
   
   Path             Type                         Accessor                              Description
   ----             ----                         --------                              -----------
   cubbyhole/       cubbyhole                    cubbyhole_a68ba238                    per-token private secret storage
   identity/        identity                     identity_56891537                     identity store
   mock-secrets/    vault-plugin-secrets-mock    vault-plugin-secrets-mock_beb96f79    n/a
   ## ...snip...
   

   Each plugin responds to read, write, list, and delete as its own behavior. Let's test the vault-plugin-secrets-mock secrets engine.

6. Write some test data in key-value format.

   $ vault write mock-secrets/test message="Hello World"
   Success! Data written to: vault-plugin-secrets-mock/test
   

7. Read the data.

   $ vault read mock-secrets/test
   Key        Value
   ---        -----
   message    Hello World
   

8. You can disable the vault-plugin-secrets-mock secrets engine just like any other secrets engine.

   $ vault secrets disable mock-secrets
   Success! Disabled the secrets engine (if it existed) at: vault-plugin-secrets-mock/
   

9. Enter Ctrl+C in the other terminal session to terminate the running Vault dev server.

Deploy a custom auth plugin

In this section, you will walk through the mock auth plugin, vault-plugin-auth-mock.

Compile vault-plugin-auth-mock

1. Change the working directory to where the mock auth plugin is located.

   $ cd ../vault-plugin-auth-mock
   

2. Compile the plugin.

   $ go build -o vault/plugins/vault-plugin-auth-mock cmd/vault-plugin-auth-mock/main.go
   

   This compiles the vault-plugin-auth-mock plugin and outputs the generated plugin in the vault/plugins/vault-plugin-auth-mock folder.

   $ tree
    .
    ├── Makefile
    ├── README.md
    ├── backend.go
    ├── cmd
    │   └── vault-plugin-auth-mock
    │       └── main.go
    ├── go.mod
    ├── go.sum
    └── vault
        └── plugins
            └── vault-plugin-auth-mock
   

Start Vault

Start a Vault server in -dev mode for demonstration. Also, set the -dev-plugin-dir to ./vault/plugins which is where the vault-plugin-auth-mock plugin is generated.

$ vault server -dev -dev-root-token-id=root -dev-plugin-dir=./vault/plugins

The output includes the following message:

The following dev plugins are registered in the catalog:
    - vault-plugin-auth-mock

Test the vault-plugin-auth-mock plugin

The name of the plugin is vault-plugin-auth-mock and it is a custom auth method.

Set the VAULT_ADDR environmental variable.

$ export VAULT_ADDR="http://127.0.0.1:8200"

Login as root.

$ vault login root

Enable the mock auth plugin.

$ vault auth enable -path=mock-auth vault-plugin-auth-mock
Success! Enabled the vault-plugin-secrets-mock secrets engine at: mock-secrets/

Check that the vault-plugin-auth-mock auth method is enabled at mock-auth.

$ vault auth list

Path          Type                      Accessor                                Description
----          ----                      --------                                -----------
mock-auth/    vault-plugin-auth-mock    auth_vault-plugin-auth-mock_02817f20    n/a
token/        token                     auth_token_be2271f9                     token based credentials

Test the vault-plugin-auth-mock plugin

Create a new user, john with password, password.

$ vault write auth/mock-auth/user/john password=password
Success! Data written to: auth/mock-auth/user/john

List to see that a user, john was created.

$ vault list auth/mock-auth/users

Keys
----
john

Now, authenticate with Vault as john.

$ vault write auth/mock-auth/login user=john password=password

Key                  Value
---                  -----
token                s.VOLpTp2wrxhSxAmY6jcvWlgX
token_accessor       TOP4LX2WsaNrpJX8d3i6Swt0
token_duration       30s
token_renewable      true
token_policies       ["default" "my-policy" "other-policy"]
identity_policies    []
policies             ["default" "my-policy" "other-policy"]
token_meta_user      john

You can disable the vault-plugin-auth-mock auth method just like any other auth methods.

$ vault auth disable mock-auth
Success! Disabled the auth method (if it existed) at: mock-auth/

Enter Ctrl+C to terminate the dev server that is running.

Setup Vault

The plugin directory is a configuration option of Vault, and can be specified in the configuration file. This setting specifies a directory that all plugin binaries must live. A plugin can not be added to Vault unless it exists in the plugin directory. There is no default for this configuration option, and if it is not set plugins can not be added to Vault.

Set plugin_directory to the desired path in the Vault configuration file. The path should exist and have proper lockdown on access permissions.

Example:

plugin_directory = "/etc/vault/vault_plugins"  # path of plugin binaries

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_cert_file = "/path/to/fullchain.pem"
  tls_key_file  = "/path/to/privkey.pem"
}

# ...

If the Vault server is already running, you need to tell it to reload its configuration by sending SIGHUP. If you stop and start the Vault server, you need to unseal it again.

Register in plugin catalog

Calculate the SHA256 sum of the compiled plugin binary.

$ SHA256=$(sha256sum /etc/vault/vault_plugins/vault-plugin-secrets-mock | cut -d ' ' -f1)

You can now view the calculated value.

$ echo $SHA256
25526a18212b3bd85c4df7530bc2f81b20fe090530be6831ffd83c0a6966429b

Use the value to register the plugin using the Vault operator vault plugin.

$ vault plugin register -sha256=$SHA256 secret vault-plugin-secrets-mock
Success! Registered plugin: vault-plugin-secrets-mock

Enable plugin

Enabling the plugin varies depending on if it's a secrets engine or an auth method.

If it's a secrets engine:

$ vault secrets enable -path=<mount_path> <plugin_name>

If it's an auth method:

$ vault auth enable -path=<mount_path> <plugin_name>

Help and reference

• Custom Plugin Backends
• Plugin System
• API: plugins-catalog & plugins-reload-backend