Grasshopper ----------> Master ( D 2 ): July 2021

hashicorp.com: Vault & Kubernetes: Better Together

Using Static secrets

(can apply TTL)

Using Dynamic secrets

(Secrets are also easy to rotate and revoke; if an employee leaves your organization, you can easily and securely revoke their access.)

Using transit encryption in Kubernetes

(Vault provides “encryption as a service,” encrypting data in transit (with TLS) and at rest (using AES 256-bit CBC encryption).)

Key k8s components

MutatingAdmissionWebhook

An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.

Using Admission Controllers

Service Account

A service account provides an identity for processes that run in a Pod.

Configure Service Accounts for Pods

How it works

The Service Account assigns an identity to a pod, which is used to grant access to secrets in Vault whereas the webhook is used to inject an init container into a Pod that mounts the Secret from Vault to a temporary volume.

Example:

Application Pod

Init Containers:

vault-agent-init:

Container ID: docker://eb9a90c1b4105102e6180bd622e954e90fc34259da37b167c9e1ea5718152db1

Image: vault:1.7.0

Image ID: docker-pullable://vault@sha256:635cf1c3f9b10fe03aad375f94cc61f63d74a189662165285a8bf1c189ea04b8

Port: <none>

Host Port: <none>

Containers:

orgchart:

Container ID: docker://425f5a5b26921ee6873233b714156201754489e1edf0b59f62bc9b928599c887

Image: jweissig/app:0.0.1

Image ID: docker-pullable://jweissig/app@sha256:54e7159831602dd8ffd8b81e1d4534c664a73e88f3f340df9c637fc16a5cf0b7

vault-agent:

Container ID: docker://4ddb15c3cdaa9c215031197dfdd62e43a842b92559e1d9a0596b184a21f144c7

Image: vault:1.7.0

Image ID: docker-pullable://vault@sha256:635cf1c3f9b10fe03aad375f94cc61f63d74a189662165285a8bf1c189ea04b8

Vault

Containers:

vault:

Container ID: docker://7194b9fd5e0c071ad4dfe12b4fd622c64bef8c324b70ed503224fc3807cb2ab4

Image: vault:1.7.0

Image ID: docker-pullable://vault@sha256:635cf1c3f9b10fe03aad375f94cc61f63d74a189662165285a8bf1c189ea04b8

Ports: 8200/TCP, 8201/TCP, 8202/TCP

Host Ports: 0/TCP, 0/TCP, 0/TCP

Vault-agent-injector

Containers:

sidecar-injector:

Container ID: docker://a53b4895a6479c16c5b3e9248c15fa85388f1ea3167a36dff81d4b75d954ff05

Image: hashicorp/vault-k8s:0.9.0

Image ID: docker-pullable://hashicorp/vault-k8s@sha256:65731b0513c95f683ee52528e6ccf24f6de0092700e869cdc5ff5d8354b5d86e

Environment:

AGENT_INJECT_LISTEN: :8080

AGENT_INJECT_LOG_LEVEL: info

AGENT_INJECT_VAULT_ADDR: http://vault.default.svc:8200

AGENT_INJECT_VAULT_AUTH_PATH: auth/kubernetes

AGENT_INJECT_VAULT_IMAGE: vault:1.7.0

AGENT_INJECT_TLS_AUTO: vault-agent-injector-cfg

AGENT_INJECT_TLS_AUTO_HOSTS: vault-agent-injector-svc,vault-agent-injector-svc.default,vault-agent-injector-svc.default.svc

AGENT_INJECT_LOG_FORMAT: standard

AGENT_INJECT_REVOKE_ON_SHUTDOWN: false

Agent Inject: Agent Inject is a mutation webhook controller that injects Vault Agent containers into pods meeting specific annotation criteria. (Requires Vault 1.3.1+)

Demo Steps

Clone github repo

$ git clone https://github.com/hashicorp/vault-guides.git
$ cd vault-guides/operations/provision-vault/kubernetes/minikube/vault-agent-sidecar

Install Vault & Vault-Agent-Injector

helm repo add hashicorp https://helm.releases.hashicorp.com

helm install vault hashicorp/vault --set "server.dev.enabled=true"

NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 8h
vault-agent-injector-67887bbd49-skk4s 1/1 Running 0 8h

Enable Key-Value Secret Engine

kubectl exec -it vault-0 -- /bin/sh

vault secrets enable -path=internal kv-v2

Create a secret at path internal/database/config with a username and password

vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"

exit

Enable the Kubernetes authentication method

kubectl exec -it vault-0 -- /bin/sh
vault auth enable kubernetes

Enables clients to authenticate with a Kubernetes Service Account Token.

Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate.

vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Success! Data written to: auth/kubernetes/config

Grant read capability to read from path: internal/data/database/config

vault policy write internal-app - <<EOF
path "internal/data/database/config" {
capabilities = ["read"]
}
EOF
Success! Uploaded policy: internal-app

Create a Kubernetes authentication role named internal-app

vault write auth/kubernetes/role/internal-app \
bound_service_account_names=internal-app \
bound_service_account_namespaces=default \
policies=internal-app \
ttl=24h
Success! Data written to: auth/kubernetes/role/internal-app

exit

Define a Kubernetes service account

kubectl get serviceaccounts

NAME SECRETS AGE
default 1 11h
vault 1 11h
vault-agent-injector 1 11h

Create a service account named internal-app .

$ cat service-account-internal-app.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: internal-app

kubectl apply --filename service-account-internal-app.yml

$ kubectl get serviceaccounts
NAME SECRETS AGE
default            1 11h
internal-app            1 9h
vault                1 11h
vault-agent-injector       1 11h

Launch an application

$ kubectl apply --filename deployment-orgchart.yml
deployment.apps/orgchart created

kubectl get pods
NAME                        READY STATUS RESTARTS AGE
orgchart-69697d9598-l878s 1/1        Running            0 6h
vault-0                            1/1        Running            0 8h
vault-agent-injector-67887bbd49-skk4s    1/1        Running            0 8h

Inject secrets into the pod

$ cat patch-inject-secrets.yml
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "internal-app"
vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"

$ kubectl patch deployment orgchart --patch "$(cat patch-inject-secrets.yml)"

deployment.apps/orgchart patched

$ kubectl get pods
NAME                        READY    STATUS RESTARTS AGE
orgchart-55c76c489d-6r7vc                0/2         Init:0/1            0    23s
orgchart-69697d9598-l878s                1/1        Running        0    20m
vault-0                                1/1        Running        0    78m
vault-agent-injector-5945fb98b5-tpglz            1/1        Running          0    78m

$ kubectl get pods
NAME                                READY STATUS RESTARTS AGE
orgchart-55c76c489d-6r7vc                        2/2    Running                0 6h
payroll                                        2/2    Running                0 5h45m
vault-0                                        1/1    Running                0 8h
vault-agent-injector-67887bbd49-skk4s                  1/1    Running                0 8h

Display the secret written to the orgchart container.

$ kubectl exec \
$(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
--container orgchart -- cat /vault/secrets/database-config.txt

data: map[password:db-secret-password username:db-readonly-user]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]

Pod with annotations

$ cat pod-payroll.yml
apiVersion: v1
kind: Pod
metadata:
name: payroll
labels:
app: payroll
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "internal-app"
vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
vault.hashicorp.com/agent-inject-template-database-config.txt: |
{{- with secret "internal/data/database/config" -}}
postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
{{- end -}}
spec:
serviceAccountName: internal-app
containers:
- name: payroll
image: jweissig/app:0.0.1

kubectl apply --filename pod-payroll.yml

kubectl get pods
NAME                                        READY    STATUS        RESTARTS AGE
orgchart-55c76c489d-6r7vc                                2/2            Running                0 6h
payroll                                               2/2            Running                0 5h45m
vault-0                                               1/1            Running                0 8h
vault-agent-injector-67887bbd49-skk4s                        1/1            Running                0 8h