1 ===========================
2 HashiCorp Vault Integration
3 ===========================
5 HashiCorp `Vault`_ can be used as a secure key management service for
6 `Server-Side Encryption`_ (SSE-KMS).
10 +---------+ +---------+ +-------+ +-------+
11 | Client | | RadosGW | | Vault | | OSD |
12 +---------+ +---------+ +-------+ +-------+
14 | key for key ID | | |
15 |-----------------+---------------->| |
19 |---------------->| request secret | |
20 | | key for key ID | |
21 | |---------------->| |
22 | |<----------------| |
26 | | encrypt object | |
27 | | with secret key | |
28 | |--------------+ | |
30 | |<-------------+ | |
32 | | store encrypted | |
34 | |------------------------------>|
36 #. `Vault secrets engines`_
37 #. `Vault authentication`_
38 #. `Vault namespaces`_
39 #. `Create a key in Vault`_
40 #. `Configure the Ceph Object Gateway`_
43 Some examples below use the Vault command line utility to interact with
44 Vault. You may need to set the following environment variable with the correct
45 address of your Vault server to use this utility::
47 export VAULT_ADDR='https://vault-server-fqdn:8200'
52 Vault provides several secrets engines, which can store, generate, and encrypt
53 data. Currently, the Object Gateway supports:
55 - `KV secrets engine`_ version 2
61 The KV secrets engine is used to store arbitrary key/value secrets in Vault. To
62 enable the KV engine version 2 in Vault, use the following command::
64 vault secrets enable -path secret kv-v2
66 The Object Gateway can be configured to use the KV engine version 2 with the
69 rgw crypt vault secret engine = kv
71 Transit secrets engine
72 ----------------------
74 The transit engine handles cryptographic functions on data in-transit. To enable
75 it in Vault, use the following command::
77 vault secrets enable transit
79 The Object Gateway can be configured to use the transit engine with the
82 rgw crypt vault secret engine = transit
87 Vault supports several authentication mechanisms. Currently, the Object
88 Gateway can be configured to authenticate to Vault using the
89 `Token authentication method`_ or a `Vault agent`_.
91 Most tokens in Vault have limited lifetimes and powers. The only
92 sort of Vault token that does not have a lifetime are root tokens.
93 For all other tokens, it is necessary to periodically refresh them,
94 either by performing initial authentication, or by renewing the token.
95 Ceph does not have any logic to perform either operation.
96 The simplest best way to use Vault tokens with ceph is to
97 also run the Vault agent and have it refresh the token file.
98 When the Vault agent is used in this mode, file system permissions
99 can be used to restrict who has the use of tokens.
101 Instead of having Vault agent refresh a token file, it can be told
102 to act as a proxy server. In this mode, Vault will add a token when
103 necessary and add it to requests passed to it before forwarding them on
104 to the real server. Vault agent will still handle token renewal just
105 as it would when storing a token in the filesystem. In this mode, it
106 is necessary to properly secure the network path rgw uses to reach the
107 Vault agent, such as having the Vault agent listen only to localhost.
109 Token policies for the object gateway
110 -------------------------------------
112 All Vault tokens have powers as specified by the polices attached
113 to that token. Multiple policies may be associated with one
114 token. You should only use the policy necessary for your
117 When using the kv secret engine with the object gateway::
119 vault policy write rgw-kv-policy -<<EOF
120 path "secret/data/*" {
121 capabilities = ["read"]
125 When using the transit secret engine with the object gateway::
127 vault policy write rgw-transit-policy -<<EOF
128 path "transit/keys/*" {
129 capabilities = [ "create", "update" ]
130 denied_parameters = {"exportable" = [], "allow_plaintext_backup" = [] }
133 path "transit/keys/*" {
134 capabilities = ["read", "delete"]
137 path "transit/keys/" {
138 capabilities = ["list"]
141 path "transit/keys/+/rotate" {
142 capabilities = [ "update" ]
146 capabilities = [ "update" ]
150 If you had previously used an older version of ceph with the
151 transit secret engine, you might need the following policy::
153 vault policy write old-rgw-transit-policy -<<EOF
154 path "transit/export/encryption-key/*" {
155 capabilities = ["read"]
163 .. note: Never use root tokens with ceph in production environments.
165 The token authentication method expects a Vault token to be present in a
166 plaintext file. The Object Gateway can be configured to use token authentication
167 with the following settings::
169 rgw crypt vault auth = token
170 rgw crypt vault token file = /run/.rgw-vault-token
171 rgw crypt vault addr = https://vault-server-fqdn:8200
173 Adjust these settings to match your configuration.
174 For security reasons, the token file must be readable by the Object Gateway
177 You might set up vault agent as follows::
179 vault write auth/approle/role/rgw-ap \
180 token_policies=rgw-transit-policy,default \
183 Change the policy here to match your configuration.
187 vault read auth/approle/role/rgw-ap/role-id -format=json | \
190 Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-role-id
194 vault read auth/approle/role/rgw-ap/role-id -format=json | \
197 Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-secret-id
199 Create configuration for the Vault agent, such as::
201 pid_file = "/run/rgw-vault-agent-pid"
204 mount_path = "auth/approle"
206 role_id_file_path ="/usr/local/etc/vault/.rgw-ap-role-id"
207 secret_id_file_path ="/usr/local/etc/vault/.rgw-ap-secret-id"
208 remove_secret_id_file_after_reading ="false"
213 path = "/run/.rgw-vault-token"
218 address = "https://vault-server-fqdn:8200"
221 Then use systemctl or another method of your choice to run
222 a persistent daemon with the following arguments::
224 /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl
226 Once the vault agent is running, the token file should be populated
232 The Vault agent is a client daemon that provides authentication to Vault and
233 manages token renewal and caching. It typically runs on the same host as the
234 Object Gateway. With a Vault agent, it is possible to use other Vault
235 authentication mechanism such as AppRole, AWS, Certs, JWT, and Azure.
237 The Object Gateway can be configured to use a Vault agent with the following
240 rgw crypt vault auth = agent
241 rgw crypt vault addr = http://127.0.0.1:8100
243 You might set up vault agent as follows::
245 vault write auth/approle/role/rgw-ap \
246 token_policies=rgw-transit-policy,default \
249 Change the policy here to match your configuration.
252 vault read auth/approle/role/rgw-ap/role-id -format=json | \
255 Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-role-id
258 vault read auth/approle/role/rgw-ap/role-id -format=json | \
261 Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-secret-id
263 Create configuration for the Vault agent, such as::
265 pid_file = "/run/rgw-vault-agent-pid"
268 mount_path = "auth/approle"
270 role_id_file_path ="/usr/local/etc/vault/.rgw-ap-role-id"
271 secret_id_file_path ="/usr/local/etc/vault/.rgw-ap-secret-id"
272 remove_secret_id_file_after_reading ="false"
277 use_auto_auth_token = true
280 address = "127.0.0.1:8100"
284 address = "https://vault-server-fqdn:8200"
287 Then use systemctl or another method of your choice to run
288 a persistent daemon with the following arguments::
290 /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl
292 Once the vault agent is running, you should find it listening
293 to port 8100 on localhost, and you should be able to interact
294 with it using the vault command.
299 In the Enterprise version, Vault supports the concept of `namespaces`_, which
300 allows centralized management for teams within an organization while ensuring
301 that those teams operate within isolated environments known as tenants.
303 The Object Gateway can be configured to access Vault within a particular
304 namespace using the following configuration setting::
306 rgw crypt vault namespace = tenant1
308 Create a key in Vault
309 =====================
311 .. note:: Keys for server-side encryption must be 256-bit long and base-64
317 A key for server-side encryption can be created in the KV version 2 engine using
318 the command line utility, as in the following example::
320 vault kv put secret/myproject/mybucketkey key=$(openssl rand -base64 32)
324 ====== Metadata ======
327 created_time 2019-08-29T17:01:09.095824999Z
332 Note that in the KV secrets engine, secrets are stored as key-value pairs, and
333 the Gateway expects the key name to be ``key``, i.e. the secret must be in the
334 form ``key=<secret key>``.
336 Using the Transit engine
337 ------------------------
339 Keys created for use with the Transit engine should no longer be marked
340 exportable. They can be created with::
342 vault write -f transit/keys/mybucketkey
344 The command above creates a keyring, which contains a key of type
345 ``aes256-gcm96`` by default. To verify that the key was correctly created, use
346 the following command::
348 vault read transit/mybucketkey
359 Configure the Ceph Object Gateway
360 =================================
362 Edit the Ceph configuration file to enable Vault as a KMS backend for
363 server-side encryption::
365 rgw crypt s3 kms backend = vault
367 Choose the Vault authentication method, e.g.::
369 rgw crypt vault auth = token
370 rgw crypt vault token file = /run/.rgw-vault-token
371 rgw crypt vault addr = https://vault-server-fqdn:8200
375 rgw crypt vault auth = agent
376 rgw crypt vault addr = http://localhost:8100
378 Choose the secrets engine::
380 rgw crypt vault secret engine = kv
384 rgw crypt vault secret engine = transit
386 Optionally, set the Vault namespace where encryption keys will be fetched from::
388 rgw crypt vault namespace = tenant1
390 Finally, the URLs where the Gateway will retrieve encryption keys from Vault can
391 be restricted by setting a path prefix. For instance, the Gateway can be
392 restricted to fetch KV keys as follows::
394 rgw crypt vault prefix = /v1/secret/data
396 Or, when using the transit secret engine::
398 rgw crypt vault prefix = /v1/transit
400 In the example above, the Gateway would only fetch transit encryption keys under
401 ``https://vault-server:8200/v1/transit``.
403 You can use custom ssl certs to authenticate with vault with help of
406 rgw crypt vault verify ssl = true
407 rgw crypt vault ssl cacert = /etc/ceph/vault.ca
408 rgw crypt vault ssl clientcert = /etc/ceph/vault.crt
409 rgw crypt vault ssl clientkey = /etc/ceph/vault.key
411 where vault.ca is CA certificate and vault.key/vault.crt are private key and ssl
412 certificate generated for RGW to access the vault server. It highly recommended to
413 set this option true, setting false is very dangerous and need to avoid since this
414 runs in very secured environments.
416 Transit engine compatibility support
417 ------------------------------------
418 The transit engine has compatibility support for previous
419 versions of ceph, which used the transit engine as a simple key store.
421 There is a "compat" option which can be given to the transit
422 engine to configure the compatibility support,
424 To entirely disable backwards support, use::
426 rgw crypt vault secret engine = transit compat=0
428 This will be the default in future versions. and is safe to use
429 for new installs using the current version.
431 This is the normal default with the current version::
433 rgw crypt vault secret engine = transit compat=1
435 This enables the new engine for newly created objects,
436 but still allows the old engine to be used for old objects.
437 In order to access old and new objects, the vault token given
438 to ceph must have both the old and new transit policies.
440 To force use of only the old engine, use::
442 rgw crypt vault secret engine = transit compat=2
444 This mode is automatically selected if the vault prefix
445 ends in export/encryption-key, which was the previously
451 When uploading an object to the Gateway, provide the SSE key ID in the request.
452 As an example, for the kv engine, using the AWS command-line client::
454 aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id myproject/mybucketkey
456 As an example, for the transit engine (new flavor), using the AWS command-line client::
458 aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id mybucketkey
460 The Object Gateway will fetch the key from Vault, encrypt the object and store
461 it in the bucket. Any request to download the object will make the Gateway
462 automatically retrieve the correspondent key from Vault and decrypt the object.
464 Note that the secret will be fetched from Vault using a URL constructed by
465 concatenating the base address (``rgw crypt vault addr``), the (optional)
466 URL prefix (``rgw crypt vault prefix``), and finally the key ID.
468 In the kv engine example above, the Gateway would fetch the secret from::
470 http://vaultserver:8200/v1/secret/data/myproject/mybucketkey
472 In the transit engine example above, the Gateway would encrypt the secret using this key::
474 http://vaultserver:8200/v1/transit/mybucketkey
476 .. _Server-Side Encryption: ../encryption
477 .. _Vault: https://www.vaultproject.io/docs/
478 .. _Token authentication method: https://www.vaultproject.io/docs/auth/token.html
479 .. _Vault agent: https://www.vaultproject.io/docs/agent/index.html
480 .. _KV Secrets engine: https://www.vaultproject.io/docs/secrets/kv/
481 .. _Transit engine: https://www.vaultproject.io/docs/secrets/transit
482 .. _namespaces: https://www.vaultproject.io/docs/enterprise/namespaces/index.html