1 ===========================
2 HashiCorp Vault Integration
3 ===========================
5 HashiCorp `Vault`_ can be used as a secure key management service for
6 `Server-Side Encryption`_ (SSE-KMS).
10 +---------+ +---------+ +-------+ +-------+
11 | Client | | RadosGW | | Vault | | OSD |
12 +---------+ +---------+ +-------+ +-------+
14 | key for key ID | | |
15 |-----------------+---------------->| |
19 |---------------->| request secret | |
20 | | key for key ID | |
21 | |---------------->| |
22 | |<----------------| |
26 | | encrypt object | |
27 | | with secret key | |
28 | |--------------+ | |
30 | |<-------------+ | |
32 | | store encrypted | |
34 | |------------------------------>|
36 #. `Vault secrets engines`_
37 #. `Vault authentication`_
38 #. `Vault namespaces`_
39 #. `Create a key in Vault`_
40 #. `Configure the Ceph Object Gateway`_
43 Some examples below use the Vault command line utility to interact with
44 Vault. You may need to set the following environment variable with the correct
45 address of your Vault server to use this utility::
47 export VAULT_ADDR='https://vault-server-fqdn:8200'
52 Vault provides several secrets engines, which can store, generate, and encrypt
53 data. Currently, the Object Gateway supports:
55 - `KV secrets engine`_ version 2
61 The KV secrets engine is used to store arbitrary key/value secrets in Vault. To
62 enable the KV engine version 2 in Vault, use the following command::
64 vault secrets enable -path secret kv-v2
66 The Object Gateway can be configured to use the KV engine version 2 with the
69 rgw crypt vault secret engine = kv
71 Transit secrets engine
72 ----------------------
74 The transit engine handles cryptographic functions on data in-transit. To enable
75 it in Vault, use the following command::
77 vault secrets enable transit
79 The Object Gateway can be configured to use the transit engine with the
82 rgw crypt vault secret engine = transit
87 Vault supports several authentication mechanisms. Currently, the Object
88 Gateway can be configured to authenticate to Vault using the
89 `Token authentication method`_ or a `Vault agent`_.
91 Most tokens in Vault have limited lifetimes and powers. The only
92 sort of Vault token that does not have a lifetime are root tokens.
93 For all other tokens, it is necessary to periodically refresh them,
94 either by performing initial authentication, or by renewing the token.
95 Ceph does not have any logic to perform either operation.
96 The simplest best way to use Vault tokens with ceph is to
97 also run the Vault agent and have it refresh the token file.
98 When the Vault agent is used in this mode, file system permissions
99 can be used to restrict who has the use of tokens.
101 Instead of having Vault agent refresh a token file, it can be told
102 to act as a proxy server. In this mode, Vault will add a token when
103 necessary and add it to requests passed to it before forwarding them on
104 to the real server. Vault agent will still handle token renewal just
105 as it would when storing a token in the filesystem. In this mode, it
106 is necessary to properly secure the network path rgw uses to reach the
107 Vault agent, such as having the Vault agent listen only to localhost.
109 Token policies for the object gateway
110 -------------------------------------
112 All Vault tokens have powers as specified by the polices attached
113 to that token. Multiple policies may be associated with one
114 token. You should only use the policies necessary for your
117 When using the kv secret engine with the object gateway::
119 vault policy write rgw-kv-policy -<<EOF
120 path "secret/data/*" {
121 capabilities = ["read"]
125 When using the transit secret engine with the object gateway::
127 vault policy write rgw-transit-policy -<<EOF
128 path "transit/keys/*" {
129 capabilities = [ "create", "update" ]
130 denied_parameters = {"exportable" = [], "allow_plaintext_backup" = [] }
133 path "transit/keys/*" {
134 capabilities = ["read", "delete"]
137 path "transit/keys/" {
138 capabilities = ["list"]
141 path "transit/keys/+/rotate" {
142 capabilities = [ "update" ]
146 capabilities = [ "update" ]
150 If you had previously used an older version of ceph with the
151 transit secret engine, you might need the following policy::
153 vault policy write old-rgw-transit-policy -<<EOF
154 path "transit/export/encryption-key/*" {
155 capabilities = ["read"]
159 If you are using both sse-kms and sse-s3, then you should point
160 each to separate containers. You could either use separate
161 vault instances, or you could use either separately mounted
162 transit instances, or different branches under a common transit
163 pointpoint. If you are not using separate vault instances, you can
164 Use these to point kms and sse-s3 to separate containers:
165 ``rgw_crypt_vault_prefix``
167 ``rgw_crypt_sse_s3_vault_prefix``.
168 When granting vault permissions to sse-kms bucket owners, you should
169 not give them permission to muck around with sse-s3 keys;
170 only ceph itself should be doing that.
175 .. note: Never use root tokens with ceph in production environments.
177 The token authentication method expects a Vault token to be present in a
178 plaintext file. The Object Gateway can be configured to use token authentication
179 with the following settings::
181 rgw crypt vault auth = token
182 rgw crypt vault token file = /run/.rgw-vault-token
183 rgw crypt vault addr = https://vault-server-fqdn:8200
185 Adjust these settings to match your configuration.
186 For security reasons, the token file must be readable by the Object Gateway
189 You might set up vault agent as follows::
191 vault write auth/approle/role/rgw-ap \
192 token_policies=rgw-transit-policy,default \
195 Change the policy here to match your configuration.
199 vault read auth/approle/role/rgw-ap/role-id -format=json | \
202 Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-role-id
206 vault read auth/approle/role/rgw-ap/role-id -format=json | \
209 Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-secret-id
211 Create configuration for the Vault agent, such as::
213 pid_file = "/run/rgw-vault-agent-pid"
216 mount_path = "auth/approle"
218 role_id_file_path ="/usr/local/etc/vault/.rgw-ap-role-id"
219 secret_id_file_path ="/usr/local/etc/vault/.rgw-ap-secret-id"
220 remove_secret_id_file_after_reading ="false"
225 path = "/run/.rgw-vault-token"
230 address = "https://vault-server-fqdn:8200"
233 Then use systemctl or another method of your choice to run
234 a persistent daemon with the following arguments::
236 /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl
238 Once the vault agent is running, the token file should be populated
244 The Vault agent is a client daemon that provides authentication to Vault and
245 manages token renewal and caching. It typically runs on the same host as the
246 Object Gateway. With a Vault agent, it is possible to use other Vault
247 authentication mechanism such as AppRole, AWS, Certs, JWT, and Azure.
249 The Object Gateway can be configured to use a Vault agent with the following
252 rgw crypt vault auth = agent
253 rgw crypt vault addr = http://127.0.0.1:8100
255 You might set up vault agent as follows::
257 vault write auth/approle/role/rgw-ap \
258 token_policies=rgw-transit-policy,default \
261 Change the policy here to match your configuration.
264 vault read auth/approle/role/rgw-ap/role-id -format=json | \
267 Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-role-id
270 vault read auth/approle/role/rgw-ap/role-id -format=json | \
273 Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-secret-id
275 Create configuration for the Vault agent, such as::
277 pid_file = "/run/rgw-vault-agent-pid"
280 mount_path = "auth/approle"
282 role_id_file_path ="/usr/local/etc/vault/.rgw-ap-role-id"
283 secret_id_file_path ="/usr/local/etc/vault/.rgw-ap-secret-id"
284 remove_secret_id_file_after_reading ="false"
289 use_auto_auth_token = true
292 address = "127.0.0.1:8100"
296 address = "https://vault-server-fqdn:8200"
299 Then use systemctl or another method of your choice to run
300 a persistent daemon with the following arguments::
302 /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl
304 Once the vault agent is running, you should find it listening
305 to port 8100 on localhost, and you should be able to interact
306 with it using the vault command.
311 In the Enterprise version, Vault supports the concept of `namespaces`_, which
312 allows centralized management for teams within an organization while ensuring
313 that those teams operate within isolated environments known as tenants.
315 The Object Gateway can be configured to access Vault within a particular
316 namespace using the following configuration setting::
318 rgw crypt vault namespace = tenant1
320 Create a key in Vault
321 =====================
323 .. note:: Keys for server-side encryption must be 256-bit long and base-64
329 A key for server-side encryption can be created in the KV version 2 engine using
330 the command line utility, as in the following example::
332 vault kv put secret/myproject/mybucketkey key=$(openssl rand -base64 32)
336 ====== Metadata ======
339 created_time 2019-08-29T17:01:09.095824999Z
344 Note that in the KV secrets engine, secrets are stored as key-value pairs, and
345 the Gateway expects the key name to be ``key``, i.e. the secret must be in the
346 form ``key=<secret key>``.
348 Using the Transit engine
349 ------------------------
351 Keys created for use with the Transit engine should no longer be marked
352 exportable. They can be created with::
354 vault write -f transit/keys/mybucketkey
356 The command above creates a keyring, which contains a key of type
357 ``aes256-gcm96`` by default. To verify that the key was correctly created, use
358 the following command::
360 vault read transit/mybucketkey
371 Configure the Ceph Object Gateway
372 =================================
374 Edit the Ceph configuration file to enable Vault as a KMS backend for
375 server-side encryption::
377 rgw crypt s3 kms backend = vault
379 Choose the Vault authentication method, e.g.::
381 rgw crypt vault auth = token
382 rgw crypt vault token file = /run/.rgw-vault-token
383 rgw crypt vault addr = https://vault-server-fqdn:8200
387 rgw crypt vault auth = agent
388 rgw crypt vault addr = http://localhost:8100
390 Choose the secrets engine::
392 rgw crypt vault secret engine = kv
396 rgw crypt vault secret engine = transit
398 Optionally, set the Vault namespace where encryption keys will be fetched from::
400 rgw crypt vault namespace = tenant1
402 Finally, the URLs where the Gateway will retrieve encryption keys from Vault can
403 be restricted by setting a path prefix. For instance, the Gateway can be
404 restricted to fetch KV keys as follows::
406 rgw crypt vault prefix = /v1/secret/data
408 Or, when using the transit secret engine::
410 rgw crypt vault prefix = /v1/transit
412 In the example above, the Gateway would only fetch transit encryption keys under
413 ``https://vault-server:8200/v1/transit``.
415 You can use custom ssl certs to authenticate with vault with help of
418 rgw crypt vault verify ssl = true
419 rgw crypt vault ssl cacert = /etc/ceph/vault.ca
420 rgw crypt vault ssl clientcert = /etc/ceph/vault.crt
421 rgw crypt vault ssl clientkey = /etc/ceph/vault.key
423 where vault.ca is CA certificate and vault.key/vault.crt are private key and ssl
424 certificate generated for RGW to access the vault server. It highly recommended to
425 set this option true, setting false is very dangerous and need to avoid since this
426 runs in very secured environments.
428 Transit engine compatibility support
429 ------------------------------------
430 The transit engine has compatibility support for previous
431 versions of ceph, which used the transit engine as a simple key store.
433 There is a "compat" option which can be given to the transit
434 engine to configure the compatibility support,
436 To entirely disable backwards support, use::
438 rgw crypt vault secret engine = transit compat=0
440 This will be the default in future versions. and is safe to use
441 for new installs using the current version.
443 This is the normal default with the current version::
445 rgw crypt vault secret engine = transit compat=1
447 This enables the new engine for newly created objects,
448 but still allows the old engine to be used for old objects.
449 In order to access old and new objects, the vault token given
450 to ceph must have both the old and new transit policies.
452 To force use of only the old engine, use::
454 rgw crypt vault secret engine = transit compat=2
456 This mode is automatically selected if the vault prefix
457 ends in export/encryption-key, which was the previously
463 When uploading an object to the Gateway, provide the SSE key ID in the request.
464 As an example, for the kv engine, using the AWS command-line client::
466 aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id myproject/mybucketkey
468 As an example, for the transit engine (new flavor), using the AWS command-line client::
470 aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id mybucketkey
472 The Object Gateway will fetch the key from Vault, encrypt the object and store
473 it in the bucket. Any request to download the object will make the Gateway
474 automatically retrieve the correspondent key from Vault and decrypt the object.
476 Note that the secret will be fetched from Vault using a URL constructed by
477 concatenating the base address (``rgw crypt vault addr``), the (optional)
478 URL prefix (``rgw crypt vault prefix``), and finally the key ID.
480 In the kv engine example above, the Gateway would fetch the secret from::
482 http://vaultserver:8200/v1/secret/data/myproject/mybucketkey
484 In the transit engine example above, the Gateway would encrypt the secret using this key::
486 http://vaultserver:8200/v1/transit/mybucketkey
488 .. _Server-Side Encryption: ../encryption
489 .. _Vault: https://www.vaultproject.io/docs/
490 .. _Token authentication method: https://www.vaultproject.io/docs/auth/token.html
491 .. _Vault agent: https://www.vaultproject.io/docs/agent/index.html
492 .. _KV Secrets engine: https://www.vaultproject.io/docs/secrets/kv/
493 .. _Transit engine: https://www.vaultproject.io/docs/secrets/transit
494 .. _namespaces: https://www.vaultproject.io/docs/enterprise/namespaces/index.html