]>
Commit | Line | Data |
---|---|---|
9f95a23c TL |
1 | =========================== |
2 | HashiCorp Vault Integration | |
3 | =========================== | |
4 | ||
5 | HashiCorp `Vault`_ can be used as a secure key management service for | |
6 | `Server-Side Encryption`_ (SSE-KMS). | |
7 | ||
f91f0fd5 TL |
8 | .. ditaa:: |
9 | ||
10 | +---------+ +---------+ +-------+ +-------+ | |
9f95a23c TL |
11 | | Client | | RadosGW | | Vault | | OSD | |
12 | +---------+ +---------+ +-------+ +-------+ | |
13 | | create secret | | | | |
14 | | key for key ID | | | | |
15 | |-----------------+---------------->| | | |
16 | | | | | | |
17 | | upload object | | | | |
18 | | with key ID | | | | |
19 | |---------------->| request secret | | | |
20 | | | key for key ID | | | |
21 | | |---------------->| | | |
22 | | |<----------------| | | |
23 | | | return secret | | | |
24 | | | key | | | |
25 | | | | | | |
26 | | | encrypt object | | | |
27 | | | with secret key | | | |
28 | | |--------------+ | | | |
29 | | | | | | | |
30 | | |<-------------+ | | | |
31 | | | | | | |
32 | | | store encrypted | | | |
33 | | | object | | | |
34 | | |------------------------------>| | |
35 | ||
36 | #. `Vault secrets engines`_ | |
37 | #. `Vault authentication`_ | |
38 | #. `Vault namespaces`_ | |
39 | #. `Create a key in Vault`_ | |
40 | #. `Configure the Ceph Object Gateway`_ | |
41 | #. `Upload object`_ | |
42 | ||
43 | Some examples below use the Vault command line utility to interact with | |
44 | Vault. You may need to set the following environment variable with the correct | |
45 | address of your Vault server to use this utility:: | |
46 | ||
f67539c2 | 47 | export VAULT_ADDR='https://vault-server-fqdn:8200' |
9f95a23c TL |
48 | |
49 | Vault secrets engines | |
50 | ===================== | |
51 | ||
52 | Vault provides several secrets engines, which can store, generate, and encrypt | |
53 | data. Currently, the Object Gateway supports: | |
54 | ||
55 | - `KV secrets engine`_ version 2 | |
56 | - `Transit engine`_ | |
57 | ||
58 | KV secrets engine | |
59 | ----------------- | |
60 | ||
61 | The KV secrets engine is used to store arbitrary key/value secrets in Vault. To | |
62 | enable the KV engine version 2 in Vault, use the following command:: | |
63 | ||
f67539c2 | 64 | vault secrets enable -path secret kv-v2 |
9f95a23c TL |
65 | |
66 | The Object Gateway can be configured to use the KV engine version 2 with the | |
67 | following setting:: | |
68 | ||
69 | rgw crypt vault secret engine = kv | |
70 | ||
71 | Transit secrets engine | |
72 | ---------------------- | |
73 | ||
74 | The transit engine handles cryptographic functions on data in-transit. To enable | |
75 | it in Vault, use the following command:: | |
76 | ||
77 | vault secrets enable transit | |
78 | ||
79 | The Object Gateway can be configured to use the transit engine with the | |
80 | following setting:: | |
81 | ||
82 | rgw crypt vault secret engine = transit | |
83 | ||
84 | Vault authentication | |
85 | ==================== | |
86 | ||
87 | Vault supports several authentication mechanisms. Currently, the Object | |
88 | Gateway can be configured to authenticate to Vault using the | |
89 | `Token authentication method`_ or a `Vault agent`_. | |
90 | ||
f67539c2 TL |
91 | Most tokens in Vault have limited lifetimes and powers. The only |
92 | sort of Vault token that does not have a lifetime are root tokens. | |
20effc67 | 93 | For all other tokens, it is necessary to periodically refresh them, |
f67539c2 TL |
94 | either by performing initial authentication, or by renewing the token. |
95 | Ceph does not have any logic to perform either operation. | |
96 | The simplest best way to use Vault tokens with ceph is to | |
97 | also run the Vault agent and have it refresh the token file. | |
98 | When the Vault agent is used in this mode, file system permissions | |
99 | can be used to restrict who has the use of tokens. | |
100 | ||
101 | Instead of having Vault agent refresh a token file, it can be told | |
102 | to act as a proxy server. In this mode, Vault will add a token when | |
103 | necessary and add it to requests passed to it before forwarding them on | |
104 | to the real server. Vault agent will still handle token renewal just | |
105 | as it would when storing a token in the filesystem. In this mode, it | |
106 | is necessary to properly secure the network path rgw uses to reach the | |
107 | Vault agent, such as having the Vault agent listen only to localhost. | |
108 | ||
109 | Token policies for the object gateway | |
110 | ------------------------------------- | |
111 | ||
112 | All Vault tokens have powers as specified by the polices attached | |
113 | to that token. Multiple policies may be associated with one | |
2a845540 | 114 | token. You should only use the policies necessary for your |
f67539c2 TL |
115 | configuration. |
116 | ||
117 | When using the kv secret engine with the object gateway:: | |
118 | ||
119 | vault policy write rgw-kv-policy -<<EOF | |
120 | path "secret/data/*" { | |
121 | capabilities = ["read"] | |
122 | } | |
123 | EOF | |
124 | ||
125 | When using the transit secret engine with the object gateway:: | |
126 | ||
127 | vault policy write rgw-transit-policy -<<EOF | |
128 | path "transit/keys/*" { | |
129 | capabilities = [ "create", "update" ] | |
130 | denied_parameters = {"exportable" = [], "allow_plaintext_backup" = [] } | |
131 | } | |
132 | ||
133 | path "transit/keys/*" { | |
134 | capabilities = ["read", "delete"] | |
135 | } | |
136 | ||
137 | path "transit/keys/" { | |
138 | capabilities = ["list"] | |
139 | } | |
140 | ||
141 | path "transit/keys/+/rotate" { | |
142 | capabilities = [ "update" ] | |
143 | } | |
144 | ||
145 | path "transit/*" { | |
146 | capabilities = [ "update" ] | |
147 | } | |
148 | EOF | |
149 | ||
150 | If you had previously used an older version of ceph with the | |
151 | transit secret engine, you might need the following policy:: | |
152 | ||
153 | vault policy write old-rgw-transit-policy -<<EOF | |
154 | path "transit/export/encryption-key/*" { | |
155 | capabilities = ["read"] | |
156 | } | |
157 | EOF | |
158 | ||
2a845540 TL |
159 | If you are using both sse-kms and sse-s3, then you should point |
160 | each to separate containers. You could either use separate | |
161 | vault instances, or you could use either separately mounted | |
162 | transit instances, or different branches under a common transit | |
163 | pointpoint. If you are not using separate vault instances, you can | |
164 | Use these to point kms and sse-s3 to separate containers: | |
165 | ``rgw_crypt_vault_prefix`` | |
166 | and/or | |
167 | ``rgw_crypt_sse_s3_vault_prefix``. | |
168 | When granting vault permissions to sse-kms bucket owners, you should | |
169 | not give them permission to muck around with sse-s3 keys; | |
170 | only ceph itself should be doing that. | |
f67539c2 | 171 | |
9f95a23c TL |
172 | Token authentication |
173 | -------------------- | |
174 | ||
f67539c2 | 175 | .. note: Never use root tokens with ceph in production environments. |
9f95a23c TL |
176 | |
177 | The token authentication method expects a Vault token to be present in a | |
178 | plaintext file. The Object Gateway can be configured to use token authentication | |
179 | with the following settings:: | |
180 | ||
181 | rgw crypt vault auth = token | |
f67539c2 TL |
182 | rgw crypt vault token file = /run/.rgw-vault-token |
183 | rgw crypt vault addr = https://vault-server-fqdn:8200 | |
9f95a23c | 184 | |
20effc67 | 185 | Adjust these settings to match your configuration. |
9f95a23c | 186 | For security reasons, the token file must be readable by the Object Gateway |
f67539c2 | 187 | only. |
9f95a23c | 188 | |
f67539c2 | 189 | You might set up vault agent as follows:: |
9f95a23c | 190 | |
f67539c2 TL |
191 | vault write auth/approle/role/rgw-ap \ |
192 | token_policies=rgw-transit-policy,default \ | |
193 | token_max_ttl=60m | |
9f95a23c | 194 | |
f67539c2 | 195 | Change the policy here to match your configuration. |
9f95a23c | 196 | |
f67539c2 | 197 | Get the role-id:: |
9f95a23c | 198 | |
f67539c2 TL |
199 | vault read auth/approle/role/rgw-ap/role-id -format=json | \ |
200 | jq -r .data.role_id | |
201 | ||
202 | Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-role-id | |
203 | ||
204 | Get the secret-id:: | |
205 | ||
206 | vault read auth/approle/role/rgw-ap/role-id -format=json | \ | |
207 | jq -r .data.role_id | |
208 | ||
209 | Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-secret-id | |
210 | ||
211 | Create configuration for the Vault agent, such as:: | |
212 | ||
213 | pid_file = "/run/rgw-vault-agent-pid" | |
214 | auto_auth { | |
215 | method "AppRole" { | |
216 | mount_path = "auth/approle" | |
217 | config = { | |
218 | role_id_file_path ="/usr/local/etc/vault/.rgw-ap-role-id" | |
219 | secret_id_file_path ="/usr/local/etc/vault/.rgw-ap-secret-id" | |
220 | remove_secret_id_file_after_reading ="false" | |
221 | } | |
222 | } | |
223 | sink "file" { | |
224 | config = { | |
225 | path = "/run/.rgw-vault-token" | |
226 | } | |
227 | } | |
228 | } | |
229 | vault { | |
230 | address = "https://vault-server-fqdn:8200" | |
231 | } | |
232 | ||
233 | Then use systemctl or another method of your choice to run | |
234 | a persistent daemon with the following arguments:: | |
9f95a23c | 235 | |
f67539c2 | 236 | /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl |
9f95a23c | 237 | |
f67539c2 TL |
238 | Once the vault agent is running, the token file should be populated |
239 | with a valid token. | |
9f95a23c TL |
240 | |
241 | Vault agent | |
242 | ----------- | |
243 | ||
244 | The Vault agent is a client daemon that provides authentication to Vault and | |
245 | manages token renewal and caching. It typically runs on the same host as the | |
246 | Object Gateway. With a Vault agent, it is possible to use other Vault | |
247 | authentication mechanism such as AppRole, AWS, Certs, JWT, and Azure. | |
248 | ||
249 | The Object Gateway can be configured to use a Vault agent with the following | |
250 | settings:: | |
251 | ||
252 | rgw crypt vault auth = agent | |
f67539c2 TL |
253 | rgw crypt vault addr = http://127.0.0.1:8100 |
254 | ||
255 | You might set up vault agent as follows:: | |
256 | ||
257 | vault write auth/approle/role/rgw-ap \ | |
258 | token_policies=rgw-transit-policy,default \ | |
259 | token_max_ttl=60m | |
260 | ||
261 | Change the policy here to match your configuration. | |
262 | ||
263 | Get the role-id: | |
264 | vault read auth/approle/role/rgw-ap/role-id -format=json | \ | |
265 | jq -r .data.role_id | |
266 | ||
267 | Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-role-id | |
268 | ||
269 | Get the secret-id: | |
270 | vault read auth/approle/role/rgw-ap/role-id -format=json | \ | |
271 | jq -r .data.role_id | |
272 | ||
273 | Store the output in some file, such as /usr/local/etc/vault/.rgw-ap-secret-id | |
274 | ||
275 | Create configuration for the Vault agent, such as:: | |
276 | ||
277 | pid_file = "/run/rgw-vault-agent-pid" | |
278 | auto_auth { | |
279 | method "AppRole" { | |
280 | mount_path = "auth/approle" | |
281 | config = { | |
282 | role_id_file_path ="/usr/local/etc/vault/.rgw-ap-role-id" | |
283 | secret_id_file_path ="/usr/local/etc/vault/.rgw-ap-secret-id" | |
284 | remove_secret_id_file_after_reading ="false" | |
285 | } | |
286 | } | |
287 | } | |
288 | cache { | |
289 | use_auto_auth_token = true | |
290 | } | |
291 | listener "tcp" { | |
292 | address = "127.0.0.1:8100" | |
293 | tls_disable = true | |
294 | } | |
295 | vault { | |
296 | address = "https://vault-server-fqdn:8200" | |
297 | } | |
298 | ||
299 | Then use systemctl or another method of your choice to run | |
300 | a persistent daemon with the following arguments:: | |
301 | ||
302 | /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl | |
303 | ||
304 | Once the vault agent is running, you should find it listening | |
305 | to port 8100 on localhost, and you should be able to interact | |
306 | with it using the vault command. | |
9f95a23c TL |
307 | |
308 | Vault namespaces | |
309 | ================ | |
310 | ||
311 | In the Enterprise version, Vault supports the concept of `namespaces`_, which | |
312 | allows centralized management for teams within an organization while ensuring | |
313 | that those teams operate within isolated environments known as tenants. | |
314 | ||
315 | The Object Gateway can be configured to access Vault within a particular | |
316 | namespace using the following configuration setting:: | |
317 | ||
318 | rgw crypt vault namespace = tenant1 | |
319 | ||
320 | Create a key in Vault | |
321 | ===================== | |
322 | ||
323 | .. note:: Keys for server-side encryption must be 256-bit long and base-64 | |
324 | encoded. | |
325 | ||
326 | Using the KV engine | |
327 | ------------------- | |
328 | ||
329 | A key for server-side encryption can be created in the KV version 2 engine using | |
330 | the command line utility, as in the following example:: | |
331 | ||
332 | vault kv put secret/myproject/mybucketkey key=$(openssl rand -base64 32) | |
333 | ||
334 | Sample output:: | |
335 | ||
336 | ====== Metadata ====== | |
337 | Key Value | |
338 | --- ----- | |
339 | created_time 2019-08-29T17:01:09.095824999Z | |
340 | deletion_time n/a | |
341 | destroyed false | |
342 | version 1 | |
343 | ||
344 | Note that in the KV secrets engine, secrets are stored as key-value pairs, and | |
345 | the Gateway expects the key name to be ``key``, i.e. the secret must be in the | |
346 | form ``key=<secret key>``. | |
347 | ||
348 | Using the Transit engine | |
349 | ------------------------ | |
350 | ||
f67539c2 TL |
351 | Keys created for use with the Transit engine should no longer be marked |
352 | exportable. They can be created with:: | |
9f95a23c | 353 | |
f67539c2 | 354 | vault write -f transit/keys/mybucketkey |
9f95a23c TL |
355 | |
356 | The command above creates a keyring, which contains a key of type | |
357 | ``aes256-gcm96`` by default. To verify that the key was correctly created, use | |
358 | the following command:: | |
359 | ||
f67539c2 | 360 | vault read transit/mybucketkey |
9f95a23c TL |
361 | |
362 | Sample output:: | |
363 | ||
364 | Key Value | |
365 | --- ----- | |
f67539c2 TL |
366 | derived false |
367 | exportable false | |
368 | name mybucketkey | |
369 | type aes256-gcm96 | |
9f95a23c TL |
370 | |
371 | Configure the Ceph Object Gateway | |
372 | ================================= | |
373 | ||
374 | Edit the Ceph configuration file to enable Vault as a KMS backend for | |
375 | server-side encryption:: | |
376 | ||
377 | rgw crypt s3 kms backend = vault | |
378 | ||
379 | Choose the Vault authentication method, e.g.:: | |
380 | ||
381 | rgw crypt vault auth = token | |
f67539c2 TL |
382 | rgw crypt vault token file = /run/.rgw-vault-token |
383 | rgw crypt vault addr = https://vault-server-fqdn:8200 | |
9f95a23c TL |
384 | |
385 | Or:: | |
386 | ||
387 | rgw crypt vault auth = agent | |
388 | rgw crypt vault addr = http://localhost:8100 | |
389 | ||
390 | Choose the secrets engine:: | |
391 | ||
392 | rgw crypt vault secret engine = kv | |
393 | ||
394 | Or:: | |
395 | ||
396 | rgw crypt vault secret engine = transit | |
397 | ||
398 | Optionally, set the Vault namespace where encryption keys will be fetched from:: | |
399 | ||
400 | rgw crypt vault namespace = tenant1 | |
401 | ||
402 | Finally, the URLs where the Gateway will retrieve encryption keys from Vault can | |
403 | be restricted by setting a path prefix. For instance, the Gateway can be | |
404 | restricted to fetch KV keys as follows:: | |
405 | ||
406 | rgw crypt vault prefix = /v1/secret/data | |
407 | ||
f67539c2 | 408 | Or, when using the transit secret engine:: |
9f95a23c | 409 | |
f67539c2 | 410 | rgw crypt vault prefix = /v1/transit |
9f95a23c TL |
411 | |
412 | In the example above, the Gateway would only fetch transit encryption keys under | |
f67539c2 TL |
413 | ``https://vault-server:8200/v1/transit``. |
414 | ||
522d829b TL |
415 | You can use custom ssl certs to authenticate with vault with help of |
416 | following options:: | |
417 | ||
418 | rgw crypt vault verify ssl = true | |
419 | rgw crypt vault ssl cacert = /etc/ceph/vault.ca | |
420 | rgw crypt vault ssl clientcert = /etc/ceph/vault.crt | |
421 | rgw crypt vault ssl clientkey = /etc/ceph/vault.key | |
422 | ||
423 | where vault.ca is CA certificate and vault.key/vault.crt are private key and ssl | |
20effc67 | 424 | certificate generated for RGW to access the vault server. It highly recommended to |
522d829b | 425 | set this option true, setting false is very dangerous and need to avoid since this |
20effc67 | 426 | runs in very secured environments. |
522d829b | 427 | |
f67539c2 TL |
428 | Transit engine compatibility support |
429 | ------------------------------------ | |
430 | The transit engine has compatibility support for previous | |
431 | versions of ceph, which used the transit engine as a simple key store. | |
432 | ||
20effc67 | 433 | There is a "compat" option which can be given to the transit |
f67539c2 TL |
434 | engine to configure the compatibility support, |
435 | ||
436 | To entirely disable backwards support, use:: | |
437 | ||
438 | rgw crypt vault secret engine = transit compat=0 | |
439 | ||
20effc67 | 440 | This will be the default in future versions. and is safe to use |
f67539c2 TL |
441 | for new installs using the current version. |
442 | ||
443 | This is the normal default with the current version:: | |
444 | ||
445 | rgw crypt vault secret engine = transit compat=1 | |
446 | ||
447 | This enables the new engine for newly created objects, | |
448 | but still allows the old engine to be used for old objects. | |
449 | In order to access old and new objects, the vault token given | |
450 | to ceph must have both the old and new transit policies. | |
451 | ||
452 | To force use of only the old engine, use:: | |
453 | ||
454 | rgw crypt vault secret engine = transit compat=2 | |
455 | ||
456 | This mode is automatically selected if the vault prefix | |
457 | ends in export/encryption-key, which was the previously | |
458 | documented setting. | |
9f95a23c TL |
459 | |
460 | Upload object | |
461 | ============= | |
462 | ||
463 | When uploading an object to the Gateway, provide the SSE key ID in the request. | |
f67539c2 | 464 | As an example, for the kv engine, using the AWS command-line client:: |
9f95a23c TL |
465 | |
466 | aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id myproject/mybucketkey | |
f67539c2 TL |
467 | |
468 | As an example, for the transit engine (new flavor), using the AWS command-line client:: | |
469 | ||
470 | aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id mybucketkey | |
9f95a23c TL |
471 | |
472 | The Object Gateway will fetch the key from Vault, encrypt the object and store | |
473 | it in the bucket. Any request to download the object will make the Gateway | |
474 | automatically retrieve the correspondent key from Vault and decrypt the object. | |
475 | ||
476 | Note that the secret will be fetched from Vault using a URL constructed by | |
477 | concatenating the base address (``rgw crypt vault addr``), the (optional) | |
f67539c2 TL |
478 | URL prefix (``rgw crypt vault prefix``), and finally the key ID. |
479 | ||
480 | In the kv engine example above, the Gateway would fetch the secret from:: | |
9f95a23c TL |
481 | |
482 | http://vaultserver:8200/v1/secret/data/myproject/mybucketkey | |
483 | ||
f67539c2 TL |
484 | In the transit engine example above, the Gateway would encrypt the secret using this key:: |
485 | ||
486 | http://vaultserver:8200/v1/transit/mybucketkey | |
487 | ||
9f95a23c TL |
488 | .. _Server-Side Encryption: ../encryption |
489 | .. _Vault: https://www.vaultproject.io/docs/ | |
490 | .. _Token authentication method: https://www.vaultproject.io/docs/auth/token.html | |
491 | .. _Vault agent: https://www.vaultproject.io/docs/agent/index.html | |
492 | .. _KV Secrets engine: https://www.vaultproject.io/docs/secrets/kv/ | |
493 | .. _Transit engine: https://www.vaultproject.io/docs/secrets/transit | |
494 | .. _namespaces: https://www.vaultproject.io/docs/enterprise/namespaces/index.html |