[[sysadmin_certificate_management]] Certificate Management ---------------------- ifdef::wiki[] :pve-toplevel: endif::wiki[] Certificates for communication within the cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each {PVE} cluster creates its own (self-signed) Certificate Authority (CA) and generates a certificate for each node which gets signed by the aforementioned CA. These certificates are used for encrypted communication with the cluster's `pveproxy` service and the Shell/Console feature if SPICE is used. The CA certificate and key are stored in the xref:chapter_pmxcfs[Proxmox Cluster File System (pmxcfs)]. Certificates for API and web GUI ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The REST API and web GUI are provided by the `pveproxy` service, which runs on each node. You have the following options for the certificate used by `pveproxy`: 1. By default the node-specific certificate in `/etc/pve/nodes/NODENAME/pve-ssl.pem` is used. This certificate is signed by the cluster CA and therefore not automatically trusted by browsers and operating systems. 2. use an externally provided certificate (e.g. signed by a commercial CA). 3. use ACME (Let's Encrypt) to get a trusted certificate with automatic renewal, this is also integrated in the {pve} API and Webinterface. For options 2 and 3 the file `/etc/pve/local/pveproxy-ssl.pem` (and `/etc/pve/local/pveproxy-ssl.key`, which needs to be without password) is used. NOTE: Keep in mind that `/etc/pve/local` is a node specific symlink to `/etc/pve/nodes/NODENAME`. Certificates are managed with the {PVE} Node management command (see the `pvenode(1)` manpage). WARNING: Do not replace or manually modify the automatically generated node certificate files in `/etc/pve/local/pve-ssl.pem` and `/etc/pve/local/pve-ssl.key` or the cluster CA files in `/etc/pve/pve-root-ca.pem` and `/etc/pve/priv/pve-root-ca.key`. Getting trusted certificates via ACME ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {PVE} includes an implementation of the **A**utomatic **C**ertificate **M**anagement **E**nvironment **ACME** protocol, allowing {pve} admins to interface with Let's Encrypt for easy setup of trusted TLS certificates which are accepted out of the box on most modern operating systems and browsers. Currently the two ACME endpoints implemented are Let's Encrypt (LE) and its staging environment (see https://letsencrypt.org). Our ACME client supports validation of `http-01` challenges using a built-in webserver and validation of `dns-01` challenges using a DNS plugin. Because of https://letsencrypt.org/docs/rate-limits/[rate-limits] you should use LE `staging` for experiments. ACME Plugin configuration is stored in `/etc/pve/priv/acme/plugins.cfg`. There is always an implicitly configured `standalone` plugin for validating `http-01` challenges via the built-in webserver spawned on port 80. There are a few prerequisites to use Let's Encrypt: * You have to accept the ToS of Let's Encrypt to register an account. For `http-01` challenges: * **Port 80** of the node needs to be reachable from the internet. * There **must** be no other listener on port 80. * The requested (sub)domain needs to resolve to a public IP of the Node. For `dns-01` challenges: * DNS needs to be handled by a server that allows automatic provisioning of TXT records At the moment the GUI uses only the default ACME account. .Example: Sample `pvenode` invocation for using Let's Encrypt certificates ---- root@proxmox:~# pvenode acme account register default mail@example.invalid Directory endpoints: 0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory) 1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory) 2) Custom Enter selection: 1 Attempting to fetch Terms of Service from 'https://acme-staging-v02.api.letsencrypt.org/directory'.. Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf Do you agree to the above terms? [y|N]y Attempting to register account with 'https://acme-staging-v02.api.letsencrypt.org/directory'.. Generating ACME account key.. Registering ACME account.. Registration successful, account URL: 'https://acme-staging-v02.api.letsencrypt.org/acme/acct/xxxxxxx' Task OK root@proxmox:~# pvenode acme account list default root@proxmox:~# pvenode config set --acme domains=example.invalid root@proxmox:~# pvenode acme cert order Loading ACME account details Placing ACME order Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/xxxxxxxxxxxxxx Getting authorization details from 'https://acme-staging-v02.api.letsencrypt.org/acme/authz/xxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxx-xxxxxxx' ... pending! Setting up webserver Triggering validation Sleeping for 5 seconds Status is 'valid'! All domains validated! Creating CSR Finalizing order Checking order status valid! Downloading certificate Setting pveproxy certificate and key Restarting pveproxy Task OK ---- Switching from the `staging` to the regular ACME directory ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Changing the ACME directory for an account is unsupported. If you want to switch an account from the `staging` ACME directory to the regular, trusted, one you need to deactivate it and recreate it. This procedure is also needed to change the default ACME account used in the GUI. .Example: Changing the `default` ACME account from the `staging` to the regular directory ---- root@proxmox:~# pvenode acme account info default Directory URL: https://acme-staging-v02.api.letsencrypt.org/directory Account URL: https://acme-staging-v02.api.letsencrypt.org/acme/acct/6332194 Terms Of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf Account information: ID: xxxxxxx Contact: - mailto:example@proxmox.com Creation date: 2018-07-31T08:41:44.54196435Z Initial IP: 192.0.2.1 Status: valid root@proxmox:~# pvenode acme account deactivate default Renaming account file from '/etc/pve/priv/acme/default' to '/etc/pve/priv/acme/_deactivated_default_4' Task OK root@proxmox:~# pvenode acme account register default example@proxmox.com Directory endpoints: 0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory) 1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory) 2) Custom Enter selection: 0 Attempting to fetch Terms of Service from 'https://acme-v02.api.letsencrypt.org/directory'.. Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf Do you agree to the above terms? [y|N]y Attempting to register account with 'https://acme-v02.api.letsencrypt.org/directory'.. Generating ACME account key.. Registering ACME account.. Registration successful, account URL: 'https://acme-v02.api.letsencrypt.org/acme/acct/39335247' Task OK ---- Automatic renewal of ACME certificates ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If a node has been successfully configured with an ACME-provided certificate (either via pvenode or via the GUI), the certificate will be automatically renewed by the pve-daily-update.service. Currently, renewal will be attempted if the certificate has expired already, or will expire in the next 30 days. Configuring ACME DNS APIs for validation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On systems where external access for validation via the `http-01` method is not possible or desired, it is possible to use the `dns-01` validation method. This validation method requires a DNS server that allows provisioning of `TXT` records via an API. {PVE} re-uses the DNS plugins developed for the `acme.sh` footnote:[acme.sh https://github.com/acmesh-official/acme.sh] project, please refer to its documentation for details on configuration of specific APIs. Combining `http-01` and `dns-01` validation is possible in case your node is reachable via multiple domains with different requirements / DNS provisioning capabilities. Mixing DNS APIs from multiple providers or instances is also possible by specifying different plugin instances per domain. A special `alias` mode can be used to handle the validation on a different domain/DNS server, in case your primary/real DNS does not support provisioning via an API. Manually set up a permanent `CNAME` record for `_acme-challenge.domain1.example` pointing to `_acme-challenge.domain2.example` and set the `alias` property in the {PVE} node configuration file to `domain2.example` to allow the DNS server of `domain2.example` to validate all challenges for `domain1.example`. .Example: Setting up the OVH API for validating a domain NOTE: the account registration steps are the same no matter which plugins are used, and are not repeated here. NOTE: `OVH_AK` and `OVH_AS` need to be obtained from OVH according to the OVH API documentation First you need to get all information so you and {pve} can access the API. ---- root@proxmox:~# cat /path/to/api-token OVH_AK=XXXXXXXXXXXXXXXX OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY root@proxmox:~# source /path/to/api-token root@proxmox:~# curl -XPOST -H"X-Ovh-Application: $OVH_AK" -H "Content-type: application/json" \ https://eu.api.ovh.com/1.0/auth/credential -d '{ "accessRules": [ {"method": "GET","path": "/auth/time"}, {"method": "GET","path": "/domain"}, {"method": "GET","path": "/domain/zone/*"}, {"method": "GET","path": "/domain/zone/*/record"}, {"method": "POST","path": "/domain/zone/*/record"}, {"method": "POST","path": "/domain/zone/*/refresh"}, {"method": "PUT","path": "/domain/zone/*/record/"}, {"method": "DELETE","path": "/domain/zone/*/record/*"} ] }' {"consumerKey":"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ","state":"pendingValidation","validationUrl":"https://eu.api.ovh.com/auth/?credentialToken=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"} (open validation URL and follow instructions to link Application Key with account/Consumer Key) root@proxmox:~# echo "OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" >> /path/to/api-token ---- Now you can setup the the ACME plugin: ---- root@proxmox:~# pvenode acme plugin add dns example_plugin --api ovh --data /path/to/api_token root@proxmox:~# pvenode acme plugin config example_plugin ┌────────┬──────────────────────────────────────────┐ │ key │ value │ ╞════════╪══════════════════════════════════════════╡ │ api │ ovh │ ├────────┼──────────────────────────────────────────┤ │ data │ OVH_AK=XXXXXXXXXXXXXXXX │ │ │ OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY │ │ │ OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ │ ├────────┼──────────────────────────────────────────┤ │ digest │ 867fcf556363ca1bea866863093fcab83edf47a1 │ ├────────┼──────────────────────────────────────────┤ │ plugin │ example_plugin │ ├────────┼──────────────────────────────────────────┤ │ type │ dns │ └────────┴──────────────────────────────────────────┘ ---- At last you can configure the domain you want to get certitficates for and place the certificate order for it: ---- root@proxmox:~# pvenode config set -acmedomain0 example.proxmox.com,plugin=example_plugin root@proxmox:~# pvenode acme cert order Loading ACME account details Placing ACME order Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/11111111/22222222 Getting authorization details from 'https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/33333333' The validation for example.proxmox.com is pending! [Wed Apr 22 09:25:30 CEST 2020] Using OVH endpoint: ovh-eu [Wed Apr 22 09:25:30 CEST 2020] Checking authentication [Wed Apr 22 09:25:30 CEST 2020] Consumer key is ok. [Wed Apr 22 09:25:31 CEST 2020] Adding record [Wed Apr 22 09:25:32 CEST 2020] Added, sleep 10 seconds. Add TXT record: _acme-challenge.example.proxmox.com Triggering validation Sleeping for 5 seconds Status is 'valid'! [Wed Apr 22 09:25:48 CEST 2020] Using OVH endpoint: ovh-eu [Wed Apr 22 09:25:48 CEST 2020] Checking authentication [Wed Apr 22 09:25:48 CEST 2020] Consumer key is ok. Remove TXT record: _acme-challenge.example.proxmox.com All domains validated! Creating CSR Checking order status Order is ready, finalizing order valid! Downloading certificate Setting pveproxy certificate and key Restarting pveproxy Task OK ----