The Proxmox Cluster file system (``pmxcfs'') is a database-driven file
system for storing configuration files, replicated in real time to all
-cluster nodes using `corosync`. We use this to store all PVE related
+cluster nodes using `corosync`. We use this to store all {PVE} related
configuration files.
-Although the file system stores all data inside a persistent database
-on disk, a copy of the data resides in RAM. This imposes restrictions
-on the maximum size, which is currently 30MB. This is still enough to
-store the configuration of several thousand virtual machines.
+Although the file system stores all data inside a persistent database on disk,
+a copy of the data resides in RAM. This imposes restrictions on the maximum
+size, which is currently 128 MiB. This is still enough to store the
+configuration of several thousand virtual machines.
This system provides the following advantages:
[width="100%",cols="m,d"]
|=======
-|`corosync.conf` | Corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)
-|`storage.cfg` | {pve} storage configuration
-|`datacenter.cfg` | {pve} datacenter wide configuration (keyboard layout, proxy, ...)
-|`user.cfg` | {pve} access control configuration (users/groups/...)
+|`authkey.pub` | Public key used by the ticket system
+|`ceph.conf` | Ceph configuration file (note: /etc/ceph/ceph.conf is a symbolic link to this)
+|`corosync.conf` | Corosync cluster configuration file (prior to {pve} 4.x, this file was called cluster.conf)
+|`datacenter.cfg` | {pve} datacenter-wide configuration (keyboard layout, proxy, ...)
|`domains.cfg` | {pve} authentication domains
-|`status.cfg` | {pve} external metrics server configuration
-|`authkey.pub` | Public key used by ticket system
-|`pve-root-ca.pem` | Public certificate of cluster CA
-|`priv/shadow.cfg` | Shadow password file
-|`priv/authkey.key` | Private key used by ticket system
-|`priv/pve-root-ca.key` | Private key of cluster CA
-|`nodes/<NAME>/pve-ssl.pem` | Public SSL certificate for web server (signed by cluster CA)
+|`firewall/cluster.fw` | Firewall configuration applied to all nodes
+|`firewall/<NAME>.fw` | Firewall configuration for individual nodes
+|`firewall/<VMID>.fw` | Firewall configuration for VMs and containers
+|`ha/crm_commands` | Displays HA operations that are currently being carried out by the CRM
+|`ha/manager_status` | JSON-formatted information regarding HA services on the cluster
+|`ha/resources.cfg` | Resources managed by high availability, and their current state
+|`nodes/<NAME>/config` | Node-specific configuration
+|`nodes/<NAME>/lxc/<VMID>.conf` | VM configuration data for LXC containers
+|`nodes/<NAME>/openvz/` | Prior to {pve} 4.0, used for container configuration data (deprecated, removed soon)
|`nodes/<NAME>/pve-ssl.key` | Private SSL key for `pve-ssl.pem`
-|`nodes/<NAME>/pveproxy-ssl.pem` | Public SSL certificate (chain) for web server (optional override for `pve-ssl.pem`)
+|`nodes/<NAME>/pve-ssl.pem` | Public SSL certificate for web server (signed by cluster CA)
|`nodes/<NAME>/pveproxy-ssl.key` | Private SSL key for `pveproxy-ssl.pem` (optional)
+|`nodes/<NAME>/pveproxy-ssl.pem` | Public SSL certificate (chain) for web server (optional override for `pve-ssl.pem`)
|`nodes/<NAME>/qemu-server/<VMID>.conf` | VM configuration data for KVM VMs
-|`nodes/<NAME>/lxc/<VMID>.conf` | VM configuration data for LXC containers
-|`firewall/cluster.fw` | Firewall configuration applied to all nodes
-|`firewall/<NAME>.fw` | Firewall configuration for individual nodes
-|`firewall/<VMID>.fw` | Firewall configuration for VMs and Containers
+|`priv/authkey.key` | Private key used by ticket system
+|`priv/authorized_keys` | SSH keys of cluster members for authentication
+|`priv/ceph*` | Ceph authentication keys and associated capabilities
+|`priv/known_hosts` | SSH keys of the cluster members for verification
+|`priv/lock/*` | Lock files used by various services to ensure safe cluster-wide operations
+|`priv/pve-root-ca.key` | Private key of cluster CA
+|`priv/shadow.cfg` | Shadow password file for PVE Realm users
+|`priv/storage/<STORAGE-ID>.pw` | Contains the password of a storage in plain text
+|`priv/tfa.cfg` | Base64-encoded two-factor authentication configuration
+|`priv/token.cfg` | API token secrets of all tokens
+|`pve-root-ca.pem` | Public certificate of cluster CA
+|`pve-www.key` | Private key used for generating CSRF tokens
+|`sdn/*` | Shared configuration files for Software Defined Networking (SDN)
+|`status.cfg` | {pve} external metrics server configuration
+|`storage.cfg` | {pve} storage configuration
+|`user.cfg` | {pve} access control configuration (users/groups/...)
+|`virtual-guest/cpu-models.conf` | For storing custom CPU models
+|`vzdump.cron` | Cluster-wide vzdump backup-job schedule
|=======
Symbolic links
~~~~~~~~~~~~~~
+Certain directories within the cluster file system use symbolic links, in order
+to point to a node's own configuration files. Thus, the files pointed to in the
+table below refer to different files on each node of the cluster.
+
[width="100%",cols="m,m"]
|=======
|`local` | `nodes/<LOCAL_HOST_NAME>`
-|`qemu-server` | `nodes/<LOCAL_HOST_NAME>/qemu-server/`
|`lxc` | `nodes/<LOCAL_HOST_NAME>/lxc/`
+|`openvz` | `nodes/<LOCAL_HOST_NAME>/openvz/` (deprecated, removed soon)
+|`qemu-server` | `nodes/<LOCAL_HOST_NAME>/qemu-server/`
|=======
node `node2` works by running the following command as root on any member node
of the cluster:
- mv /etc/pve/nodes/node1/qemu-server/100.conf /etc/pve/nodes/node2/
+ mv /etc/pve/nodes/node1/qemu-server/100.conf /etc/pve/nodes/node2/qemu-server/
WARNING: Before manually recovering a guest like this, make absolutely sure
that the failed source node is really powered off/fenced. Otherwise {pve}'s