4 include::attributes.txt[]
11 pmxcfs - Proxmox Cluster File System
16 include::pmxcfs.8-cli.adoc[]
23 Proxmox Cluster File System (pmxcfs)
24 ====================================
25 include::attributes.txt[]
32 The Proxmox Cluster file system (``pmxcfs'') is a database-driven file
33 system for storing configuration files, replicated in real time to all
34 cluster nodes using `corosync`. We use this to store all PVE related
37 Although the file system stores all data inside a persistent database
38 on disk, a copy of the data resides in RAM. That imposes restriction
39 on the maximum size, which is currently 30MB. This is still enough to
40 store the configuration of several thousand virtual machines.
42 This system provides the following advantages:
44 * seamless replication of all configuration to all nodes in real time
45 * provides strong consistency checks to avoid duplicate VM IDs
46 * read-only when a node loses quorum
47 * automatic updates of the corosync cluster configuration to all nodes
48 * includes a distributed locking mechanism
54 The file system is based on FUSE, so the behavior is POSIX like. But
55 some feature are simply not implemented, because we do not need them:
57 * you can just generate normal files and directories, but no symbolic
60 * you can't rename non-empty directories (because this makes it easier
61 to guarantee that VMIDs are unique).
63 * you can't change file permissions (permissions are based on path)
65 * `O_EXCL` creates were not atomic (like old NFS)
67 * `O_TRUNC` creates are not atomic (FUSE restriction)
73 All files and directories are owned by user `root` and have group
74 `www-data`. Only root has write permissions, but group `www-data` can
75 read most files. Files below the following paths:
78 /etc/pve/nodes/${NAME}/priv/
80 are only accessible by root.
86 We use the http://www.corosync.org[Corosync Cluster Engine] for
87 cluster communication, and http://www.sqlite.org[SQlite] for the
88 database file. The file system is implemented in user space using
89 http://fuse.sourceforge.net[FUSE].
94 The file system is mounted at:
101 [width="100%",cols="m,d"]
103 |`corosync.conf` | Corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)
104 |`storage.cfg` | {pve} storage configuration
105 |`datacenter.cfg` | {pve} datacenter wide configuration (keyboard layout, proxy, ...)
106 |`user.cfg` | {pve} access control configuration (users/groups/...)
107 |`domains.cfg` | {pve} authentication domains
108 |`authkey.pub` | Public key used by ticket system
109 |`pve-root-ca.pem` | Public certificate of cluster CA
110 |`priv/shadow.cfg` | Shadow password file
111 |`priv/authkey.key` | Private key used by ticket system
112 |`priv/pve-root-ca.key` | Private key of cluster CA
113 |`nodes/<NAME>/pve-ssl.pem` | Public SSL certificate for web server (signed by cluster CA)
114 |`nodes/<NAME>/pve-ssl.key` | Private SSL key for `pve-ssl.pem`
115 |`nodes/<NAME>/pveproxy-ssl.pem` | Public SSL certificate (chain) for web server (optional override for `pve-ssl.pem`)
116 |`nodes/<NAME>/pveproxy-ssl.key` | Private SSL key for `pveproxy-ssl.pem` (optional)
117 |`nodes/<NAME>/qemu-server/<VMID>.conf` | VM configuration data for KVM VMs
118 |`nodes/<NAME>/lxc/<VMID>.conf` | VM configuration data for LXC containers
119 |`firewall/cluster.fw` | Firewall configuration applied to all nodes
120 |`firewall/<NAME>.fw` | Firewall configuration for individual nodes
121 |`firewall/<VMID>.fw` | Firewall configuration for VMs and Containers
128 [width="100%",cols="m,m"]
130 |`local` | `nodes/<LOCAL_HOST_NAME>`
131 |`qemu-server` | `nodes/<LOCAL_HOST_NAME>/qemu-server/`
132 |`lxc` | `nodes/<LOCAL_HOST_NAME>/lxc/`
136 Special status files for debugging (JSON)
137 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
139 [width="100%",cols="m,d"]
141 |`.version` |File versions (to detect file modifications)
142 |`.members` |Info about cluster members
143 |`.vmlist` |List of all VMs
144 |`.clusterlog` |Cluster log (last 50 entries)
145 |`.rrd` |RRD data (most recent entries)
149 Enable/Disable debugging
150 ~~~~~~~~~~~~~~~~~~~~~~~~
152 You can enable verbose syslog messages with:
154 echo "1" >/etc/pve/.debug
156 And disable verbose syslog messages with:
158 echo "0" >/etc/pve/.debug
164 If you have major problems with your Proxmox VE host, e.g. hardware
165 issues, it could be helpful to just copy the pmxcfs database file
166 `/var/lib/pve-cluster/config.db` and move it to a new Proxmox VE
167 host. On the new host (with nothing running), you need to stop the
168 `pve-cluster` service and replace the `config.db` file (needed permissions
169 `0600`). Second, adapt `/etc/hostname` and `/etc/hosts` according to the
170 lost Proxmox VE host, then reboot and check. (And don't forget your
174 Remove Cluster configuration
175 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
177 The recommended way is to reinstall the node after you removed it from
178 your cluster. This makes sure that all secret cluster/ssh keys and any
179 shared configuration data is destroyed.
181 In some cases, you might prefer to put a node back to local mode
182 without reinstall, which is described here:
184 * stop the cluster file system in `/etc/pve/`
186 # systemctl stop pve-cluster
188 * start it again but forcing local mode
192 * remove the cluster configuration
194 # rm /etc/pve/cluster.conf
195 # rm /etc/cluster/cluster.conf
196 # rm /var/lib/pve-cluster/corosync.authkey
198 * stop the cluster file system again
200 # systemctl stop pve-cluster
202 * restart PVE services (or reboot)
204 # systemctl start pve-cluster
205 # systemctl restart pvedaemon
206 # systemctl restart pveproxy
207 # systemctl restart pvestatd
211 include::pve-copyright.adoc[]