1 Proxmox Cluster file system (pmxcfs)
2 ====================================
4 The Proxmox Cluster file system (pmxcfs) is a database-driven file
5 system for storing configuration files, replicated in real time to all
6 cluster nodes using corosync. We use this to store all PVE related
9 Although the file system stores all data inside a persistent database
10 on disk, a copy of the data resides in RAM. That imposes restriction
11 on the maximal size, which is currently 30MB. This is still enough to
12 store the configuration of several thousand virtual machines.
17 * seamless replication of all configuration to all nodes in real time
18 * provides strong consistency checks to avoid duplicate VM IDs
19 * read-only when a node loses quorum
20 * automatic updates of the corosync cluster configuration to all nodes
21 * includes a distributed locking mechanism
26 The file system is based on FUSE, so the behavior is POSIX like. But
27 some feature are simply not implemented, because we do not need them:
29 * you can just generate normal files and directories, but no symbolic
32 * you can't rename non-empty directories (because this makes it easier
33 to guarantee that VMIDs are unique).
35 * you can't change file permissions (permissions are based on path)
37 * `O_EXCL` creates were not atomic (like old NFS)
39 * `O_TRUNC` creates are not atomic (FUSE restriction)
45 All files and directories are owned by user 'root' and have group
46 'www-data'. Only root has write permissions, but group 'www-data' can
47 read most files. Files below the following paths:
50 /etc/pve/nodes/${NAME}/priv/
52 are only accessible by root.
57 We use the http://www.corosync.org[Corosync Cluster Engine] for
58 cluster communication, and http://www.sqlite.org[SQlite] for the
59 database file. The filesystem is implemented in user space using
60 http://fuse.sourceforge.net[FUSE].
65 The file system is mounted at:
72 [width="100%",cols="m,d"]
74 |corosync.conf |corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)
75 |storage.cfg |{pve} storage configuration
76 |datacenter.cfg |{pve} datacenter wide configuration (keyboard layout, proxy, ...)
77 |user.cfg |{pve} access control configuration (users/groups/...)
78 |domains.cfg |{pve} Authentication domains
79 |authkey.pub | public key used by ticket system
80 |pve-root-ca.pem | public certificate of cluster CA
81 |priv/shadow.cfg | shadow password file
82 |priv/authkey.key | private key used by ticket system
83 |priv/pve-root-ca.key | private key of cluster CA
84 |nodes/<NAME>/pve-ssl.pem | public ssl certificate for web server (signed by cluster CA)
85 |nodes/<NAME>/pve-ssl.key | private ssl key for pve-ssl.pem
86 |nodes/<NAME>/pveproxy-ssl.pem | public ssl certificate (chain) for web server (optional override for pve-ssl.pem)
87 |nodes/<NAME>/pveproxy-ssl.key | private ssl key for pveproxy-ssl.pem (optional)
88 |nodes/<NAME>/qemu-server/<VMID>.conf | VM configuration data for KVM VMs
89 |nodes/<NAME>/lxc/<VMID>.conf | VM configuration data for LXC containers
90 |firewall/cluster.fw | Firewall config applied to all nodes
91 |firewall/<NAME>.fw | Firewall config for individual nodes
92 |firewall/<VMID>.fw | Firewall config for VMs and Containers
98 [width="100%",cols="m,m"]
100 |local |nodes/<LOCAL_HOST_NAME>
101 |qemu-server |nodes/<LOCAL_HOST_NAME>/qemu-server/
102 |lxc |nodes/<LOCAL_HOST_NAME>/lxc/
105 Special status files for debugging (JSON)
106 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
108 [width="100%",cols="m,d"]
110 | .version |file versions (to detect file modifications)
111 | .members |Info about cluster members
112 | .vmlist |List of all VMs
113 | .clusterlog |Cluster log (last 50 entries)
114 | .rrd |RRD data (most recent entries)
117 Enable/Disable debugging
118 ~~~~~~~~~~~~~~~~~~~~~~~~
120 You can enable verbose syslog messages with:
122 echo "1" >/etc/pve/.debug
124 And disable verbose syslog messages with:
126 echo "0" >/etc/pve/.debug
132 If you have major problems with your Proxmox VE host, e.g. hardware
133 issues, it could be helpful to just copy the pmxcfs database file
134 /var/lib/pve-cluster/config.db and move it to a new Proxmox VE
135 host. On the new host (with nothing running), you need to stop the
136 pve-cluster service and replace the config.db file (needed permissions
137 0600). Second, adapt '/etc/hostname' and '/etc/hosts' according to the
138 lost Proxmox VE host, then reboot and check. (And donĀ“t forget your
141 Remove Cluster configuration
142 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
144 The recommended way is to reinstall the node after you removed it from
145 your cluster. This makes sure that all secret cluster/ssh keys and any
146 shared configuration data is destroyed.
148 In some cases, you might prefer to put a node back to local mode
149 without reinstall, which is described here:
151 * stop the cluster file system in '/etc/pve/'
153 # systemctl stop pve-cluster
155 * start it again but forcing local mode
159 * remove the cluster config
161 # rm /etc/pve/cluster.conf
162 # rm /etc/cluster/cluster.conf
163 # rm /var/lib/pve-cluster/corosync.authkey
165 * stop the cluster file system again
167 # service pve-cluster stop
169 * restart pve services (or reboot)
171 # service pve-cluster start
172 # service pvedaemon restart
173 # service pveproxy restart
174 # service pvestatd restart