33b8e3e97c8b9a604102f0a4dc2790b0c6f7dbeb
[pve-docs.git] / pmxcfs.adoc
1 ifdef::manvolnum[]
2 PVE({manvolnum})
3 ================
4 include::attributes.txt[]
5
6 NAME
7 ----
8
9 pmxcfs - Proxmox Cluster File System
10
11 SYNOPSYS
12 --------
13
14 include::pmxcfs.8-cli.adoc[]
15
16 DESCRIPTION
17 -----------
18 endif::manvolnum[]
19
20 ifndef::manvolnum[]
21 Proxmox Cluster File System (pmxcfs)
22 ====================================
23 include::attributes.txt[]
24 endif::manvolnum[]
25
26 The Proxmox Cluster file system (pmxcfs) is a database-driven file
27 system for storing configuration files, replicated in real time to all
28 cluster nodes using corosync. We use this to store all PVE related
29 configuration files.
30
31 Although the file system stores all data inside a persistent database
32 on disk, a copy of the data resides in RAM. That imposes restriction
33 on the maximal size, which is currently 30MB. This is still enough to
34 store the configuration of several thousand virtual machines.
35
36 This system provides the following advantages:
37
38 * seamless replication of all configuration to all nodes in real time
39 * provides strong consistency checks to avoid duplicate VM IDs
40 * read-only when a node loses quorum
41 * automatic updates of the corosync cluster configuration to all nodes
42 * includes a distributed locking mechanism
43
44 POSIX Compatibility
45 -------------------
46
47 The file system is based on FUSE, so the behavior is POSIX like. But
48 some feature are simply not implemented, because we do not need them:
49
50 * you can just generate normal files and directories, but no symbolic
51 links, ...
52
53 * you can't rename non-empty directories (because this makes it easier
54 to guarantee that VMIDs are unique).
55
56 * you can't change file permissions (permissions are based on path)
57
58 * `O_EXCL` creates were not atomic (like old NFS)
59
60 * `O_TRUNC` creates are not atomic (FUSE restriction)
61
62
63 File access rights
64 ------------------
65
66 All files and directories are owned by user 'root' and have group
67 'www-data'. Only root has write permissions, but group 'www-data' can
68 read most files. Files below the following paths:
69
70 /etc/pve/priv/
71 /etc/pve/nodes/${NAME}/priv/
72
73 are only accessible by root.
74
75
76 Technology
77 ----------
78
79 We use the http://www.corosync.org[Corosync Cluster Engine] for
80 cluster communication, and http://www.sqlite.org[SQlite] for the
81 database file. The filesystem is implemented in user space using
82 http://fuse.sourceforge.net[FUSE].
83
84 File system layout
85 ------------------
86
87 The file system is mounted at:
88
89 /etc/pve
90
91 Files
92 ~~~~~
93
94 [width="100%",cols="m,d"]
95 |=======
96 |corosync.conf |corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)
97 |storage.cfg |{pve} storage configuration
98 |datacenter.cfg |{pve} datacenter wide configuration (keyboard layout, proxy, ...)
99 |user.cfg |{pve} access control configuration (users/groups/...)
100 |domains.cfg |{pve} Authentication domains
101 |authkey.pub | public key used by ticket system
102 |pve-root-ca.pem | public certificate of cluster CA
103 |priv/shadow.cfg | shadow password file
104 |priv/authkey.key | private key used by ticket system
105 |priv/pve-root-ca.key | private key of cluster CA
106 |nodes/<NAME>/pve-ssl.pem | public ssl certificate for web server (signed by cluster CA)
107 |nodes/<NAME>/pve-ssl.key | private ssl key for pve-ssl.pem
108 |nodes/<NAME>/pveproxy-ssl.pem | public ssl certificate (chain) for web server (optional override for pve-ssl.pem)
109 |nodes/<NAME>/pveproxy-ssl.key | private ssl key for pveproxy-ssl.pem (optional)
110 |nodes/<NAME>/qemu-server/<VMID>.conf | VM configuration data for KVM VMs
111 |nodes/<NAME>/lxc/<VMID>.conf | VM configuration data for LXC containers
112 |firewall/cluster.fw | Firewall config applied to all nodes
113 |firewall/<NAME>.fw | Firewall config for individual nodes
114 |firewall/<VMID>.fw | Firewall config for VMs and Containers
115 |=======
116
117 Symbolic links
118 ~~~~~~~~~~~~~~
119
120 [width="100%",cols="m,m"]
121 |=======
122 |local |nodes/<LOCAL_HOST_NAME>
123 |qemu-server |nodes/<LOCAL_HOST_NAME>/qemu-server/
124 |lxc |nodes/<LOCAL_HOST_NAME>/lxc/
125 |=======
126
127 Special status files for debugging (JSON)
128 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
129
130 [width="100%",cols="m,d"]
131 |=======
132 | .version |file versions (to detect file modifications)
133 | .members |Info about cluster members
134 | .vmlist |List of all VMs
135 | .clusterlog |Cluster log (last 50 entries)
136 | .rrd |RRD data (most recent entries)
137 |=======
138
139 Enable/Disable debugging
140 ~~~~~~~~~~~~~~~~~~~~~~~~
141
142 You can enable verbose syslog messages with:
143
144 echo "1" >/etc/pve/.debug
145
146 And disable verbose syslog messages with:
147
148 echo "0" >/etc/pve/.debug
149
150
151 Recovery
152 --------
153
154 If you have major problems with your Proxmox VE host, e.g. hardware
155 issues, it could be helpful to just copy the pmxcfs database file
156 /var/lib/pve-cluster/config.db and move it to a new Proxmox VE
157 host. On the new host (with nothing running), you need to stop the
158 pve-cluster service and replace the config.db file (needed permissions
159 0600). Second, adapt '/etc/hostname' and '/etc/hosts' according to the
160 lost Proxmox VE host, then reboot and check. (And donĀ“t forget your
161 VM/CT data)
162
163 Remove Cluster configuration
164 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
165
166 The recommended way is to reinstall the node after you removed it from
167 your cluster. This makes sure that all secret cluster/ssh keys and any
168 shared configuration data is destroyed.
169
170 In some cases, you might prefer to put a node back to local mode
171 without reinstall, which is described here:
172
173 * stop the cluster file system in '/etc/pve/'
174
175 # systemctl stop pve-cluster
176
177 * start it again but forcing local mode
178
179 # pmxcfs -l
180
181 * remove the cluster config
182
183 # rm /etc/pve/cluster.conf
184 # rm /etc/cluster/cluster.conf
185 # rm /var/lib/pve-cluster/corosync.authkey
186
187 * stop the cluster file system again
188
189 # systemctl stop pve-cluster
190
191 * restart pve services (or reboot)
192
193 # systemctl start pve-cluster
194 # systemctl restart pvedaemon
195 # systemctl restart pveproxy
196 # systemctl restart pvestatd
197
198
199 ifdef::manvolnum[]
200 include::pve-copyright.adoc[]
201 endif::manvolnum[]