]> git.proxmox.com Git - pve-docs.git/blob - pmxcfs.adoc
use asciidoc-pve whenever possible
[pve-docs.git] / pmxcfs.adoc
1 ifdef::manvolnum[]
2 PVE(8)
3 ======
4 include::attributes.txt[]
5 :pve-toplevel:
6
7 NAME
8 ----
9
10 pmxcfs - Proxmox Cluster File System
11
12 SYNOPSIS
13 --------
14
15 include::pmxcfs.8-cli.adoc[]
16
17 DESCRIPTION
18 -----------
19 endif::manvolnum[]
20
21 ifndef::manvolnum[]
22 Proxmox Cluster File System (pmxcfs)
23 ====================================
24 include::attributes.txt[]
25 endif::manvolnum[]
26
27 ifdef::wiki[]
28 :pve-toplevel:
29 endif::wiki[]
30
31 The Proxmox Cluster file system (``pmxcfs'') is a database-driven file
32 system for storing configuration files, replicated in real time to all
33 cluster nodes using `corosync`. We use this to store all PVE related
34 configuration files.
35
36 Although the file system stores all data inside a persistent database
37 on disk, a copy of the data resides in RAM. That imposes restriction
38 on the maximum size, which is currently 30MB. This is still enough to
39 store the configuration of several thousand virtual machines.
40
41 This system provides the following advantages:
42
43 * seamless replication of all configuration to all nodes in real time
44 * provides strong consistency checks to avoid duplicate VM IDs
45 * read-only when a node loses quorum
46 * automatic updates of the corosync cluster configuration to all nodes
47 * includes a distributed locking mechanism
48
49
50 POSIX Compatibility
51 -------------------
52
53 The file system is based on FUSE, so the behavior is POSIX like. But
54 some feature are simply not implemented, because we do not need them:
55
56 * you can just generate normal files and directories, but no symbolic
57 links, ...
58
59 * you can't rename non-empty directories (because this makes it easier
60 to guarantee that VMIDs are unique).
61
62 * you can't change file permissions (permissions are based on path)
63
64 * `O_EXCL` creates were not atomic (like old NFS)
65
66 * `O_TRUNC` creates are not atomic (FUSE restriction)
67
68
69 File Access Rights
70 ------------------
71
72 All files and directories are owned by user `root` and have group
73 `www-data`. Only root has write permissions, but group `www-data` can
74 read most files. Files below the following paths:
75
76 /etc/pve/priv/
77 /etc/pve/nodes/${NAME}/priv/
78
79 are only accessible by root.
80
81
82 Technology
83 ----------
84
85 We use the http://www.corosync.org[Corosync Cluster Engine] for
86 cluster communication, and http://www.sqlite.org[SQlite] for the
87 database file. The file system is implemented in user space using
88 http://fuse.sourceforge.net[FUSE].
89
90 File System Layout
91 ------------------
92
93 The file system is mounted at:
94
95 /etc/pve
96
97 Files
98 ~~~~~
99
100 [width="100%",cols="m,d"]
101 |=======
102 |`corosync.conf` | Corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)
103 |`storage.cfg` | {pve} storage configuration
104 |`datacenter.cfg` | {pve} datacenter wide configuration (keyboard layout, proxy, ...)
105 |`user.cfg` | {pve} access control configuration (users/groups/...)
106 |`domains.cfg` | {pve} authentication domains
107 |`authkey.pub` | Public key used by ticket system
108 |`pve-root-ca.pem` | Public certificate of cluster CA
109 |`priv/shadow.cfg` | Shadow password file
110 |`priv/authkey.key` | Private key used by ticket system
111 |`priv/pve-root-ca.key` | Private key of cluster CA
112 |`nodes/<NAME>/pve-ssl.pem` | Public SSL certificate for web server (signed by cluster CA)
113 |`nodes/<NAME>/pve-ssl.key` | Private SSL key for `pve-ssl.pem`
114 |`nodes/<NAME>/pveproxy-ssl.pem` | Public SSL certificate (chain) for web server (optional override for `pve-ssl.pem`)
115 |`nodes/<NAME>/pveproxy-ssl.key` | Private SSL key for `pveproxy-ssl.pem` (optional)
116 |`nodes/<NAME>/qemu-server/<VMID>.conf` | VM configuration data for KVM VMs
117 |`nodes/<NAME>/lxc/<VMID>.conf` | VM configuration data for LXC containers
118 |`firewall/cluster.fw` | Firewall configuration applied to all nodes
119 |`firewall/<NAME>.fw` | Firewall configuration for individual nodes
120 |`firewall/<VMID>.fw` | Firewall configuration for VMs and Containers
121 |=======
122
123
124 Symbolic links
125 ~~~~~~~~~~~~~~
126
127 [width="100%",cols="m,m"]
128 |=======
129 |`local` | `nodes/<LOCAL_HOST_NAME>`
130 |`qemu-server` | `nodes/<LOCAL_HOST_NAME>/qemu-server/`
131 |`lxc` | `nodes/<LOCAL_HOST_NAME>/lxc/`
132 |=======
133
134
135 Special status files for debugging (JSON)
136 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
137
138 [width="100%",cols="m,d"]
139 |=======
140 |`.version` |File versions (to detect file modifications)
141 |`.members` |Info about cluster members
142 |`.vmlist` |List of all VMs
143 |`.clusterlog` |Cluster log (last 50 entries)
144 |`.rrd` |RRD data (most recent entries)
145 |=======
146
147
148 Enable/Disable debugging
149 ~~~~~~~~~~~~~~~~~~~~~~~~
150
151 You can enable verbose syslog messages with:
152
153 echo "1" >/etc/pve/.debug
154
155 And disable verbose syslog messages with:
156
157 echo "0" >/etc/pve/.debug
158
159
160 Recovery
161 --------
162
163 If you have major problems with your Proxmox VE host, e.g. hardware
164 issues, it could be helpful to just copy the pmxcfs database file
165 `/var/lib/pve-cluster/config.db` and move it to a new Proxmox VE
166 host. On the new host (with nothing running), you need to stop the
167 `pve-cluster` service and replace the `config.db` file (needed permissions
168 `0600`). Second, adapt `/etc/hostname` and `/etc/hosts` according to the
169 lost Proxmox VE host, then reboot and check. (And don't forget your
170 VM/CT data)
171
172
173 Remove Cluster configuration
174 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
175
176 The recommended way is to reinstall the node after you removed it from
177 your cluster. This makes sure that all secret cluster/ssh keys and any
178 shared configuration data is destroyed.
179
180 In some cases, you might prefer to put a node back to local mode
181 without reinstall, which is described here:
182
183 * stop the cluster file system in `/etc/pve/`
184
185 # systemctl stop pve-cluster
186
187 * start it again but forcing local mode
188
189 # pmxcfs -l
190
191 * remove the cluster configuration
192
193 # rm /etc/pve/cluster.conf
194 # rm /etc/cluster/cluster.conf
195 # rm /var/lib/pve-cluster/corosync.authkey
196
197 * stop the cluster file system again
198
199 # systemctl stop pve-cluster
200
201 * restart PVE services (or reboot)
202
203 # systemctl start pve-cluster
204 # systemctl restart pvedaemon
205 # systemctl restart pveproxy
206 # systemctl restart pvestatd
207
208
209 ifdef::manvolnum[]
210 include::pve-copyright.adoc[]
211 endif::manvolnum[]