pmxcfs.adoc

   1 ifdef::manvolnum[]
   2 PVE({manvolnum})
   3 ================
   4 include::attributes.txt[]
   5
   6 NAME
   7 ----
   8
   9 pmxcfs - Proxmox Cluster File System
  10
  11 SYNOPSYS
  12 --------
  13
  14 include::pmxcfs.8-cli.adoc[]
  15
  16 DESCRIPTION
  17 -----------
  18 endif::manvolnum[]
  19
  20 ifndef::manvolnum[]
  21 Proxmox Cluster File System (pmxcfs)
  22 ====================================
  23 include::attributes.txt[]
  24 endif::manvolnum[]
  25
  26 The Proxmox Cluster file system (``pmxcfs'') is a database-driven file
  27 system for storing configuration files, replicated in real time to all
  28 cluster nodes using `corosync`. We use this to store all PVE related
  29 configuration files.
  30
  31 Although the file system stores all data inside a persistent database
  32 on disk, a copy of the data resides in RAM. That imposes restriction
  33 on the maximum size, which is currently 30MB. This is still enough to
  34 store the configuration of several thousand virtual machines.
  35
  36 This system provides the following advantages:
  37
  38 * seamless replication of all configuration to all nodes in real time
  39 * provides strong consistency checks to avoid duplicate VM IDs
  40 * read-only when a node loses quorum
  41 * automatic updates of the corosync cluster configuration to all nodes
  42 * includes a distributed locking mechanism
  43
  44
  45 POSIX Compatibility
  46 -------------------
  47
  48 The file system is based on FUSE, so the behavior is POSIX like. But
  49 some feature are simply not implemented, because we do not need them:
  50
  51 * you can just generate normal files and directories, but no symbolic
  52   links, ...
  53
  54 * you can't rename non-empty directories (because this makes it easier
  55   to guarantee that VMIDs are unique).
  56
  57 * you can't change file permissions (permissions are based on path)
  58
  59 * `O_EXCL` creates were not atomic (like old NFS)
  60
  61 * `O_TRUNC` creates are not atomic (FUSE restriction)
  62
  63
  64 File Access Rights
  65 ------------------
  66
  67 All files and directories are owned by user `root` and have group
  68 `www-data`. Only root has write permissions, but group `www-data` can
  69 read most files. Files below the following paths:
  70
  71  /etc/pve/priv/
  72  /etc/pve/nodes/${NAME}/priv/
  73
  74 are only accessible by root.
  75
  76
  77 Technology
  78 ----------
  79
  80 We use the http://www.corosync.org[Corosync Cluster Engine] for
  81 cluster communication, and http://www.sqlite.org[SQlite] for the
  82 database file. The file system is implemented in user space using
  83 http://fuse.sourceforge.net[FUSE].
  84
  85 File System Layout
  86 ------------------
  87
  88 The file system is mounted at:
  89
  90  /etc/pve
  91
  92 Files
  93 ~~~~~
  94
  95 [width="100%",cols="m,d"]
  96 |=======
  97 |`corosync.conf`                        | Corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)
  98 |`storage.cfg`                          | {pve} storage configuration
  99 |`datacenter.cfg`                       | {pve} datacenter wide configuration (keyboard layout, proxy, ...)
 100 |`user.cfg`                             | {pve} access control configuration (users/groups/...)
 101 |`domains.cfg`                          | {pve} authentication domains
 102 |`authkey.pub`                          | Public key used by ticket system
 103 |`pve-root-ca.pem`                      | Public certificate of cluster CA
 104 |`priv/shadow.cfg`                      | Shadow password file
 105 |`priv/authkey.key`                     | Private key used by ticket system
 106 |`priv/pve-root-ca.key`                 | Private key of cluster CA
 107 |`nodes/<NAME>/pve-ssl.pem`             | Public SSL certificate for web server (signed by cluster CA)
 108 |`nodes/<NAME>/pve-ssl.key`             | Private SSL key for `pve-ssl.pem`
 109 |`nodes/<NAME>/pveproxy-ssl.pem`        | Public SSL certificate (chain) for web server (optional override for `pve-ssl.pem`)
 110 |`nodes/<NAME>/pveproxy-ssl.key`        | Private SSL key for `pveproxy-ssl.pem` (optional)
 111 |`nodes/<NAME>/qemu-server/<VMID>.conf` | VM configuration data for KVM VMs
 112 |`nodes/<NAME>/lxc/<VMID>.conf`         | VM configuration data for LXC containers
 113 |`firewall/cluster.fw`                  | Firewall configuration applied to all nodes
 114 |`firewall/<NAME>.fw`                   | Firewall configuration for individual nodes
 115 |`firewall/<VMID>.fw`                   | Firewall configuration for VMs and Containers
 116 |=======
 117
 118
 119 Symbolic links
 120 ~~~~~~~~~~~~~~
 121
 122 [width="100%",cols="m,m"]
 123 |=======
 124 |`local`         | `nodes/<LOCAL_HOST_NAME>`
 125 |`qemu-server`   | `nodes/<LOCAL_HOST_NAME>/qemu-server/`
 126 |`lxc`           | `nodes/<LOCAL_HOST_NAME>/lxc/`
 127 |=======
 128
 129
 130 Special status files for debugging (JSON)
 131 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 132
 133 [width="100%",cols="m,d"]
 134 |=======
 135 |`.version`    |File versions (to detect file modifications)
 136 |`.members`    |Info about cluster members
 137 |`.vmlist`     |List of all VMs
 138 |`.clusterlog` |Cluster log (last 50 entries)
 139 |`.rrd`        |RRD data (most recent entries)
 140 |=======
 141
 142
 143 Enable/Disable debugging
 144 ~~~~~~~~~~~~~~~~~~~~~~~~
 145
 146 You can enable verbose syslog messages with:
 147
 148  echo "1" >/etc/pve/.debug
 149
 150 And disable verbose syslog messages with:
 151
 152  echo "0" >/etc/pve/.debug
 153
 154
 155 Recovery
 156 --------
 157
 158 If you have major problems with your Proxmox VE host, e.g. hardware
 159 issues, it could be helpful to just copy the pmxcfs database file
 160 `/var/lib/pve-cluster/config.db` and move it to a new Proxmox VE
 161 host. On the new host (with nothing running), you need to stop the
 162 `pve-cluster` service and replace the `config.db` file (needed permissions
 163 `0600`). Second, adapt `/etc/hostname` and `/etc/hosts` according to the
 164 lost Proxmox VE host, then reboot and check. (And don't forget your
 165 VM/CT data)
 166
 167
 168 Remove Cluster configuration
 169 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 170
 171 The recommended way is to reinstall the node after you removed it from
 172 your cluster. This makes sure that all secret cluster/ssh keys and any
 173 shared configuration data is destroyed.
 174
 175 In some cases, you might prefer to put a node back to local mode
 176 without reinstall, which is described here:
 177
 178 * stop the cluster file system in `/etc/pve/`
 179
 180  # systemctl stop pve-cluster
 181
 182 * start it again but forcing local mode
 183
 184  # pmxcfs -l
 185
 186 * remove the cluster configuration
 187
 188  # rm /etc/pve/cluster.conf
 189  # rm /etc/cluster/cluster.conf
 190  # rm /var/lib/pve-cluster/corosync.authkey
 191
 192 * stop the cluster file system again
 193
 194  # systemctl stop pve-cluster
 195
 196 * restart PVE services (or reboot)
 197
 198  # systemctl start pve-cluster
 199  # systemctl restart pvedaemon
 200  # systemctl restart pveproxy
 201  # systemctl restart pvestatd
 202
 203
 204 ifdef::manvolnum[]
 205 include::pve-copyright.adoc[]
 206 endif::manvolnum[]