pmxcfs.adoc

   1 ifdef::manvolnum[]
   2 PVE(8)
   3 ======
   4 include::attributes.txt[]
   5
   6 :pve-toplevel:
   7
   8 NAME
   9 ----
  10
  11 pmxcfs - Proxmox Cluster File System
  12
  13 SYNOPSIS
  14 --------
  15
  16 include::pmxcfs.8-cli.adoc[]
  17
  18 DESCRIPTION
  19 -----------
  20 endif::manvolnum[]
  21
  22 ifndef::manvolnum[]
  23 Proxmox Cluster File System (pmxcfs)
  24 ====================================
  25 include::attributes.txt[]
  26 endif::manvolnum[]
  27
  28 ifdef::wiki[]
  29 :pve-toplevel:
  30 endif::wiki[]
  31
  32 The Proxmox Cluster file system (``pmxcfs'') is a database-driven file
  33 system for storing configuration files, replicated in real time to all
  34 cluster nodes using `corosync`. We use this to store all PVE related
  35 configuration files.
  36
  37 Although the file system stores all data inside a persistent database
  38 on disk, a copy of the data resides in RAM. That imposes restriction
  39 on the maximum size, which is currently 30MB. This is still enough to
  40 store the configuration of several thousand virtual machines.
  41
  42 This system provides the following advantages:
  43
  44 * seamless replication of all configuration to all nodes in real time
  45 * provides strong consistency checks to avoid duplicate VM IDs
  46 * read-only when a node loses quorum
  47 * automatic updates of the corosync cluster configuration to all nodes
  48 * includes a distributed locking mechanism
  49
  50
  51 POSIX Compatibility
  52 -------------------
  53
  54 The file system is based on FUSE, so the behavior is POSIX like. But
  55 some feature are simply not implemented, because we do not need them:
  56
  57 * you can just generate normal files and directories, but no symbolic
  58   links, ...
  59
  60 * you can't rename non-empty directories (because this makes it easier
  61   to guarantee that VMIDs are unique).
  62
  63 * you can't change file permissions (permissions are based on path)
  64
  65 * `O_EXCL` creates were not atomic (like old NFS)
  66
  67 * `O_TRUNC` creates are not atomic (FUSE restriction)
  68
  69
  70 File Access Rights
  71 ------------------
  72
  73 All files and directories are owned by user `root` and have group
  74 `www-data`. Only root has write permissions, but group `www-data` can
  75 read most files. Files below the following paths:
  76
  77  /etc/pve/priv/
  78  /etc/pve/nodes/${NAME}/priv/
  79
  80 are only accessible by root.
  81
  82
  83 Technology
  84 ----------
  85
  86 We use the http://www.corosync.org[Corosync Cluster Engine] for
  87 cluster communication, and http://www.sqlite.org[SQlite] for the
  88 database file. The file system is implemented in user space using
  89 http://fuse.sourceforge.net[FUSE].
  90
  91 File System Layout
  92 ------------------
  93
  94 The file system is mounted at:
  95
  96  /etc/pve
  97
  98 Files
  99 ~~~~~
 100
 101 [width="100%",cols="m,d"]
 102 |=======
 103 |`corosync.conf`                        | Corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)
 104 |`storage.cfg`                          | {pve} storage configuration
 105 |`datacenter.cfg`                       | {pve} datacenter wide configuration (keyboard layout, proxy, ...)
 106 |`user.cfg`                             | {pve} access control configuration (users/groups/...)
 107 |`domains.cfg`                          | {pve} authentication domains
 108 |`authkey.pub`                          | Public key used by ticket system
 109 |`pve-root-ca.pem`                      | Public certificate of cluster CA
 110 |`priv/shadow.cfg`                      | Shadow password file
 111 |`priv/authkey.key`                     | Private key used by ticket system
 112 |`priv/pve-root-ca.key`                 | Private key of cluster CA
 113 |`nodes/<NAME>/pve-ssl.pem`             | Public SSL certificate for web server (signed by cluster CA)
 114 |`nodes/<NAME>/pve-ssl.key`             | Private SSL key for `pve-ssl.pem`
 115 |`nodes/<NAME>/pveproxy-ssl.pem`        | Public SSL certificate (chain) for web server (optional override for `pve-ssl.pem`)
 116 |`nodes/<NAME>/pveproxy-ssl.key`        | Private SSL key for `pveproxy-ssl.pem` (optional)
 117 |`nodes/<NAME>/qemu-server/<VMID>.conf` | VM configuration data for KVM VMs
 118 |`nodes/<NAME>/lxc/<VMID>.conf`         | VM configuration data for LXC containers
 119 |`firewall/cluster.fw`                  | Firewall configuration applied to all nodes
 120 |`firewall/<NAME>.fw`                   | Firewall configuration for individual nodes
 121 |`firewall/<VMID>.fw`                   | Firewall configuration for VMs and Containers
 122 |=======
 123
 124
 125 Symbolic links
 126 ~~~~~~~~~~~~~~
 127
 128 [width="100%",cols="m,m"]
 129 |=======
 130 |`local`         | `nodes/<LOCAL_HOST_NAME>`
 131 |`qemu-server`   | `nodes/<LOCAL_HOST_NAME>/qemu-server/`
 132 |`lxc`           | `nodes/<LOCAL_HOST_NAME>/lxc/`
 133 |=======
 134
 135
 136 Special status files for debugging (JSON)
 137 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 138
 139 [width="100%",cols="m,d"]
 140 |=======
 141 |`.version`    |File versions (to detect file modifications)
 142 |`.members`    |Info about cluster members
 143 |`.vmlist`     |List of all VMs
 144 |`.clusterlog` |Cluster log (last 50 entries)
 145 |`.rrd`        |RRD data (most recent entries)
 146 |=======
 147
 148
 149 Enable/Disable debugging
 150 ~~~~~~~~~~~~~~~~~~~~~~~~
 151
 152 You can enable verbose syslog messages with:
 153
 154  echo "1" >/etc/pve/.debug
 155
 156 And disable verbose syslog messages with:
 157
 158  echo "0" >/etc/pve/.debug
 159
 160
 161 Recovery
 162 --------
 163
 164 If you have major problems with your Proxmox VE host, e.g. hardware
 165 issues, it could be helpful to just copy the pmxcfs database file
 166 `/var/lib/pve-cluster/config.db` and move it to a new Proxmox VE
 167 host. On the new host (with nothing running), you need to stop the
 168 `pve-cluster` service and replace the `config.db` file (needed permissions
 169 `0600`). Second, adapt `/etc/hostname` and `/etc/hosts` according to the
 170 lost Proxmox VE host, then reboot and check. (And don't forget your
 171 VM/CT data)
 172
 173
 174 Remove Cluster configuration
 175 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 176
 177 The recommended way is to reinstall the node after you removed it from
 178 your cluster. This makes sure that all secret cluster/ssh keys and any
 179 shared configuration data is destroyed.
 180
 181 In some cases, you might prefer to put a node back to local mode
 182 without reinstall, which is described here:
 183
 184 * stop the cluster file system in `/etc/pve/`
 185
 186  # systemctl stop pve-cluster
 187
 188 * start it again but forcing local mode
 189
 190  # pmxcfs -l
 191
 192 * remove the cluster configuration
 193
 194  # rm /etc/pve/cluster.conf
 195  # rm /etc/cluster/cluster.conf
 196  # rm /var/lib/pve-cluster/corosync.authkey
 197
 198 * stop the cluster file system again
 199
 200  # systemctl stop pve-cluster
 201
 202 * restart PVE services (or reboot)
 203
 204  # systemctl start pve-cluster
 205  # systemctl restart pvedaemon
 206  # systemctl restart pveproxy
 207  # systemctl restart pvestatd
 208
 209
 210 ifdef::manvolnum[]
 211 include::pve-copyright.adoc[]
 212 endif::manvolnum[]