pmxcfs.adoc

   1 ifdef::manvolnum[]
   2 pmxcfs(8)
   3 =========
   4 include::attributes.txt[]
   5 :pve-toplevel:
   6
   7 NAME
   8 ----
   9
  10 pmxcfs - Proxmox Cluster File System
  11
  12 SYNOPSIS
  13 --------
  14
  15 include::pmxcfs.8-cli.adoc[]
  16
  17 DESCRIPTION
  18 -----------
  19 endif::manvolnum[]
  20
  21 ifndef::manvolnum[]
  22 Proxmox Cluster File System (pmxcfs)
  23 ====================================
  24 include::attributes.txt[]
  25 endif::manvolnum[]
  26 ifdef::wiki[]
  27 :pve-toplevel:
  28 endif::wiki[]
  29
  30 The Proxmox Cluster file system (``pmxcfs'') is a database-driven file
  31 system for storing configuration files, replicated in real time to all
  32 cluster nodes using `corosync`. We use this to store all PVE related
  33 configuration files.
  34
  35 Although the file system stores all data inside a persistent database
  36 on disk, a copy of the data resides in RAM. That imposes restriction
  37 on the maximum size, which is currently 30MB. This is still enough to
  38 store the configuration of several thousand virtual machines.
  39
  40 This system provides the following advantages:
  41
  42 * seamless replication of all configuration to all nodes in real time
  43 * provides strong consistency checks to avoid duplicate VM IDs
  44 * read-only when a node loses quorum
  45 * automatic updates of the corosync cluster configuration to all nodes
  46 * includes a distributed locking mechanism
  47
  48
  49 POSIX Compatibility
  50 -------------------
  51
  52 The file system is based on FUSE, so the behavior is POSIX like. But
  53 some feature are simply not implemented, because we do not need them:
  54
  55 * you can just generate normal files and directories, but no symbolic
  56   links, ...
  57
  58 * you can't rename non-empty directories (because this makes it easier
  59   to guarantee that VMIDs are unique).
  60
  61 * you can't change file permissions (permissions are based on path)
  62
  63 * `O_EXCL` creates were not atomic (like old NFS)
  64
  65 * `O_TRUNC` creates are not atomic (FUSE restriction)
  66
  67
  68 File Access Rights
  69 ------------------
  70
  71 All files and directories are owned by user `root` and have group
  72 `www-data`. Only root has write permissions, but group `www-data` can
  73 read most files. Files below the following paths:
  74
  75  /etc/pve/priv/
  76  /etc/pve/nodes/${NAME}/priv/
  77
  78 are only accessible by root.
  79
  80
  81 Technology
  82 ----------
  83
  84 We use the http://www.corosync.org[Corosync Cluster Engine] for
  85 cluster communication, and http://www.sqlite.org[SQlite] for the
  86 database file. The file system is implemented in user space using
  87 http://fuse.sourceforge.net[FUSE].
  88
  89 File System Layout
  90 ------------------
  91
  92 The file system is mounted at:
  93
  94  /etc/pve
  95
  96 Files
  97 ~~~~~
  98
  99 [width="100%",cols="m,d"]
 100 |=======
 101 |`corosync.conf`                        | Corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)
 102 |`storage.cfg`                          | {pve} storage configuration
 103 |`datacenter.cfg`                       | {pve} datacenter wide configuration (keyboard layout, proxy, ...)
 104 |`user.cfg`                             | {pve} access control configuration (users/groups/...)
 105 |`domains.cfg`                          | {pve} authentication domains
 106 |`authkey.pub`                          | Public key used by ticket system
 107 |`pve-root-ca.pem`                      | Public certificate of cluster CA
 108 |`priv/shadow.cfg`                      | Shadow password file
 109 |`priv/authkey.key`                     | Private key used by ticket system
 110 |`priv/pve-root-ca.key`                 | Private key of cluster CA
 111 |`nodes/<NAME>/pve-ssl.pem`             | Public SSL certificate for web server (signed by cluster CA)
 112 |`nodes/<NAME>/pve-ssl.key`             | Private SSL key for `pve-ssl.pem`
 113 |`nodes/<NAME>/pveproxy-ssl.pem`        | Public SSL certificate (chain) for web server (optional override for `pve-ssl.pem`)
 114 |`nodes/<NAME>/pveproxy-ssl.key`        | Private SSL key for `pveproxy-ssl.pem` (optional)
 115 |`nodes/<NAME>/qemu-server/<VMID>.conf` | VM configuration data for KVM VMs
 116 |`nodes/<NAME>/lxc/<VMID>.conf`         | VM configuration data for LXC containers
 117 |`firewall/cluster.fw`                  | Firewall configuration applied to all nodes
 118 |`firewall/<NAME>.fw`                   | Firewall configuration for individual nodes
 119 |`firewall/<VMID>.fw`                   | Firewall configuration for VMs and Containers
 120 |=======
 121
 122
 123 Symbolic links
 124 ~~~~~~~~~~~~~~
 125
 126 [width="100%",cols="m,m"]
 127 |=======
 128 |`local`         | `nodes/<LOCAL_HOST_NAME>`
 129 |`qemu-server`   | `nodes/<LOCAL_HOST_NAME>/qemu-server/`
 130 |`lxc`           | `nodes/<LOCAL_HOST_NAME>/lxc/`
 131 |=======
 132
 133
 134 Special status files for debugging (JSON)
 135 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 136
 137 [width="100%",cols="m,d"]
 138 |=======
 139 |`.version`    |File versions (to detect file modifications)
 140 |`.members`    |Info about cluster members
 141 |`.vmlist`     |List of all VMs
 142 |`.clusterlog` |Cluster log (last 50 entries)
 143 |`.rrd`        |RRD data (most recent entries)
 144 |=======
 145
 146
 147 Enable/Disable debugging
 148 ~~~~~~~~~~~~~~~~~~~~~~~~
 149
 150 You can enable verbose syslog messages with:
 151
 152  echo "1" >/etc/pve/.debug
 153
 154 And disable verbose syslog messages with:
 155
 156  echo "0" >/etc/pve/.debug
 157
 158
 159 Recovery
 160 --------
 161
 162 If you have major problems with your Proxmox VE host, e.g. hardware
 163 issues, it could be helpful to just copy the pmxcfs database file
 164 `/var/lib/pve-cluster/config.db` and move it to a new Proxmox VE
 165 host. On the new host (with nothing running), you need to stop the
 166 `pve-cluster` service and replace the `config.db` file (needed permissions
 167 `0600`). Second, adapt `/etc/hostname` and `/etc/hosts` according to the
 168 lost Proxmox VE host, then reboot and check. (And don't forget your
 169 VM/CT data)
 170
 171
 172 Remove Cluster configuration
 173 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 174
 175 The recommended way is to reinstall the node after you removed it from
 176 your cluster. This makes sure that all secret cluster/ssh keys and any
 177 shared configuration data is destroyed.
 178
 179 In some cases, you might prefer to put a node back to local mode
 180 without reinstall, which is described here:
 181
 182 * stop the cluster file system in `/etc/pve/`
 183
 184  # systemctl stop pve-cluster
 185
 186 * start it again but forcing local mode
 187
 188  # pmxcfs -l
 189
 190 * remove the cluster configuration
 191
 192  # rm /etc/pve/cluster.conf
 193  # rm /etc/cluster/cluster.conf
 194  # rm /var/lib/pve-cluster/corosync.authkey
 195
 196 * stop the cluster file system again
 197
 198  # systemctl stop pve-cluster
 199
 200 * restart PVE services (or reboot)
 201
 202  # systemctl start pve-cluster
 203  # systemctl restart pvedaemon
 204  # systemctl restart pveproxy
 205  # systemctl restart pvestatd
 206
 207
 208 ifdef::manvolnum[]
 209 include::pve-copyright.adoc[]
 210 endif::manvolnum[]