X-Git-Url: https://git.proxmox.com/?p=pve-docs.git;a=blobdiff_plain;f=pmxcfs.adoc;h=5a68598b9621319db66dbcfc2b1f9789f779bcb9;hp=3474d736a7d17af3751d53870a9e9ff28c4824ea;hb=6df6e5109267e8366035072edfaf8c64335f1e8b;hpb=8c1189b640ae7d10119ff1c046580f48749d38bd diff --git a/pmxcfs.adoc b/pmxcfs.adoc index 3474d73..5a68598 100644 --- a/pmxcfs.adoc +++ b/pmxcfs.adoc @@ -1,17 +1,17 @@ ifdef::manvolnum[] -PVE({manvolnum}) -================ -include::attributes.txt[] +pmxcfs(8) +========= +:pve-toplevel: NAME ---- pmxcfs - Proxmox Cluster File System -SYNOPSYS +SYNOPSIS -------- -include::pmxcfs.8-cli.adoc[] +include::pmxcfs.8-synopsis.adoc[] DESCRIPTION ----------- @@ -20,7 +20,7 @@ endif::manvolnum[] ifndef::manvolnum[] Proxmox Cluster File System (pmxcfs) ==================================== -include::attributes.txt[] +:pve-toplevel: endif::manvolnum[] The Proxmox Cluster file system (``pmxcfs'') is a database-driven file @@ -30,7 +30,7 @@ configuration files. Although the file system stores all data inside a persistent database on disk, a copy of the data resides in RAM. That imposes restriction -on the maximal size, which is currently 30MB. This is still enough to +on the maximum size, which is currently 30MB. This is still enough to store the configuration of several thousand virtual machines. This system provides the following advantages: @@ -41,6 +41,7 @@ This system provides the following advantages: * automatic updates of the corosync cluster configuration to all nodes * includes a distributed locking mechanism + POSIX Compatibility ------------------- @@ -60,7 +61,7 @@ some feature are simply not implemented, because we do not need them: * `O_TRUNC` creates are not atomic (FUSE restriction) -File access rights +File Access Rights ------------------ All files and directories are owned by user `root` and have group @@ -78,10 +79,10 @@ Technology We use the http://www.corosync.org[Corosync Cluster Engine] for cluster communication, and http://www.sqlite.org[SQlite] for the -database file. The filesystem is implemented in user space using +database file. The file system is implemented in user space using http://fuse.sourceforge.net[FUSE]. -File system layout +File System Layout ------------------ The file system is mounted at: @@ -114,6 +115,7 @@ Files |`firewall/.fw` | Firewall configuration for VMs and Containers |======= + Symbolic links ~~~~~~~~~~~~~~ @@ -124,6 +126,7 @@ Symbolic links |`lxc` | `nodes//lxc/` |======= + Special status files for debugging (JSON) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -136,6 +139,7 @@ Special status files for debugging (JSON) |`.rrd` |RRD data (most recent entries) |======= + Enable/Disable debugging ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -160,6 +164,7 @@ host. On the new host (with nothing running), you need to stop the lost Proxmox VE host, then reboot and check. (And don't forget your VM/CT data) + Remove Cluster configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -167,34 +172,48 @@ The recommended way is to reinstall the node after you removed it from your cluster. This makes sure that all secret cluster/ssh keys and any shared configuration data is destroyed. -In some cases, you might prefer to put a node back to local mode -without reinstall, which is described here: - -* stop the cluster file system in `/etc/pve/` - - # systemctl stop pve-cluster +In some cases, you might prefer to put a node back to local mode without +reinstall, which is described in +<> -* start it again but forcing local mode - # pmxcfs -l +Recovering/Moving Guests from Failed Nodes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -* remove the cluster config +For the guest configuration files in `nodes//qemu-server/` (VMs) and +`nodes//lxc/` (containers), {pve} sees the containing node `` as +owner of the respective guest. This concept enables the usage of local locks +instead of expensive cluster-wide locks for preventing concurrent guest +configuration changes. - # rm /etc/pve/cluster.conf - # rm /etc/cluster/cluster.conf - # rm /var/lib/pve-cluster/corosync.authkey +As a consequence, if the owning node of a guest fails (e.g., because of a power +outage, fencing event, ..), a regular migration is not possible (even if all +the disks are located on shared storage) because such a local lock on the +(dead) owning node is unobtainable. This is not a problem for HA-managed +guests, as {pve}'s High Availability stack includes the necessary +(cluster-wide) locking and watchdog functionality to ensure correct and +automatic recovery of guests from fenced nodes. -* stop the cluster file system again +If a non-HA-managed guest has only shared disks (and no other local resources +which are only available on the failed node are configured), a manual recovery +is possible by simply moving the guest configuration file from the failed +node's directory in `/etc/pve/` to an alive node's directory (which changes the +logical owner or location of the guest). - # systemctl stop pve-cluster +For example, recovering the VM with ID `100` from a dead `node1` to another +node `node2` works with the following command executed when logged in as root +on any member node of the cluster: -* restart pve services (or reboot) + mv /etc/pve/nodes/node1/qemu-server/100.conf /etc/pve/nodes/node2/ - # systemctl start pve-cluster - # systemctl restart pvedaemon - # systemctl restart pveproxy - # systemctl restart pvestatd +WARNING: Before manually recovering a guest like this, make absolutely sure +that the failed source node is really powered off/fenced. Otherwise {pve}'s +locking principles are violated by the `mv` command, which can have unexpected +consequences. +WARNING: Guest with local disks (or other local resources which are only +available on the dead node) are not recoverable like this. Either wait for the +failed node to rejoin the cluster or restore such guests from backups. ifdef::manvolnum[] include::pve-copyright.adoc[]