From ac1e389617f07d9c273dbb7996a21e24a4f07dd0 Mon Sep 17 00:00:00 2001 From: Dietmar Maurer Date: Tue, 5 Jan 2016 10:11:13 +0100 Subject: [PATCH] import pmxcfs docs --- pmxcfs.adoc | 170 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 pmxcfs.adoc diff --git a/pmxcfs.adoc b/pmxcfs.adoc new file mode 100644 index 0000000..eb5de34 --- /dev/null +++ b/pmxcfs.adoc @@ -0,0 +1,170 @@ +Proxmox Cluster file system (pmxcfs) +==================================== + +The Proxmox Cluster file system (pmxcfs) is a database-driven file +system for storing configuration files, replicated in real time to all +cluster nodes using corosync. We use this to store all PVE related +configuration files. + +Although the file system stores all data inside a persistent database +on disk, a copy of the data resides in RAM. That imposes restriction +on the maximal size, which is currently 30MB. This is still enough to +store the configuration of several thousand virtual machines. + +Advantages +---------- + +* seamless replication of all configuration to all nodes in real time +* provides strong consistency checks to avoid duplicate VM IDs +* read-only when a node looses quorum +* automatic updates of the corosync cluster configuration to all nodes +* includes a distributed locking mechanism + +POSIX Compatibility +~~~~~~~~~~~~~~~~~~~ + +The file system is based on FUSE, so the behavior is POSIX like. But +some feature are simply not implemented, because we do not need them: + +* you can just generate normal files and directories, but no symbolic + links, ... + +* you can't rename non-empty directories (because this makes it easier + to guarantee that VMIDs are unique). + +* you can't change file permissions (permissions are based on path) + +* `O_EXCL` creates were not atomic (like old NFS) + +* `O_TRUNC` creates are not atomic (FUSE restriction) + + +File access rights +~~~~~~~~~~~~~~~~~~ + +All files and directories are owned by user 'root' and have group +'www-data'. Only root has write permissions, but group 'www-data' can +read most files. Files below the following paths: + + /etc/pve/priv/ + /etc/pve/nodes/${NAME}/priv/ + +are only accessible by root. + +Technology +---------- + +We use the http://www.corosync.org[Corosync Cluster Engine] for +cluster communication, and http://www.sqlite.org[SQlite] for the +database file. The filesystem is implemented in user space using +http://fuse.sourceforge.net[FUSE]. + +File system layout +------------------ + +The file system is mounted at: + + /etc/pve + +Files +~~~~~ + +[width="100%",cols="m,d"] +|======= +|corosync.conf |corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf) +|storage.cfg |{pve} storage configuration +|user.cfg |{pve} access control configuration (users/groups/...) +|domains.cfg |{pve} Authentication domains +|authkey.pub | public key used by ticket system +|priv/shadow.cfg | shadow password file +|priv/authkey.key | private key used by ticket system +|nodes//pve-ssl.pem | public ssl key for web server +|nodes//priv/pve-ssl.key | private ssl key +|nodes//qemu-server/.conf | VM configuration data for KVM VMs +|nodes//lxc/.conf | VM configuration data for LXC containers +|firewall/cluster.fw | Firewall config applied to all nodes +|firewall/.fw | Firewall config for individual nodes +|firewall/.fw | Firewall config for VMs and Containers +|======= + +Symbolic links +~~~~~~~~~~~~~~ + +[width="100%",cols="m,m"] +|======= +|local |nodes/ +|qemu-server |nodes//qemu-server/ +|lxc |nodes//lxc/ +|======= + +Special status files for debugging (JSON) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +[width="100%",cols="m,d"] +|======= +| .version |file versions (to detect file modifications) +| .members |Info about cluster members +| .vmlist |List of all VMs +| .clusterlog |Cluster log (last 50 entries) +| .rrd |RRD data (most recent entries) +|======= + +Enable/Disable debugging +~~~~~~~~~~~~~~~~~~~~~~~~ + +You can enable verbose syslog messages with: + + echo "1" >/etc/pve/.debug + +And disable verbose syslog messages with: + + echo "0" >/etc/pve/.debug + + +Recovery +-------- + +If you have major problems with your Proxmox VE host, e.g. hardware +issues, it could be helpful to just copy the pmxcfs database file +/var/lib/pve-cluster/config.db and move it to a new Proxmox VE +host. On the new host (with nothing running), you need to stop the +pve-cluster service and replace the config.db file (needed permissions +0600). Second, adapt '/etc/hostname' and '/etc/hosts' according to the +lost Proxmox VE host, then reboot and check. (And don´t forget your +VM/CT data) + +Remove Cluster configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The recommended way is to reinstall the node after you removed it from +your cluster. This makes sure that all secret cluster/ssh keys and any +shared configuration data is destroyed. + +In some cases, you might prefer to put a node back to local mode +without reinstall, which is described here: + +* stop the cluster file system in '/etc/pve/' + + # systemctl stop pve-cluster + +* start it again but forcing local mode + + # pmxcfs -l + +* remove the cluster config + + # rm /etc/pve/cluster.conf + # rm /etc/cluster/cluster.conf + # rm /var/lib/pve-cluster/corosync.authkey + +* stop the cluster file system again + + # service pve-cluster stop + +* restart pve services (or reboot) + + # service pve-cluster start + # service pvedaemon restart + # service pveproxy restart + # service pvestatd restart + -- 2.39.2