+[[chapter_pvesr]]
ifdef::manvolnum[]
pvesr(1)
========
:pve-toplevel:
endif::manvolnum[]
-The {PVE} storage replication tool (`pvesr`) manage the Proxmox VE Storage Based Replication.
-Storage Replication bring guest redundancy for local storage's,
-reduce the migration time and will only replicate new data.
+The `pvesr` command-line tool manages the {PVE} storage replication
+framework. Storage replication brings redundancy for guests using
+local storage and reduces migration time.
-It will replicate the vdisk of guest to an other node this make that data available
-without using shared/distributed storage. So in case of a node failure your guest data still available
-on the replicated node.
+It replicates guest volumes to another node so that all data is available
+without using shared storage. Replication uses snapshots to minimize traffic
+sent over the network. Therefore, new data is sent only incrementally after
+the initial full sync. In the case of a node failure, your guest data is
+still available on the replicated node.
-The minimal replication interval are 1 minute and the maximal interval is once a week.
-Interval schedule format is a subset of `systemd` calendar events.
-Every interval time your guest vdisk data will be synchronized,
-but only the new data will replicated. This reduce the amount of data to a minimum.
-New data are data what are written to the vdisk after the last replication.
+The replication is done automatically in configurable intervals.
+The minimum replication interval is one minute, and the maximal interval
+once a week. The format used to specify those intervals is a subset of
+`systemd` calendar events, see
+xref:pvesr_schedule_time_format[Schedule Format] section:
-Every guest can replicate to many target nodes, but only one replication job per target node is allowed.
+It is possible to replicate a guest to multiple target nodes,
+but not twice to the same target node.
-The migration of guests, where storage replication is activated, is currently only offline possible.
-When the guest will migrate to the target of the replication, only the delta of the data must migrated and
-the replication direction will switched automatically in the opposite direction.
-If you migrate to a node where you do not replicate, it will send the whole vdisk data to the new node and after the migration it continuous the replication job as usually.
+Each replications bandwidth can be limited, to avoid overloading a storage
+or server.
-WARNING: High-Availability is possible with Storage Replication but this can lead to lose data. So be aware of this problem before you use this combination.
+Only changes since the last replication (so-called `deltas`) need to be
+transferred if the guest is migrated to a node to which it already is
+replicated. This reduces the time needed significantly. The replication
+direction automatically switches if you migrate a guest to the replication
+target node.
+
+For example: VM100 is currently on `nodeA` and gets replicated to `nodeB`.
+You migrate it to `nodeB`, so now it gets automatically replicated back from
+`nodeB` to `nodeA`.
+
+If you migrate to a node where the guest is not replicated, the whole disk
+data must send over. After the migration, the replication job continues to
+replicate this guest to the configured nodes.
+
+[IMPORTANT]
+====
+High-Availability is allowed in combination with storage replication, but there
+may be some data loss between the last synced time and the time a node failed.
+====
Supported Storage Types
-----------------------
.Storage Types
[width="100%",options="header"]
-|============================================
-|Description |PVE type |Snapshots|Stable
+|=============================================
+|Description |Plugin type |Snapshots|Stable
|ZFS (local) |zfspool |yes |yes
-|============================================
-
-Schedule
---------
-
-Proxmox VE has a very flexible replication scheduler with will explained in detail here.
+|=============================================
-A schedule string has following format.
+[[pvesr_schedule_time_format]]
+Schedule Format
+---------------
+Replication uses xref:chapter_calendar_events[calendar events] for
+configuring the schedule.
-[day of the week] <time>
+Error Handling
+--------------
-Day of the week are taken of sun, mon, tue, wed, thu, fri and sat.
-It is allowed to make a list of this token.
-Or alternative you can use a span.
+If a replication job encounters problems, it is placed in an error state.
+In this state, the configured replication intervals get suspended
+temporarily. The failed replication is repeatedly tried again in a
+30 minute interval.
+Once this succeeds, the original schedule gets activated again.
-The second part are the time, here the string contains two sub-string separate by `:`.
-First comes the hours, than the minutes.
-0..23 are the allowed values for hours and 0..59 for minutes.
-If hours is missing it will substitute with `*`, this also counts for the day of the week.
-The `*` will be interpreted as wildcard. In case of the "all day of the week" it would be sun..sat, for the hours 0..23 or for the minutes 0..59.
+Possible issues
+~~~~~~~~~~~~~~~
-Examples:
-~~~~~~~~~
+Some of the most common issues are in the following list. Depending on your
+setup there may be another cause.
-.Schedule Examples
-[width="100%",options="header"]
-|============================================
-|String |Meaning
-|"mon,tue,wed,thu,fri" => "mon..fir" |All working days at 0:00.
-|"sat,sun" => "sat..sun" |Only on weekend at 0:00.
-|"mon,wed,fri" |Only on Monday, Wednesday and Friday at 0:00.
-|"12:05" => "sun..sat 12:05" |All weekdays at 12:05 PM.
-|"*/5" |All five minutes on all weekdays.
-|"mon..wed 30/10" |Monday, Tuesday, Wednesday at all hours and 30, 40, 50 minutes.
-|"fri 12..13:5/20" |Friday at 12:05, 12:25, 12:45, 13:05, 13:25 and 13:45.
-|"12/2:05" |All weekdays every 2 hours start at 12:05 until 22:05
-|============================================
-The simplest case is the `*` which means every min what is the minimum interval.
-
-
-Error State
------------
+* Network is not working.
-It can happen that a job will have a error.
-Common issues for example:
+* No free space left on the replication target storage.
-Network is not working.
+* Storage with same storage ID available on the target node
-No space on the disk.
-...
+NOTE: You can always use the replication log to find out what is causing the problem.
-In this case storage replication will retry every 30 miuntes.
-
-
-Migrating a guest in case of Error
+Migrating a guest in case of Error
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+// FIXME: move this to better fitting chapter (sysadmin ?) and only link to
+// it here
-Assumed that you have two guests (KVM 100| LXC 200) running on node A (pve1) and replicate to node B (pve2).
-Node A is fails and can not get back online. Now you have to migrate the guest to Node B manually.
-
-Connect to node B with ssh.
-
-- check if that the cluster has quorum.
+In the case of a grave error, a virtual guest may get stuck on a failed
+node. You then need to move it manually to a working node again.
- pvecm status
+Example
+~~~~~~~
-Is quorum is ok continuing with the skip[ the next step.
+Let's assume that you have two guests (VM 100 and CT 200) running on node A
+and replicate to node B.
+Node A failed and can not get back online. Now you have to migrate the guest
+to Node B manually.
-- If you have no quorum, it is important to make your pmxcfs writable.
+- connect to node B over ssh or open its shell via the web UI
-WARNING: If expected is set, please avoid large changes on the nodes before the cluster is quorate online because there is a little change to a split brain.
-
- pvecm expected 1
+- check if that the cluster is quorate
++
+----
+# pvecm status
+----
-- move the guest config form the origin node A to node B.
- cd /etc/pve/node && mv pve1/qemu-server/100.conf pve2/qemu-server/100.conf
- cd /etc/pve/node && mv pve1/lxc/200.conf pve2/lxc/100.conf
+- If you have no quorum, we strongly advise to fix this first and make the
+ node operable again. Only if this is not possible at the moment, you may
+ use the following command to enforce quorum on the current node:
++
+----
+# pvecm expected 1
+----
-- Now you can start the guests.
+WARNING: Avoid changes which affect the cluster if `expected votes` are set
+(for example adding/removing nodes, storages, virtual guests) at all costs.
+Only use it to get vital guests up and running again or to resolve the quorum
+issue itself.
- qm start 100
- pct start 100
+- move both guest configuration files form the origin node A to node B:
++
+----
+# mv /etc/pve/nodes/A/qemu-server/100.conf /etc/pve/nodes/B/qemu-server/100.conf
+# mv /etc/pve/nodes/A/lxc/200.conf /etc/pve/nodes/B/lxc/200.conf
+----
+- Now you can start the guests again:
++
+----
+# qm start 100
+# pct start 200
+----
-Create Replication
-------------------
+Remember to replace the VMIDs and node names with your respective values.
-[thumbnail="gui-create-replica-on-node.png"]
+Managing Jobs
+-------------
-You can easy create a replication job on the GUI or as well on the command line.
-There are only a few setting to configure.
+[thumbnail="screenshot/gui-qemu-add-replication-job.png"]
-The only required parameter on the GUI is the target node.
-If you create a job on the command line also a unique id is needed.
-This id will automatically generated in case of GUI created jobs.
-The id contains the VMID and a consecutive number separated by a minus.
-All other parameters are optional.
+You can use the web GUI to create, modify, and remove replication jobs
+easily. Additionally, the command-line interface (CLI) tool `pvesr` can be
+used to do this.
-Update Replication Job
-----------------------
+You can find the replication panel on all levels (datacenter, node, virtual
+guest) in the web GUI. They differ in which jobs get shown:
+all, node- or guest-specific jobs.
-For now is possible to update rate limit and the scheduler.
-It can be done on the GUI or on the cli.
+When adding a new job, you need to specify the guest if not already selected
+as well as the target node. The replication
+xref:pvesr_schedule_time_format[schedule] can be set if the default of `all
+15 minutes` is not desired. You may impose a rate-limit on a replication
+job. The rate limit can help to keep the load on the storage acceptable.
+A replication job is identified by a cluster-wide unique ID. This ID is
+composed of the VMID in addition to a job number.
+This ID must only be specified manually if the CLI tool is used.
-Using the Command Line Interface
---------------------------------
-Examples
-~~~~~~~~
+Command-line Interface Examples
+-------------------------------
-Create a replication job for VM 100 first job, which will run all 5 min with max bandwidth of 10MByte/s.
+Create a replication job which runs every 5 minutes with a limited bandwidth
+of 10 Mbps (megabytes per second) for the guest with ID 100.
- pvesr create-local-job 100-0 pve1 --schedule */5 --rate 10
+----
+# pvesr create-local-job 100-0 pve1 --schedule "*/5" --rate 10
+----
-Disable a running job for ID 100-0
+Disable an active job with ID `100-0`.
- pvesr disable 100-0
+----
+# pvesr disable 100-0
+----
-Enable a running job for ID 100-0
+Enable a deactivated job with ID `100-0`.
- pvesr disable 100-0
+----
+# pvesr enable 100-0
+----
-Change the schedule interval to once a hour
+Change the schedule interval of the job with ID `100-0` to once per hour.
- pvesr update 100-0 --schedule '*/00'
+----
+# pvesr update 100-0 --schedule '*/00'
+----
ifdef::manvolnum[]
include::pve-copyright.adoc[]
projects / pve-docs.git / blobdiff