The runtime costs for containers is low, usually negligible. However, there are
some drawbacks that need be considered:
-* Only Linux distributions can be run in containers.It is not possible to run
- other Operating Systems like, for example, FreeBSD or Microsoft Windows
+* Only Linux distributions can be run in Proxmox Containers. It is not possible to run
+ other operating systems like, for example, FreeBSD or Microsoft Windows
inside a container.
* For security reasons, access to host resources needs to be restricted.
- Containers run in their own separate namespaces. Additionally some syscalls
- are not allowed within containers.
+ Therefore, containers run in their own separate namespaces. Additionally some
+ syscalls (user space requests to the Linux kernel) are not allowed within containers.
-{pve} uses https://linuxcontainers.org/[Linux Containers (LXC)] as underlying
+{pve} uses https://linuxcontainers.org/lxc/introduction/[Linux Containers (LXC)] as its underlying
container technology. The ``Proxmox Container Toolkit'' (`pct`) simplifies the
-usage and management of LXC containers.
+usage and management of LXC, by providing an interface that abstracts
+complex tasks.
Containers are tightly integrated with {pve}. This means that they are aware of
the cluster setup, and they can use the same network and storage resources as
virtual machines. You can also use the {pve} firewall, or manage containers
using the HA framework.
-Our primary goal is to offer an environment as one would get from a VM, but
-without the additional overhead. We call this ``System Containers''.
+Our primary goal is to offer an environment that provides the benefits of using a
+VM, but without the additional overhead. This means that Proxmox Containers can
+be categorized as ``System Containers'', rather than ``Application Containers''.
-NOTE: If you want to run micro-containers, for example, 'Docker' or 'rkt', it
-is best to run them inside a VM.
+NOTE: If you want to run application containers, for example, 'Docker' images, it
+is recommended that you run them inside a Proxmox Qemu VM. This will give you
+all the advantages of application containerization, while also providing the
+benefits that VMs offer, such as strong isolation from the host and the ability
+to live-migrate, which otherwise isn't possible with containers.
Technology Overview
* Modern Linux kernels
-* Image based deployment (templates)
+* Image based deployment (xref:pct_supported_distributions[templates])
* Uses {pve} xref:chapter_storage[storage library]
* Container setup from host (network, DNS, storage, etc.)
+[[pct_supported_distributions]]
+Supported Distributions
+-----------------------
+
+List of officially supported distributions can be found below.
+
+Templates for the following distributions are available through our
+repositories. You can use xref:pct_container_images[pveam] tool or the
+Graphical User Interface to download them.
+
+Alpine Linux
+~~~~~~~~~~~~
+
+[quote, 'https://alpinelinux.org']
+____
+Alpine Linux is a security-oriented, lightweight Linux distribution based on
+musl libc and busybox.
+____
+
+For currently supported releases see:
+
+https://alpinelinux.org/releases/
+
+Arch Linux
+~~~~~~~~~~
+
+[quote, 'https://archlinux.org/']
+____
+Arch Linux, a lightweight and flexible Linux® distribution that tries to Keep It Simple.
+____
+
+Arch Linux is using a rolling-release model, see its wiki for more details:
+
+https://wiki.archlinux.org/title/Arch_Linux
+
+CentOS, Almalinux, Rocky Linux
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CentOS / CentOS Stream
+^^^^^^^^^^^^^^^^^^^^^^
+
+[quote, 'https://centos.org']
+____
+The CentOS Linux distribution is a stable, predictable, manageable and
+reproducible platform derived from the sources of Red Hat Enterprise Linux
+(RHEL)
+____
+
+For currently supported releases see:
+
+https://wiki.centos.org/About/Product
+
+Almalinux
+^^^^^^^^^
+
+[quote, 'https://almalinux.org']
+____
+An Open Source, community owned and governed, forever-free enterprise Linux
+distribution, focused on long-term stability, providing a robust
+production-grade platform. AlmaLinux OS is 1:1 binary compatible with RHEL® and
+pre-Stream CentOS.
+____
+
+
+For currently supported releases see:
+
+https://en.wikipedia.org/wiki/AlmaLinux#Releases
+
+Rocky Linux
+^^^^^^^^^^^
+
+[quote, 'https://rockylinux.org']
+____
+Rocky Linux is a community enterprise operating system designed to be 100%
+bug-for-bug compatible with America's top enterprise Linux distribution now
+that its downstream partner has shifted direction.
+____
+
+For currently supported releases see:
+
+https://en.wikipedia.org/wiki/Rocky_Linux#Releases
+
+Debian
+~~~~~~
+
+[quote, 'https://www.debian.org/intro/index#software']
+____
+Debian is a free operating system, developed and maintained by the Debian
+project. A free Linux distribution with thousands of applications to meet our
+users' needs.
+____
+
+For currently supported releases see:
+
+https://www.debian.org/releases/stable/releasenotes
+
+Devuan
+~~~~~~
+
+[quote, 'https://www.devuan.org']
+____
+Devuan GNU+Linux is a fork of Debian without systemd that allows users to
+reclaim control over their system by avoiding unnecessary entanglements and
+ensuring Init Freedom.
+____
+
+For currently supported releases see:
+
+https://www.devuan.org/os/releases
+
+Fedora
+~~~~~~
+
+[quote, 'https://getfedora.org']
+____
+Fedora creates an innovative, free, and open source platform for hardware,
+clouds, and containers that enables software developers and community members
+to build tailored solutions for their users.
+____
+
+For currently supported releases see:
+
+https://fedoraproject.org/wiki/Releases
+
+Gentoo
+~~~~~~
+
+[quote, 'https://www.gentoo.org']
+____
+a highly flexible, source-based Linux distribution.
+____
+
+Gentoo is using a rolling-release model.
+
+OpenSUSE
+~~~~~~~~
+
+[quote, 'https://www.opensuse.org']
+____
+The makers' choice for sysadmins, developers and desktop users.
+____
+
+For currently supported releases see:
+
+https://get.opensuse.org/leap/
+
+Ubuntu
+~~~~~~
+
+[quote, 'https://ubuntu.com/']
+____
+Ubuntu is the modern, open source operating system on Linux for the enterprise
+server, desktop, cloud, and IoT.
+____
+
+For currently supported releases see:
+
+https://wiki.ubuntu.com/Releases
+
[[pct_container_images]]
Container Images
----------------
Container images, sometimes also referred to as ``templates'' or
``appliances'', are `tar` archives which contain everything to run a container.
-{pve} itself provides a variety of basic templates for the most common Linux
-distributions. They can be downloaded using the GUI or the `pveam` (short for
-{pve} Appliance Manager) command line utility.
-Additionally, https://www.turnkeylinux.org/[TurnKey Linux] container templates
-are also available to download.
+{pve} itself provides a variety of basic templates for the
+xref:pct_supported_distributions[most common Linux distributions]. They can be
+downloaded using the GUI or the `pveam` (short for {pve} Appliance Manager)
+command line utility. Additionally, https://www.turnkeylinux.org/[TurnKey
+Linux] container templates are also available to download.
The list of available templates is updated daily through the 'pve-daily-update'
timer. You can also trigger an update manually by executing:
.List available system images
----
# pveam available --section system
-system alpine-3.10-default_20190626_amd64.tar.xz
-system alpine-3.9-default_20190224_amd64.tar.xz
-system archlinux-base_20190924-1_amd64.tar.gz
-system centos-6-default_20191016_amd64.tar.xz
+system alpine-3.12-default_20200823_amd64.tar.xz
+system alpine-3.13-default_20210419_amd64.tar.xz
+system alpine-3.14-default_20210623_amd64.tar.xz
+system archlinux-base_20210420-1_amd64.tar.gz
system centos-7-default_20190926_amd64.tar.xz
-system centos-8-default_20191016_amd64.tar.xz
-system debian-10.0-standard_10.0-1_amd64.tar.gz
-system debian-8.0-standard_8.11-1_amd64.tar.gz
+system centos-8-default_20201210_amd64.tar.xz
system debian-9.0-standard_9.7-1_amd64.tar.gz
-system fedora-30-default_20190718_amd64.tar.xz
-system fedora-31-default_20191029_amd64.tar.xz
-system gentoo-current-default_20190718_amd64.tar.xz
-system opensuse-15.0-default_20180907_amd64.tar.xz
-system opensuse-15.1-default_20190719_amd64.tar.xz
+system debian-10-standard_10.7-1_amd64.tar.gz
+system devuan-3.0-standard_3.0_amd64.tar.gz
+system fedora-33-default_20201115_amd64.tar.xz
+system fedora-34-default_20210427_amd64.tar.xz
+system gentoo-current-default_20200310_amd64.tar.xz
+system opensuse-15.2-default_20200824_amd64.tar.xz
system ubuntu-16.04-standard_16.04.5-1_amd64.tar.gz
system ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz
-system ubuntu-19.04-standard_19.04-1_amd64.tar.gz
-system ubuntu-19.10-standard_19.10-1_amd64.tar.gz
+system ubuntu-20.04-standard_20.04-1_amd64.tar.gz
+system ubuntu-20.10-standard_20.10-1_amd64.tar.gz
+system ubuntu-21.04-standard_21.04-1_amd64.tar.gz
----
Before you can use such a template, you need to download them into one of your
NOTE: The bind mount source path must not contain any symlinks.
For example, to make the directory `/mnt/bindmounts/shared` accessible in the
-container with ID `100` under the path `/shared`, use a configuration line like
-`mp0: /mnt/bindmounts/shared,mp=/shared` in `/etc/pve/lxc/100.conf`.
-Alternatively, use `pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared` to
-achieve the same result.
+container with ID `100` under the path `/shared`, add a configuration line such as:
+
+----
+mp0: /mnt/bindmounts/shared,mp=/shared
+----
+
+into `/etc/pve/lxc/100.conf`.
+
+Or alternatively use the `pct` tool:
+
+----
+pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared
+----
+
+to achieve the same result.
Device Mount Points
makes sense between the machines running locally on a host, and not
cluster-wide.
+If you require a delay between the host boot and the booting of the first
+container, see the section on
+xref:first_guest_boot_delay[Proxmox VE Node Management].
+
+
Hookscripts
~~~~~~~~~~~
configuration file located at `/etc/pve/lxc/CTID.conf`:
----
-lxc.apparmor_profile = unconfined
+lxc.apparmor.profile = unconfined
----
WARNING: Please note that this is not recommended for production use.
-// TODO: describe cgroups + seccomp a bit more.
+[[pct_cgroup]]
+Control Groups ('cgroup')
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+'cgroup' is a kernel
+mechanism used to hierarchically organize processes and distribute system
+resources.
+
+The main resources controlled via 'cgroups' are CPU time, memory and swap
+limits, and access to device nodes. 'cgroups' are also used to "freeze" a
+container before taking snapshots.
+
+There are 2 versions of 'cgroups' currently available,
+https://www.kernel.org/doc/html/v5.11/admin-guide/cgroup-v1/index.html[legacy]
+and
+https://www.kernel.org/doc/html/v5.11/admin-guide/cgroup-v2.html['cgroupv2'].
+
+Since {pve} 7.0, the default is a pure 'cgroupv2' environment. Previously a
+"hybrid" setup was used, where resource control was mainly done in 'cgroupv1'
+with an additional 'cgroupv2' controller which could take over some subsystems
+via the 'cgroup_no_v1' kernel command line parameter. (See the
+https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html[kernel
+parameter documentation] for details.)
+
+[[pct_cgroup_compat]]
+CGroup Version Compatibility
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The main difference between pure 'cgroupv2' and the old hybrid environments
+regarding {pve} is that with 'cgroupv2' memory and swap are now controlled
+independently. The memory and swap settings for containers can map directly to
+these values, whereas previously only the memory limit and the limit of the
+*sum* of memory and swap could be limited.
+
+Another important difference is that the 'devices' controller is configured in a
+completely different way. Because of this, file system quotas are currently not
+supported in a pure 'cgroupv2' environment.
+
+'cgroupv2' support by the container's OS is needed to run in a pure 'cgroupv2'
+environment. Containers running 'systemd' version 231 or newer support
+'cgroupv2' footnote:[this includes all newest major versions of container
+templates shipped by {pve}], as do containers not using 'systemd' as init
+system footnote:[for example Alpine Linux].
+
+[NOTE]
+====
+CentOS 7 and Ubuntu 16.10 are two prominent Linux distributions releases,
+which have a 'systemd' version that is too old to run in a 'cgroupv2'
+environment, you can either
+
+* Upgrade the whole distribution to a newer release. For the examples above, that
+ could be Ubuntu 18.04 or 20.04, and CentOS 8 (or RHEL/CentOS derivatives like
+ AlmaLinux or Rocky Linux). This has the benefit to get the newest bug and
+ security fixes, often also new features, and moving the EOL date in the future.
+
+* Upgrade the Containers systemd version. If the distribution provides a
+ backports repository this can be an easy and quick stop-gap measurement.
+
+* Move the container, or its services, to a Virtual Machine. Virtual Machines
+ have a much less interaction with the host, that's why one can install
+ decades old OS versions just fine there.
+
+* Switch back to the legacy 'cgroup' controller. Note that while it can be a
+ valid solution, it's not a permanent one. There's a high likelihood that a
+ future {pve} major release, for example 8.0, cannot support the legacy
+ controller anymore.
+====
+
+[[pct_cgroup_change_version]]
+Changing CGroup Version
+^^^^^^^^^^^^^^^^^^^^^^^
+
+TIP: If file system quotas are not required and all containers support 'cgroupv2',
+it is recommended to stick to the new default.
+
+To switch back to the previous version the following kernel command line
+parameter can be used:
+
+----
+systemd.unified_cgroup_hierarchy=0
+----
+
+See xref:sysboot_edit_kernel_cmdline[this section] on editing the kernel boot
+command line on where to add the parameter.
+
+// TODO: seccomp a bit more.
// TODO: pve-lxc-syscalld
Quotas allow to set limits inside a container for the amount of disk space that
each user can use.
+NOTE: This currently requires the use of legacy 'cgroups'.
+
NOTE: This only works on ext4 image based storage types and currently only
works with privileged containers.
file is restored using the following steps:
. Extract mount points and their options from backup
-. Create volumes for storage backed mount points (on storage provided with the
- `storage` parameter, or default local storage if unset)
+. Create volumes for storage backed mount points on the storage provided with
+ the `storage` parameter (default: `local`).
. Extract files from backup archive
. Add bind and device mount points to restored configuration (limited to root
user)
# pct set 100 -memory 512
----
+Destroying a container always removes it from Access Control Lists and it always
+removes the firewall configuration of the container. You have to activate
+'--purge', if you want to additionally remove the container from replication jobs,
+backup jobs and HA resource configurations.
+
+----
+# pct destroy 100 --purge
+----
+
+Move a mount point volume to a different storage.
+
+----
+# pct move-volume 100 mp0 other-storage
+----
+
+Reassign a volume to a different CT. This will remove the volume `mp0` from
+the source CT and attaches it as `mp1` to the target CT. In the background
+the volume is being renamed so that the name matches the new owner.
+
+----
+# pct move-volume 100 mp0 --target-vmid 200 --target-volume mp1
+----
+
Obtaining Debugging Logs
~~~~~~~~~~~~~~~~~~~~~~~~
In case `pct start` is unable to start a specific container, it might be
-helpful to collect debugging output by running `lxc-start` (replace `ID` with
-the container's ID):
+helpful to collect debugging output by passing the `--debug` flag (replace `CTID` with
+the container's CTID):
+
+----
+# pct start CTID --debug
+----
+
+Alternatively, you can use the following `lxc-start` command, which will save
+the debug log to the file specified by the `-o` output option:
----
-# lxc-start -n ID -F -l DEBUG -o /tmp/lxc-ID.log
+# lxc-start -n CTID -F -l DEBUG -o /tmp/lxc-CTID.log
----
This command will attempt to start the container in foreground mode, to stop
-the container run `pct shutdown ID` or `pct stop ID` in a second terminal.
+the container run `pct shutdown CTID` or `pct stop CTID` in a second terminal.
-The collected debug log is written to `/tmp/lxc-ID.log`.
+The collected debug log is written to `/tmp/lxc-CTID.log`.
NOTE: If you have changed the container's configuration since the last start
attempt with `pct start`, you need to run `pct start` at least once to also