10 pct - Tool to manage Linux Containers (LXC) on Proxmox VE
16 include::pct.1-synopsis.adoc[]
23 Proxmox Container Toolkit
24 =========================
28 :title: Linux Container
31 Containers are a lightweight alternative to fully virtualized
32 VMs. Instead of emulating a complete Operating System (OS), containers
33 simply use the OS of the host they run on. This implies that all
34 containers use the same kernel, and that they can access resources
35 from the host directly.
37 This is great because containers do not waste CPU power nor memory due
38 to kernel emulation. Container run-time costs are close to zero and
39 usually negligible. But there are also some drawbacks you need to
42 * You can only run Linux based OS inside containers, i.e. it is not
43 possible to run FreeBSD or MS Windows inside.
45 * For security reasons, access to host resources needs to be
46 restricted. This is done with AppArmor, SecComp filters and other
47 kernel features. Be prepared that some syscalls are not allowed
50 {pve} uses https://linuxcontainers.org/[LXC] as underlying container
51 technology. We consider LXC as low-level library, which provides
52 countless options. It would be too difficult to use those tools
53 directly. Instead, we provide a small wrapper called `pct`, the
54 "Proxmox Container Toolkit".
56 The toolkit is tightly coupled with {pve}. That means that it is aware
57 of the cluster setup, and it can use the same network and storage
58 resources as fully virtualized VMs. You can even use the {pve}
59 firewall, or manage containers using the HA framework.
61 Our primary goal is to offer an environment as one would get from a
62 VM, but without the additional overhead. We call this "System
65 NOTE: If you want to run micro-containers (with docker, rkt, ...), it
66 is best to run them inside a VM.
72 * LXC (https://linuxcontainers.org/)
74 * Integrated into {pve} graphical user interface (GUI)
76 * Easy to use command line tool `pct`
78 * Access via {pve} REST API
80 * lxcfs to provide containerized /proc file system
82 * AppArmor/Seccomp to improve security
84 * CRIU: for live migration (planned)
86 * Use latest available kernels (4.4.X)
88 * Image based deployment (templates)
90 * Use {pve} storage library
92 * Container setup from host (network, DNS, storage, ...)
95 Security Considerations
96 -----------------------
98 Containers use the same kernel as the host, so there is a big attack
99 surface for malicious users. You should consider this fact if you
100 provide containers to totally untrusted people. In general, fully
101 virtualized VMs provide better isolation.
103 The good news is that LXC uses many kernel security features like
104 AppArmor, CGroups and PID and user namespaces, which makes containers
105 usage quite secure. We distinguish two types of containers:
108 Privileged Containers
109 ~~~~~~~~~~~~~~~~~~~~~
111 Security is done by dropping capabilities, using mandatory access
112 control (AppArmor), SecComp filters and namespaces. The LXC team
113 considers this kind of container as unsafe, and they will not consider
114 new container escape exploits to be security issues worthy of a CVE
115 and quick fix. So you should use this kind of containers only inside a
116 trusted environment, or when no untrusted task is running as root in
120 Unprivileged Containers
121 ~~~~~~~~~~~~~~~~~~~~~~~
123 This kind of containers use a new kernel feature called user
124 namespaces. The root UID 0 inside the container is mapped to an
125 unprivileged user outside the container. This means that most security
126 issues (container escape, resource abuse, ...) in those containers
127 will affect a random unprivileged user, and so would be a generic
128 kernel security bug rather than an LXC issue. The LXC team thinks
129 unprivileged containers are safe by design.
132 Guest Operating System Configuration
133 ------------------------------------
135 We normally try to detect the operating system type inside the
136 container, and then modify some files inside the container to make
137 them work as expected. Here is a short list of things we do at
140 set /etc/hostname:: to set the container name
142 modify /etc/hosts:: to allow lookup of the local hostname
144 network setup:: pass the complete network setup to the container
146 configure DNS:: pass information about DNS servers
148 adapt the init system:: for example, fix the number of spawned getty processes
150 set the root password:: when creating a new container
152 rewrite ssh_host_keys:: so that each container has unique keys
154 randomize crontab:: so that cron does not start at the same time on all containers
156 Changes made by {PVE} are enclosed by comment markers:
164 Those markers will be inserted at a reasonable location in the
165 file. If such a section already exists, it will be updated in place
166 and will not be moved.
168 Modification of a file can be prevented by adding a `.pve-ignore.`
169 file for it. For instance, if the file `/etc/.pve-ignore.hosts`
170 exists then the `/etc/hosts` file will not be touched. This can be a
171 simple empty file creatd via:
173 # touch /etc/.pve-ignore.hosts
175 Most modifications are OS dependent, so they differ between different
176 distributions and versions. You can completely disable modifications
177 by manually setting the `ostype` to `unmanaged`.
179 OS type detection is done by testing for certain files inside the
182 Ubuntu:: inspect /etc/lsb-release (`DISTRIB_ID=Ubuntu`)
184 Debian:: test /etc/debian_version
186 Fedora:: test /etc/fedora-release
188 RedHat or CentOS:: test /etc/redhat-release
190 ArchLinux:: test /etc/arch-release
192 Alpine:: test /etc/alpine-release
194 Gentoo:: test /etc/gentoo-release
196 NOTE: Container start fails if the configured `ostype` differs from the auto
200 [[pct_container_images]]
204 Container images, sometimes also referred to as ``templates'' or
205 ``appliances'', are `tar` archives which contain everything to run a
206 container. You can think of it as a tidy container backup. Like most
207 modern container toolkits, `pct` uses those images when you create a
208 new container, for example:
210 pct create 999 local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz
212 {pve} itself ships a set of basic templates for most common
213 operating systems, and you can download them using the `pveam` (short
214 for {pve} Appliance Manager) command line utility. You can also
215 download https://www.turnkeylinux.org/[TurnKey Linux] containers using
216 that tool (or the graphical user interface).
218 Our image repositories contain a list of available images, and there
219 is a cron job run each day to download that list. You can trigger that
220 update manually with:
224 After that you can view the list of available images using:
228 You can restrict this large list by specifying the `section` you are
229 interested in, for example basic `system` images:
231 .List available system images
233 # pveam available --section system
234 system archlinux-base_2015-24-29-1_x86_64.tar.gz
235 system centos-7-default_20160205_amd64.tar.xz
236 system debian-6.0-standard_6.0-7_amd64.tar.gz
237 system debian-7.0-standard_7.0-3_amd64.tar.gz
238 system debian-8.0-standard_8.0-1_amd64.tar.gz
239 system ubuntu-12.04-standard_12.04-1_amd64.tar.gz
240 system ubuntu-14.04-standard_14.04-1_amd64.tar.gz
241 system ubuntu-15.04-standard_15.04-1_amd64.tar.gz
242 system ubuntu-15.10-standard_15.10-1_amd64.tar.gz
245 Before you can use such a template, you need to download them into one
246 of your storages. You can simply use storage `local` for that
247 purpose. For clustered installations, it is preferred to use a shared
248 storage so that all nodes can access those images.
250 pveam download local debian-8.0-standard_8.0-1_amd64.tar.gz
252 You are now ready to create containers using that image, and you can
253 list all downloaded images on storage `local` with:
257 local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz 190.20MB
260 The above command shows you the full {pve} volume identifiers. They include
261 the storage name, and most other {pve} commands can use them. For
262 example you can delete that image later with:
264 pveam remove local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz
267 [[pct_container_storage]]
271 Traditional containers use a very simple storage model, only allowing
272 a single mount point, the root file system. This was further
273 restricted to specific file system types like `ext4` and `nfs`.
274 Additional mounts are often done by user provided scripts. This turned
275 out to be complex and error prone, so we try to avoid that now.
277 Our new LXC based container model is more flexible regarding
278 storage. First, you can have more than a single mount point. This
279 allows you to choose a suitable storage for each application. For
280 example, you can use a relatively slow (and thus cheap) storage for
281 the container root file system. Then you can use a second mount point
282 to mount a very fast, distributed storage for your database
283 application. See section <<pct_mount_points,Mount Points>> for further
286 The second big improvement is that you can use any storage type
287 supported by the {pve} storage library. That means that you can store
288 your containers on local `lvmthin` or `zfs`, shared `iSCSI` storage,
289 or even on distributed storage systems like `ceph`. It also enables us
290 to use advanced storage features like snapshots and clones. `vzdump`
291 can also use the snapshot feature to provide consistent container
294 Last but not least, you can also mount local devices directly, or
295 mount local directories using bind mounts. That way you can access
296 local storage inside containers with zero overhead. Such bind mounts
297 also provide an easy way to share data between different containers.
303 WARNING: Because of existing issues in the Linux kernel's freezer
304 subsystem the usage of FUSE mounts inside a container is strongly
305 advised against, as containers need to be frozen for suspend or
306 snapshot mode backups.
308 If FUSE mounts cannot be replaced by other mounting mechanisms or storage
309 technologies, it is possible to establish the FUSE mount on the Proxmox host
310 and use a bind mount point to make it accessible inside the container.
313 Using Quotas Inside Containers
314 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
316 Quotas allow to set limits inside a container for the amount of disk
317 space that each user can use. This only works on ext4 image based
318 storage types and currently does not work with unprivileged
321 Activating the `quota` option causes the following mount options to be
322 used for a mount point:
323 `usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0`
325 This allows quotas to be used like you would on any other system. You
326 can initialize the `/aquota.user` and `/aquota.group` files by running
333 and edit the quotas via the `edquota` command. Refer to the documentation
334 of the distribution running inside the container for details.
336 NOTE: You need to run the above commands for every mount point by passing
337 the mount point's path instead of just `/`.
340 Using ACLs Inside Containers
341 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
343 The standard Posix **A**ccess **C**ontrol **L**ists are also available inside containers.
344 ACLs allow you to set more detailed file ownership than the traditional user/
356 [thumbnail="gui-create-ct-cpu.png"]
358 You can restrict the number of visible CPUs inside the container using
359 the `cores` option. This is implemented using the Linux 'cpuset'
360 cgroup (**c**ontrol *group*). A special task inside `pvestatd` tries
361 to distribute running containers among available CPUs. You can view
362 the assigned CPUs using the following command:
366 ---------------------
370 ---------------------
373 Containers use the host kernel directly, so all task inside a
374 container are handled by the host CPU scheduler. {pve} uses the Linux
375 'CFS' (**C**ompletely **F**air **S**cheduler) scheduler by default,
376 which has additional bandwidth control options.
380 `cpulimit`: :: You can use this option to further limit assigned CPU
381 time. Please note that this is a floating point number, so it is
382 perfectly valid to assign two cores to a container, but restrict
383 overall CPU consumption to half a core.
390 `cpuunits`: :: This is a relative weight passed to the kernel
391 scheduler. The larger the number is, the more CPU time this container
392 gets. Number is relative to the weights of all the other running
393 containers. The default is 1024. You can use this setting to
394 prioritize some containers.
401 [thumbnail="gui-create-ct-memory.png"]
403 Container memory is controlled using the cgroup memory controller.
407 `memory`: :: Limit overall memory usage. This corresponds
408 to the `memory.limit_in_bytes` cgroup setting.
410 `swap`: :: Allows the container to use additional swap memory from the
411 host swap space. This corresponds to the `memory.memsw.limit_in_bytes`
412 cgroup setting, which is set to the sum of both value (`memory +
420 [thumbnail="gui-create-ct-root-disk.png"]
422 The root mount point is configured with the `rootfs` property, and you can
423 configure up to 10 additional mount points. The corresponding options
424 are called `mp0` to `mp9`, and they can contain the following setting:
426 include::pct-mountpoint-opts.adoc[]
428 Currently there are basically three types of mount points: storage backed
429 mount points, bind mounts and device mounts.
431 .Typical container `rootfs` configuration
433 rootfs: thin1:base-100-disk-1,size=8G
437 Storage Backed Mount Points
438 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
440 Storage backed mount points are managed by the {pve} storage subsystem and come
441 in three different flavors:
443 - Image based: these are raw images containing a single ext4 formatted file
445 - ZFS subvolumes: these are technically bind mounts, but with managed storage,
446 and thus allow resizing and snapshotting.
447 - Directories: passing `size=0` triggers a special case where instead of a raw
448 image a directory is created.
454 Bind mounts allow you to access arbitrary directories from your Proxmox VE host
455 inside a container. Some potential use cases are:
457 - Accessing your home directory in the guest
458 - Accessing an USB device directory in the guest
459 - Accessing an NFS mount from the host in the guest
461 Bind mounts are considered to not be managed by the storage subsystem, so you
462 cannot make snapshots or deal with quotas from inside the container. With
463 unprivileged containers you might run into permission problems caused by the
464 user mapping and cannot use ACLs.
466 NOTE: The contents of bind mount points are not backed up when using `vzdump`.
468 WARNING: For security reasons, bind mounts should only be established
469 using source directories especially reserved for this purpose, e.g., a
470 directory hierarchy under `/mnt/bindmounts`. Never bind mount system
471 directories like `/`, `/var` or `/etc` into a container - this poses a
474 NOTE: The bind mount source path must not contain any symlinks.
476 For example, to make the directory `/mnt/bindmounts/shared` accessible in the
477 container with ID `100` under the path `/shared`, use a configuration line like
478 `mp0: /mnt/bindmounts/shared,mp=/shared` in `/etc/pve/lxc/100.conf`.
479 Alternatively, use `pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared` to
480 achieve the same result.
486 Device mount points allow to mount block devices of the host directly into the
487 container. Similar to bind mounts, device mounts are not managed by {PVE}'s
488 storage subsystem, but the `quota` and `acl` options will be honored.
490 NOTE: Device mount points should only be used under special circumstances. In
491 most cases a storage backed mount point offers the same performance and a lot
494 NOTE: The contents of device mount points are not backed up when using `vzdump`.
497 [[pct_container_network]]
501 [thumbnail="gui-create-ct-network.png"]
503 You can configure up to 10 network interfaces for a single
504 container. The corresponding options are called `net0` to `net9`, and
505 they can contain the following setting:
507 include::pct-network-opts.adoc[]
510 [[pct_startup_and_shutdown]]
511 Automatic Start and Shutdown of Containers
512 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
514 After creating your containers, you probably want them to start automatically
515 when the host system boots. For this you need to select the option 'Start at
516 boot' from the 'Options' Tab of your container in the web interface, or set it with
517 the following command:
519 pct set <ctid> -onboot 1
521 .Start and Shutdown Order
522 // use the screenshot from qemu - its the same
523 [thumbnail="gui-qemu-edit-start-order.png"]
525 If you want to fine tune the boot order of your containers, you can use the following
528 * *Start/Shutdown order*: Defines the start order priority. E.g. set it to 1 if
529 you want the CT to be the first to be started. (We use the reverse startup
530 order for shutdown, so a container with a start order of 1 would be the last to
532 * *Startup delay*: Defines the interval between this container start and subsequent
533 containers starts . E.g. set it to 240 if you want to wait 240 seconds before starting
535 * *Shutdown timeout*: Defines the duration in seconds {pve} should wait
536 for the container to be offline after issuing a shutdown command.
537 By default this value is set to 60, which means that {pve} will issue a
538 shutdown request, wait 60s for the machine to be offline, and if after 60s
539 the machine is still online will notify that the shutdown action failed.
541 Please note that containers without a Start/Shutdown order parameter will always
542 start after those where the parameter is set, and this parameter only
543 makes sense between the machines running locally on a host, and not
554 It is possible to use the `vzdump` tool for container backup. Please
555 refer to the `vzdump` manual page for details.
558 Restoring Container Backups
559 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
561 Restoring container backups made with `vzdump` is possible using the
562 `pct restore` command. By default, `pct restore` will attempt to restore as much
563 of the backed up container configuration as possible. It is possible to override
564 the backed up configuration by manually setting container options on the command
565 line (see the `pct` manual page for details).
567 NOTE: `pvesm extractconfig` can be used to view the backed up configuration
568 contained in a vzdump archive.
570 There are two basic restore modes, only differing by their handling of mount
574 ``Simple'' Restore Mode
575 ^^^^^^^^^^^^^^^^^^^^^^^
577 If neither the `rootfs` parameter nor any of the optional `mpX` parameters
578 are explicitly set, the mount point configuration from the backed up
579 configuration file is restored using the following steps:
581 . Extract mount points and their options from backup
582 . Create volumes for storage backed mount points (on storage provided with the
583 `storage` parameter, or default local storage if unset)
584 . Extract files from backup archive
585 . Add bind and device mount points to restored configuration (limited to root user)
587 NOTE: Since bind and device mount points are never backed up, no files are
588 restored in the last step, but only the configuration options. The assumption
589 is that such mount points are either backed up with another mechanism (e.g.,
590 NFS space that is bind mounted into many containers), or not intended to be
593 This simple mode is also used by the container restore operations in the web
597 ``Advanced'' Restore Mode
598 ^^^^^^^^^^^^^^^^^^^^^^^^^
600 By setting the `rootfs` parameter (and optionally, any combination of `mpX`
601 parameters), the `pct restore` command is automatically switched into an
602 advanced mode. This advanced mode completely ignores the `rootfs` and `mpX`
603 configuration options contained in the backup archive, and instead only
604 uses the options explicitly provided as parameters.
606 This mode allows flexible configuration of mount point settings at restore time,
609 * Set target storages, volume sizes and other options for each mount point
611 * Redistribute backed up files according to new mount point scheme
612 * Restore to device and/or bind mount points (limited to root user)
615 Managing Containers with `pct`
616 ------------------------------
618 `pct` is the tool to manage Linux Containers on {pve}. You can create
619 and destroy containers, and control execution (start, stop, migrate,
620 ...). You can use pct to set parameters in the associated config file,
621 like network configuration or memory limits.
627 Create a container based on a Debian template (provided you have
628 already downloaded the template via the web interface)
630 pct create 100 /var/lib/vz/template/cache/debian-8.0-standard_8.0-1_amd64.tar.gz
636 Start a login session via getty
640 Enter the LXC namespace and run a shell as root user
644 Display the configuration
648 Add a network interface called `eth0`, bridged to the host bridge `vmbr0`,
649 set the address and gateway, while it's running
651 pct set 100 -net0 name=eth0,bridge=vmbr0,ip=192.168.15.147/24,gw=192.168.15.1
653 Reduce the memory of the container to 512MB
655 pct set 100 -memory 512
658 Obtaining Debugging Logs
659 ~~~~~~~~~~~~~~~~~~~~~~~~
661 In case `pct start` is unable to start a specific container, it might be
662 helpful to collect debugging output by running `lxc-start` (replace `ID` with
665 lxc-start -n ID -F -l DEBUG -o /tmp/lxc-ID.log
667 This command will attempt to start the container in foreground mode, to stop the container run `pct shutdown ID` or `pct stop ID` in a second terminal.
669 The collected debug log is written to `/tmp/lxc-ID.log`.
671 NOTE: If you have changed the container's configuration since the last start
672 attempt with `pct start`, you need to run `pct start` at least once to also
673 update the configuration used by `lxc-start`.
676 [[pct_configuration]]
680 The `/etc/pve/lxc/<CTID>.conf` file stores container configuration,
681 where `<CTID>` is the numeric ID of the given container. Like all
682 other files stored inside `/etc/pve/`, they get automatically
683 replicated to all other cluster nodes.
685 NOTE: CTIDs < 100 are reserved for internal purposes, and CTIDs need to be
688 .Example Container Configuration
695 net0: bridge=vmbr0,hwaddr=66:64:66:64:64:36,ip=dhcp,name=eth0,type=veth
696 rootfs: local:107/vm-107-disk-1.raw,size=7G
699 Those configuration files are simple text files, and you can edit them
700 using a normal text editor (`vi`, `nano`, ...). This is sometimes
701 useful to do small corrections, but keep in mind that you need to
702 restart the container to apply such changes.
704 For that reason, it is usually better to use the `pct` command to
705 generate and modify those files, or do the whole thing using the GUI.
706 Our toolkit is smart enough to instantaneously apply most changes to
707 running containers. This feature is called "hot plug", and there is no
708 need to restart the container in that case.
714 Container configuration files use a simple colon separated key/value
715 format. Each line has the following format:
722 Blank lines in those files are ignored, and lines starting with a `#`
723 character are treated as comments and are also ignored.
725 It is possible to add low-level, LXC style configuration directly, for
728 lxc.init_cmd: /sbin/my_own_init
732 lxc.init_cmd = /sbin/my_own_init
734 Those settings are directly passed to the LXC low-level tools.
741 When you create a snapshot, `pct` stores the configuration at snapshot
742 time into a separate snapshot section within the same configuration
743 file. For example, after creating a snapshot called ``testsnapshot'',
744 your configuration file will look like this:
746 .Container configuration with snapshot
760 There are a few snapshot related properties like `parent` and
761 `snaptime`. The `parent` property is used to store the parent/child
762 relationship between snapshots. `snaptime` is the snapshot creation
763 time stamp (Unix epoch).
770 include::pct.conf.5-opts.adoc[]
776 Container migrations, snapshots and backups (`vzdump`) set a lock to
777 prevent incompatible concurrent actions on the affected container. Sometimes
778 you need to remove such a lock manually (e.g., after a power failure).
782 CAUTION: Only do that if you are sure the action which set the lock is
791 `/etc/pve/lxc/<CTID>.conf`::
793 Configuration file for the container '<CTID>'.
796 include::pve-copyright.adoc[]