10 pct - Tool to manage Linux Containers (LXC) on Proxmox VE
16 include::pct.1-synopsis.adoc[]
23 Proxmox Container Toolkit
24 =========================
28 :title: Linux Container
31 Containers are a lightweight alternative to fully virtualized
32 VMs. Instead of emulating a complete Operating System (OS), containers
33 simply use the OS of the host they run on. This implies that all
34 containers use the same kernel, and that they can access resources
35 from the host directly.
37 This is great because containers do not waste CPU power nor memory due
38 to kernel emulation. Container run-time costs are close to zero and
39 usually negligible. But there are also some drawbacks you need to
42 * You can only run Linux based OS inside containers, i.e. it is not
43 possible to run FreeBSD or MS Windows inside.
45 * For security reasons, access to host resources needs to be
46 restricted. This is done with AppArmor, SecComp filters and other
47 kernel features. Be prepared that some syscalls are not allowed
50 {pve} uses https://linuxcontainers.org/[LXC] as underlying container
51 technology. We consider LXC as low-level library, which provides
52 countless options. It would be too difficult to use those tools
53 directly. Instead, we provide a small wrapper called `pct`, the
54 "Proxmox Container Toolkit".
56 The toolkit is tightly coupled with {pve}. That means that it is aware
57 of the cluster setup, and it can use the same network and storage
58 resources as fully virtualized VMs. You can even use the {pve}
59 firewall, or manage containers using the HA framework.
61 Our primary goal is to offer an environment as one would get from a
62 VM, but without the additional overhead. We call this "System
65 NOTE: If you want to run micro-containers (with docker, rkt, ...), it
66 is best to run them inside a VM.
72 * LXC (https://linuxcontainers.org/)
74 * Integrated into {pve} graphical user interface (GUI)
76 * Easy to use command line tool `pct`
78 * Access via {pve} REST API
80 * lxcfs to provide containerized /proc file system
82 * AppArmor/Seccomp to improve security
84 * CRIU: for live migration (planned)
86 * Use latest available kernels (4.4.X)
88 * Image based deployment (templates)
90 * Use {pve} storage library
92 * Container setup from host (network, DNS, storage, ...)
95 Security Considerations
96 -----------------------
98 Containers use the same kernel as the host, so there is a big attack
99 surface for malicious users. You should consider this fact if you
100 provide containers to totally untrusted people. In general, fully
101 virtualized VMs provide better isolation.
103 The good news is that LXC uses many kernel security features like
104 AppArmor, CGroups and PID and user namespaces, which makes containers
105 usage quite secure. We distinguish two types of containers:
108 Privileged Containers
109 ~~~~~~~~~~~~~~~~~~~~~
111 Security is done by dropping capabilities, using mandatory access
112 control (AppArmor), SecComp filters and namespaces. The LXC team
113 considers this kind of container as unsafe, and they will not consider
114 new container escape exploits to be security issues worthy of a CVE
115 and quick fix. So you should use this kind of containers only inside a
116 trusted environment, or when no untrusted task is running as root in
120 Unprivileged Containers
121 ~~~~~~~~~~~~~~~~~~~~~~~
123 This kind of containers use a new kernel feature called user
124 namespaces. The root UID 0 inside the container is mapped to an
125 unprivileged user outside the container. This means that most security
126 issues (container escape, resource abuse, ...) in those containers
127 will affect a random unprivileged user, and so would be a generic
128 kernel security bug rather than an LXC issue. The LXC team thinks
129 unprivileged containers are safe by design.
132 Guest Operating System Configuration
133 ------------------------------------
135 We normally try to detect the operating system type inside the
136 container, and then modify some files inside the container to make
137 them work as expected. Here is a short list of things we do at
140 set /etc/hostname:: to set the container name
142 modify /etc/hosts:: to allow lookup of the local hostname
144 network setup:: pass the complete network setup to the container
146 configure DNS:: pass information about DNS servers
148 adapt the init system:: for example, fix the number of spawned getty processes
150 set the root password:: when creating a new container
152 rewrite ssh_host_keys:: so that each container has unique keys
154 randomize crontab:: so that cron does not start at the same time on all containers
156 Changes made by {PVE} are enclosed by comment markers:
164 Those markers will be inserted at a reasonable location in the
165 file. If such a section already exists, it will be updated in place
166 and will not be moved.
168 Modification of a file can be prevented by adding a `.pve-ignore.`
169 file for it. For instance, if the file `/etc/.pve-ignore.hosts`
170 exists then the `/etc/hosts` file will not be touched. This can be a
171 simple empty file creatd via:
173 # touch /etc/.pve-ignore.hosts
175 Most modifications are OS dependent, so they differ between different
176 distributions and versions. You can completely disable modifications
177 by manually setting the `ostype` to `unmanaged`.
179 OS type detection is done by testing for certain files inside the
182 Ubuntu:: inspect /etc/lsb-release (`DISTRIB_ID=Ubuntu`)
184 Debian:: test /etc/debian_version
186 Fedora:: test /etc/fedora-release
188 RedHat or CentOS:: test /etc/redhat-release
190 ArchLinux:: test /etc/arch-release
192 Alpine:: test /etc/alpine-release
194 Gentoo:: test /etc/gentoo-release
196 NOTE: Container start fails if the configured `ostype` differs from the auto
200 [[pct_container_images]]
204 Container images, sometimes also referred to as ``templates'' or
205 ``appliances'', are `tar` archives which contain everything to run a
206 container. You can think of it as a tidy container backup. Like most
207 modern container toolkits, `pct` uses those images when you create a
208 new container, for example:
210 pct create 999 local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz
212 {pve} itself ships a set of basic templates for most common
213 operating systems, and you can download them using the `pveam` (short
214 for {pve} Appliance Manager) command line utility. You can also
215 download https://www.turnkeylinux.org/[TurnKey Linux] containers using
216 that tool (or the graphical user interface).
218 Our image repositories contain a list of available images, and there
219 is a cron job run each day to download that list. You can trigger that
220 update manually with:
224 After that you can view the list of available images using:
228 You can restrict this large list by specifying the `section` you are
229 interested in, for example basic `system` images:
231 .List available system images
233 # pveam available --section system
234 system archlinux-base_2015-24-29-1_x86_64.tar.gz
235 system centos-7-default_20160205_amd64.tar.xz
236 system debian-6.0-standard_6.0-7_amd64.tar.gz
237 system debian-7.0-standard_7.0-3_amd64.tar.gz
238 system debian-8.0-standard_8.0-1_amd64.tar.gz
239 system ubuntu-12.04-standard_12.04-1_amd64.tar.gz
240 system ubuntu-14.04-standard_14.04-1_amd64.tar.gz
241 system ubuntu-15.04-standard_15.04-1_amd64.tar.gz
242 system ubuntu-15.10-standard_15.10-1_amd64.tar.gz
245 Before you can use such a template, you need to download them into one
246 of your storages. You can simply use storage `local` for that
247 purpose. For clustered installations, it is preferred to use a shared
248 storage so that all nodes can access those images.
250 pveam download local debian-8.0-standard_8.0-1_amd64.tar.gz
252 You are now ready to create containers using that image, and you can
253 list all downloaded images on storage `local` with:
257 local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz 190.20MB
260 The above command shows you the full {pve} volume identifiers. They include
261 the storage name, and most other {pve} commands can use them. For
262 example you can delete that image later with:
264 pveam remove local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz
267 [[pct_container_storage]]
271 Traditional containers use a very simple storage model, only allowing
272 a single mount point, the root file system. This was further
273 restricted to specific file system types like `ext4` and `nfs`.
274 Additional mounts are often done by user provided scripts. This turned
275 out to be complex and error prone, so we try to avoid that now.
277 Our new LXC based container model is more flexible regarding
278 storage. First, you can have more than a single mount point. This
279 allows you to choose a suitable storage for each application. For
280 example, you can use a relatively slow (and thus cheap) storage for
281 the container root file system. Then you can use a second mount point
282 to mount a very fast, distributed storage for your database
283 application. See section <<pct_mount_points,Mount Points>> for further
286 The second big improvement is that you can use any storage type
287 supported by the {pve} storage library. That means that you can store
288 your containers on local `lvmthin` or `zfs`, shared `iSCSI` storage,
289 or even on distributed storage systems like `ceph`. It also enables us
290 to use advanced storage features like snapshots and clones. `vzdump`
291 can also use the snapshot feature to provide consistent container
294 Last but not least, you can also mount local devices directly, or
295 mount local directories using bind mounts. That way you can access
296 local storage inside containers with zero overhead. Such bind mounts
297 also provide an easy way to share data between different containers.
303 WARNING: Because of existing issues in the Linux kernel's freezer
304 subsystem the usage of FUSE mounts inside a container is strongly
305 advised against, as containers need to be frozen for suspend or
306 snapshot mode backups.
308 If FUSE mounts cannot be replaced by other mounting mechanisms or storage
309 technologies, it is possible to establish the FUSE mount on the Proxmox host
310 and use a bind mount point to make it accessible inside the container.
313 Using Quotas Inside Containers
314 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
316 Quotas allow to set limits inside a container for the amount of disk
317 space that each user can use. This only works on ext4 image based
318 storage types and currently does not work with unprivileged
321 Activating the `quota` option causes the following mount options to be
322 used for a mount point:
323 `usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0`
325 This allows quotas to be used like you would on any other system. You
326 can initialize the `/aquota.user` and `/aquota.group` files by running
333 and edit the quotas via the `edquota` command. Refer to the documentation
334 of the distribution running inside the container for details.
336 NOTE: You need to run the above commands for every mount point by passing
337 the mount point's path instead of just `/`.
340 Using ACLs Inside Containers
341 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
343 The standard Posix **A**ccess **C**ontrol **L**ists are also available inside containers.
344 ACLs allow you to set more detailed file ownership than the traditional user/
356 [thumbnail="gui-create-ct-cpu.png"]
358 You can restrict the number of visible CPUs inside the container using
359 the `cores` option. This is implemented using the Linux 'cpuset'
360 cgroup (**c**ontrol *group*). A special task inside `pvestatd` tries
361 to distribute running containers among available CPUs. You can view
362 the assigned CPUs using the following command:
366 ---------------------
370 ---------------------
373 Containers use the host kernel directly, so all task inside a
374 container are handled by the host CPU scheduler. {pve} uses the Linux
375 'CFS' (**C**ompletely **F**air **S**cheduler) scheduler by default,
376 which has additional bandwidth control options.
380 `cpulimit`: :: You can use this option to further limit assigned CPU
381 time. Please note that this is a floating point number, so it is
382 perfectly valid to assign two cores to a container, but restrict
383 overall CPU consumption to half a core.
390 `cpuunits`: :: This is a relative weight passed to the kernel
391 scheduler. The larger the number is, the more CPU time this container
392 gets. Number is relative to the weights of all the other running
393 containers. The default is 1024. You can use this setting to
394 prioritize some containers.
401 [thumbnail="gui-create-ct-memory.png"]
403 Container memory is controlled using the cgroup memory controller.
407 `memory`: :: Limit overall memory usage. This corresponds
408 to the `memory.limit_in_bytes` cgroup setting.
410 `swap`: :: Allows the container to use additional swap memory from the
411 host swap space. This corresponds to the `memory.memsw.limit_in_bytes`
412 cgroup setting, which is set to the sum of both value (`memory +
420 [thumbnail="gui-create-ct-root-disk.png"]
422 The root mount point is configured with the `rootfs` property, and you can
423 configure up to 10 additional mount points. The corresponding options
424 are called `mp0` to `mp9`, and they can contain the following setting:
426 include::pct-mountpoint-opts.adoc[]
428 Currently there are basically three types of mount points: storage backed
429 mount points, bind mounts and device mounts.
431 .Typical container `rootfs` configuration
433 rootfs: thin1:base-100-disk-1,size=8G
437 Storage Backed Mount Points
438 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
440 Storage backed mount points are managed by the {pve} storage subsystem and come
441 in three different flavors:
443 - Image based: these are raw images containing a single ext4 formatted file
445 - ZFS subvolumes: these are technically bind mounts, but with managed storage,
446 and thus allow resizing and snapshotting.
447 - Directories: passing `size=0` triggers a special case where instead of a raw
448 image a directory is created.
454 Bind mounts allow you to access arbitrary directories from your Proxmox VE host
455 inside a container. Some potential use cases are:
457 - Accessing your home directory in the guest
458 - Accessing an USB device directory in the guest
459 - Accessing an NFS mount from the host in the guest
461 Bind mounts are considered to not be managed by the storage subsystem, so you
462 cannot make snapshots or deal with quotas from inside the container. With
463 unprivileged containers you might run into permission problems caused by the
464 user mapping and cannot use ACLs.
466 NOTE: The contents of bind mount points are not backed up when using `vzdump`.
468 WARNING: For security reasons, bind mounts should only be established
469 using source directories especially reserved for this purpose, e.g., a
470 directory hierarchy under `/mnt/bindmounts`. Never bind mount system
471 directories like `/`, `/var` or `/etc` into a container - this poses a
474 NOTE: The bind mount source path must not contain any symlinks.
476 For example, to make the directory `/mnt/bindmounts/shared` accessible in the
477 container with ID `100` under the path `/shared`, use a configuration line like
478 `mp0: /mnt/bindmounts/shared,mp=/shared` in `/etc/pve/lxc/100.conf`.
479 Alternatively, use `pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared` to
480 achieve the same result.
486 Device mount points allow to mount block devices of the host directly into the
487 container. Similar to bind mounts, device mounts are not managed by {PVE}'s
488 storage subsystem, but the `quota` and `acl` options will be honored.
490 NOTE: Device mount points should only be used under special circumstances. In
491 most cases a storage backed mount point offers the same performance and a lot
494 NOTE: The contents of device mount points are not backed up when using `vzdump`.
497 [[pct_container_network]]
501 [thumbnail="gui-create-ct-network.png"]
503 You can configure up to 10 network interfaces for a single
504 container. The corresponding options are called `net0` to `net9`, and
505 they can contain the following setting:
507 include::pct-network-opts.adoc[]
510 [[pct_startup_and_shutdown]]
511 Automatic Start and Shutdown of Containers
512 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
514 After creating your containers, you probably want them to start automatically
515 when the host system boots. For this you need to select the option 'Start at
516 boot' from the 'Options' Tab of your container in the web interface, or set it with
517 the following command:
519 pct set <ctid> -onboot 1
521 If you want to fine tune the boot order of your containers, you can use the following
524 * *Start/Shutdown order*: Defines the start order priority. E.g. set it to 1 if
525 you want the CT to be the first to be started. (We use the reverse startup
526 order for shutdown, so a container with a start order of 1 would be the last to
528 * *Startup delay*: Defines the interval between this container start and subsequent
529 containers starts . E.g. set it to 240 if you want to wait 240 seconds before starting
531 * *Shutdown timeout*: Defines the duration in seconds {pve} should wait
532 for the container to be offline after issuing a shutdown command.
533 By default this value is set to 60, which means that {pve} will issue a
534 shutdown request, wait 60s for the machine to be offline, and if after 60s
535 the machine is still online will notify that the shutdown action failed.
537 Please note that containers without a Start/Shutdown order parameter will always
538 start after those where the parameter is set, and this parameter only
539 makes sense between the machines running locally on a host, and not
550 It is possible to use the `vzdump` tool for container backup. Please
551 refer to the `vzdump` manual page for details.
554 Restoring Container Backups
555 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
557 Restoring container backups made with `vzdump` is possible using the
558 `pct restore` command. By default, `pct restore` will attempt to restore as much
559 of the backed up container configuration as possible. It is possible to override
560 the backed up configuration by manually setting container options on the command
561 line (see the `pct` manual page for details).
563 NOTE: `pvesm extractconfig` can be used to view the backed up configuration
564 contained in a vzdump archive.
566 There are two basic restore modes, only differing by their handling of mount
570 ``Simple'' Restore Mode
571 ^^^^^^^^^^^^^^^^^^^^^^^
573 If neither the `rootfs` parameter nor any of the optional `mpX` parameters
574 are explicitly set, the mount point configuration from the backed up
575 configuration file is restored using the following steps:
577 . Extract mount points and their options from backup
578 . Create volumes for storage backed mount points (on storage provided with the
579 `storage` parameter, or default local storage if unset)
580 . Extract files from backup archive
581 . Add bind and device mount points to restored configuration (limited to root user)
583 NOTE: Since bind and device mount points are never backed up, no files are
584 restored in the last step, but only the configuration options. The assumption
585 is that such mount points are either backed up with another mechanism (e.g.,
586 NFS space that is bind mounted into many containers), or not intended to be
589 This simple mode is also used by the container restore operations in the web
593 ``Advanced'' Restore Mode
594 ^^^^^^^^^^^^^^^^^^^^^^^^^
596 By setting the `rootfs` parameter (and optionally, any combination of `mpX`
597 parameters), the `pct restore` command is automatically switched into an
598 advanced mode. This advanced mode completely ignores the `rootfs` and `mpX`
599 configuration options contained in the backup archive, and instead only
600 uses the options explicitly provided as parameters.
602 This mode allows flexible configuration of mount point settings at restore time,
605 * Set target storages, volume sizes and other options for each mount point
607 * Redistribute backed up files according to new mount point scheme
608 * Restore to device and/or bind mount points (limited to root user)
611 Managing Containers with `pct`
612 ------------------------------
614 `pct` is the tool to manage Linux Containers on {pve}. You can create
615 and destroy containers, and control execution (start, stop, migrate,
616 ...). You can use pct to set parameters in the associated config file,
617 like network configuration or memory limits.
623 Create a container based on a Debian template (provided you have
624 already downloaded the template via the web interface)
626 pct create 100 /var/lib/vz/template/cache/debian-8.0-standard_8.0-1_amd64.tar.gz
632 Start a login session via getty
636 Enter the LXC namespace and run a shell as root user
640 Display the configuration
644 Add a network interface called `eth0`, bridged to the host bridge `vmbr0`,
645 set the address and gateway, while it's running
647 pct set 100 -net0 name=eth0,bridge=vmbr0,ip=192.168.15.147/24,gw=192.168.15.1
649 Reduce the memory of the container to 512MB
651 pct set 100 -memory 512
654 Obtaining Debugging Logs
655 ~~~~~~~~~~~~~~~~~~~~~~~~
657 In case `pct start` is unable to start a specific container, it might be
658 helpful to collect debugging output by running `lxc-start` (replace `ID` with
661 lxc-start -n ID -F -l DEBUG -o /tmp/lxc-ID.log
663 This command will attempt to start the container in foreground mode, to stop the container run `pct shutdown ID` or `pct stop ID` in a second terminal.
665 The collected debug log is written to `/tmp/lxc-ID.log`.
667 NOTE: If you have changed the container's configuration since the last start
668 attempt with `pct start`, you need to run `pct start` at least once to also
669 update the configuration used by `lxc-start`.
672 [[pct_configuration]]
676 The `/etc/pve/lxc/<CTID>.conf` file stores container configuration,
677 where `<CTID>` is the numeric ID of the given container. Like all
678 other files stored inside `/etc/pve/`, they get automatically
679 replicated to all other cluster nodes.
681 NOTE: CTIDs < 100 are reserved for internal purposes, and CTIDs need to be
684 .Example Container Configuration
691 net0: bridge=vmbr0,hwaddr=66:64:66:64:64:36,ip=dhcp,name=eth0,type=veth
692 rootfs: local:107/vm-107-disk-1.raw,size=7G
695 Those configuration files are simple text files, and you can edit them
696 using a normal text editor (`vi`, `nano`, ...). This is sometimes
697 useful to do small corrections, but keep in mind that you need to
698 restart the container to apply such changes.
700 For that reason, it is usually better to use the `pct` command to
701 generate and modify those files, or do the whole thing using the GUI.
702 Our toolkit is smart enough to instantaneously apply most changes to
703 running containers. This feature is called "hot plug", and there is no
704 need to restart the container in that case.
710 Container configuration files use a simple colon separated key/value
711 format. Each line has the following format:
718 Blank lines in those files are ignored, and lines starting with a `#`
719 character are treated as comments and are also ignored.
721 It is possible to add low-level, LXC style configuration directly, for
724 lxc.init_cmd: /sbin/my_own_init
728 lxc.init_cmd = /sbin/my_own_init
730 Those settings are directly passed to the LXC low-level tools.
737 When you create a snapshot, `pct` stores the configuration at snapshot
738 time into a separate snapshot section within the same configuration
739 file. For example, after creating a snapshot called ``testsnapshot'',
740 your configuration file will look like this:
742 .Container configuration with snapshot
756 There are a few snapshot related properties like `parent` and
757 `snaptime`. The `parent` property is used to store the parent/child
758 relationship between snapshots. `snaptime` is the snapshot creation
759 time stamp (Unix epoch).
766 include::pct.conf.5-opts.adoc[]
772 Container migrations, snapshots and backups (`vzdump`) set a lock to
773 prevent incompatible concurrent actions on the affected container. Sometimes
774 you need to remove such a lock manually (e.g., after a power failure).
778 CAUTION: Only do that if you are sure the action which set the lock is
787 `/etc/pve/lxc/<CTID>.conf`::
789 Configuration file for the container '<CTID>'.
792 include::pve-copyright.adoc[]