formatting cleanup
[pve-docs.git] / pct.adoc
CommitLineData
0c6b782f
DM
1ifdef::manvolnum[]
2PVE({manvolnum})
3================
38fd0958 4include::attributes.txt[]
0c6b782f
DM
5
6NAME
7----
8
9pct - Tool to manage Linux Containers (LXC) on Proxmox VE
10
11
12SYNOPSYS
13--------
14
15include::pct.1-synopsis.adoc[]
16
17DESCRIPTION
18-----------
19endif::manvolnum[]
20
21ifndef::manvolnum[]
22Proxmox Container Toolkit
23=========================
38fd0958 24include::attributes.txt[]
0c6b782f
DM
25endif::manvolnum[]
26
4a2ae9ed
DM
27
28Containers are a lightweight alternative to fully virtualized
29VMs. Instead of emulating a complete Operating System (OS), containers
30simply use the OS of the host they run on. This implies that all
31containers use the same kernel, and that they can access resources
32from the host directly.
33
34This is great because containers do not waste CPU power nor memory due
35to kernel emulation. Container run-time costs are close to zero and
36usually negligible. But there are also some drawbacks you need to
37consider:
38
39* You can only run Linux based OS inside containers, i.e. it is not
a8e99754 40 possible to run FreeBSD or MS Windows inside.
4a2ae9ed 41
a8e99754 42* For security reasons, access to host resources needs to be
4a2ae9ed 43 restricted. This is done with AppArmor, SecComp filters and other
a8e99754 44 kernel features. Be prepared that some syscalls are not allowed
4a2ae9ed
DM
45 inside containers.
46
47{pve} uses https://linuxcontainers.org/[LXC] as underlying container
48technology. We consider LXC as low-level library, which provides
a8e99754 49countless options. It would be too difficult to use those tools
4a2ae9ed
DM
50directly. Instead, we provide a small wrapper called `pct`, the
51"Proxmox Container Toolkit".
52
a8e99754 53The toolkit is tightly coupled with {pve}. That means that it is aware
4a2ae9ed
DM
54of the cluster setup, and it can use the same network and storage
55resources as fully virtualized VMs. You can even use the {pve}
56firewall, or manage containers using the HA framework.
57
58Our primary goal is to offer an environment as one would get from a
59VM, but without the additional overhead. We call this "System
60Containers".
61
99d2e25b 62NOTE: If you want to run micro-containers (with docker, rkt, ...), it
70a42028 63is best to run them inside a VM.
4a2ae9ed
DM
64
65
66Security Considerations
67-----------------------
68
69Containers use the same kernel as the host, so there is a big attack
70surface for malicious users. You should consider this fact if you
71provide containers to totally untrusted people. In general, fully
a8e99754 72virtualized VMs provide better isolation.
4a2ae9ed
DM
73
74The good news is that LXC uses many kernel security features like
75AppArmor, CGroups and PID and user namespaces, which makes containers
76usage quite secure. We distinguish two types of containers:
77
78Privileged containers
79~~~~~~~~~~~~~~~~~~~~~
80
81Security is done by dropping capabilities, using mandatory access
82control (AppArmor), SecComp filters and namespaces. The LXC team
83considers this kind of container as unsafe, and they will not consider
84new container escape exploits to be security issues worthy of a CVE
85and quick fix. So you should use this kind of containers only inside a
86trusted environment, or when no untrusted task is running as root in
87the container.
88
89Unprivileged containers
90~~~~~~~~~~~~~~~~~~~~~~~
91
a8e99754 92This kind of containers use a new kernel feature called user
4a2ae9ed
DM
93namespaces. The root uid 0 inside the container is mapped to an
94unprivileged user outside the container. This means that most security
95issues (container escape, resource abuse, ...) in those containers
96will affect a random unprivileged user, and so would be a generic
a8e99754 97kernel security bug rather than an LXC issue. The LXC team thinks
4a2ae9ed
DM
98unprivileged containers are safe by design.
99
7fc230db
DM
100
101Configuration
102-------------
103
8c1189b6
FG
104The `/etc/pve/lxc/<CTID>.conf` file stores container configuration,
105where `<CTID>` is the numeric ID of the given container. Like all
106other files stored inside `/etc/pve/`, they get automatically
166e63d6
FG
107replicated to all other cluster nodes.
108
109NOTE: CTIDs < 100 are reserved for internal purposes, and CTIDs need to be
110unique cluster wide.
7fc230db 111
105bc8f1
DM
112.Example Container Configuration
113----
114ostype: debian
115arch: amd64
116hostname: www
117memory: 512
118swap: 512
119net0: bridge=vmbr0,hwaddr=66:64:66:64:64:36,ip=dhcp,name=eth0,type=veth
120rootfs: local:107/vm-107-disk-1.raw,size=7G
121----
122
7fc230db 123Those configuration files are simple text files, and you can edit them
8c1189b6 124using a normal text editor (`vi`, `nano`, ...). This is sometimes
55fb2a21
DM
125useful to do small corrections, but keep in mind that you need to
126restart the container to apply such changes.
127
8c1189b6 128For that reason, it is usually better to use the `pct` command to
55fb2a21
DM
129generate and modify those files, or do the whole thing using the GUI.
130Our toolkit is smart enough to instantaneously apply most changes to
105bc8f1
DM
131running containers. This feature is called "hot plug", and there is no
132need to restart the container in that case.
7fc230db
DM
133
134File Format
135~~~~~~~~~~~
136
137Container configuration files use a simple colon separated key/value
138format. Each line has the following format:
139
140 # this is a comment
141 OPTION: value
142
8c1189b6 143Blank lines in those files are ignored, and lines starting with a `#`
7fc230db
DM
144character are treated as comments and are also ignored.
145
146It is possible to add low-level, LXC style configuration directly, for
147example:
148
149 lxc.init_cmd: /sbin/my_own_init
150
151or
152
153 lxc.init_cmd = /sbin/my_own_init
154
155Those settings are directly passed to the LXC low-level tools.
156
105bc8f1
DM
157Snapshots
158~~~~~~~~~
159
8c1189b6 160When you create a snapshot, `pct` stores the configuration at snapshot
105bc8f1 161time into a separate snapshot section within the same configuration
8c1189b6 162file. For example, after creating a snapshot called ``testsnapshot'',
105bc8f1
DM
163your configuration file will look like this:
164
165.Container Configuration with Snapshot
166----
167memory: 512
168swap: 512
169parent: testsnaphot
170...
171
172[testsnaphot]
173memory: 512
174swap: 512
175snaptime: 1457170803
176...
177----
178
8c1189b6
FG
179There are a few snapshot related properties like `parent` and
180`snaptime`. The `parent` property is used to store the parent/child
181relationship between snapshots. `snaptime` is the snapshot creation
182time stamp (Unix epoch).
183
7fc230db 184
3f13c1c3
DM
185Guest Operating System Configuration
186~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
187
188We normally try to detect the operating system type inside the
189container, and then modify some files inside the container to make
190them work as expected. Here is a short list of things we do at
191container startup:
192
193set /etc/hostname:: to set the container name
194
a8e99754 195modify /etc/hosts:: to allow lookup of the local hostname
3f13c1c3
DM
196
197network setup:: pass the complete network setup to the container
198
199configure DNS:: pass information about DNS servers
200
a8e99754 201adapt the init system:: for example, fix the number of spawned getty processes
3f13c1c3
DM
202
203set the root password:: when creating a new container
204
205rewrite ssh_host_keys:: so that each container has unique keys
206
a8e99754 207randomize crontab:: so that cron does not start at the same time on all containers
3f13c1c3 208
25535d34
WB
209Changes made by {PVE} are enclosed by comment markers:
210
37638f59
DM
211----
212# --- BEGIN PVE ---
213<data>
214# --- END PVE ---
215----
25535d34 216
37638f59
DM
217Those markers will be inserted at a reasonable location in the
218file. If such a section already exists, it will be updated in place
219and will not be moved.
25535d34 220
37638f59
DM
221Modification of a file can be prevented by adding a `.pve-ignore.`
222file for it. For instance, if the file `/etc/.pve-ignore.hosts`
223exists then the `/etc/hosts` file will not be touched. This can be a
224simple empty file creatd via:
25535d34
WB
225
226 # touch /etc/.pve-ignore.hosts
227
37638f59
DM
228Most modifications are OS dependent, so they differ between different
229distributions and versions. You can completely disable modifications
8c1189b6 230by manually setting the `ostype` to `unmanaged`.
3f13c1c3
DM
231
232OS type detection is done by testing for certain files inside the
233container:
234
8c1189b6 235Ubuntu:: inspect /etc/lsb-release (`DISTRIB_ID=Ubuntu`)
3f13c1c3
DM
236
237Debian:: test /etc/debian_version
238
239Fedora:: test /etc/fedora-release
240
241RedHat or CentOS:: test /etc/redhat-release
242
243ArchLinux:: test /etc/arch-release
244
245Alpine:: test /etc/alpine-release
246
c617d721
WB
247Gentoo:: test /etc/gentoo-release
248
8c1189b6 249NOTE: Container start fails if the configured `ostype` differs from the auto
3f13c1c3
DM
250detected type.
251
a7f36905
DM
252Options
253~~~~~~~
254
255include::pct.conf.5-opts.adoc[]
256
d61bab51
DM
257
258Container Images
259----------------
260
8c1189b6
FG
261Container images, sometimes also referred to as ``templates'' or
262``appliances'', are `tar` archives which contain everything to run a
d61bab51 263container. You can think of it as a tidy container backup. Like most
8c1189b6 264modern container toolkits, `pct` uses those images when you create a
d61bab51
DM
265new container, for example:
266
267 pct create 999 local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz
268
269Proxmox itself ships a set of basic templates for most common
8c1189b6 270operating systems, and you can download them using the `pveam` (short
d61bab51
DM
271for {pve} Appliance Manager) command line utility. You can also
272download https://www.turnkeylinux.org/[TurnKey Linux] containers using
273that tool (or the graphical user interface).
274
3a6fa247
DM
275Our image repositories contain a list of available images, and there
276is a cron job run each day to download that list. You can trigger that
277update manually with:
278
279 pveam update
280
281After that you can view the list of available images using:
282
283 pveam available
284
8c1189b6
FG
285You can restrict this large list by specifying the `section` you are
286interested in, for example basic `system` images:
3a6fa247
DM
287
288.List available system images
289----
290# pveam available --section system
291system archlinux-base_2015-24-29-1_x86_64.tar.gz
292system centos-7-default_20160205_amd64.tar.xz
293system debian-6.0-standard_6.0-7_amd64.tar.gz
294system debian-7.0-standard_7.0-3_amd64.tar.gz
295system debian-8.0-standard_8.0-1_amd64.tar.gz
296system ubuntu-12.04-standard_12.04-1_amd64.tar.gz
297system ubuntu-14.04-standard_14.04-1_amd64.tar.gz
298system ubuntu-15.04-standard_15.04-1_amd64.tar.gz
299system ubuntu-15.10-standard_15.10-1_amd64.tar.gz
300----
301
a8e99754 302Before you can use such a template, you need to download them into one
8c1189b6 303of your storages. You can simply use storage `local` for that
3a6fa247
DM
304purpose. For clustered installations, it is preferred to use a shared
305storage so that all nodes can access those images.
306
307 pveam download local debian-8.0-standard_8.0-1_amd64.tar.gz
308
24f73a63 309You are now ready to create containers using that image, and you can
8c1189b6 310list all downloaded images on storage `local` with:
24f73a63
DM
311
312----
313# pveam list local
314local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz 190.20MB
315----
316
a8e99754 317The above command shows you the full {pve} volume identifiers. They include
24f73a63
DM
318the storage name, and most other {pve} commands can use them. For
319examply you can delete that image later with:
320
321 pveam remove local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz
3a6fa247 322
d61bab51 323
70a42028
DM
324Container Storage
325-----------------
326
327Traditional containers use a very simple storage model, only allowing
328a single mount point, the root file system. This was further
8c1189b6
FG
329restricted to specific file system types like `ext4` and `nfs`.
330Additional mounts are often done by user provided scripts. This turned
a8e99754 331out to be complex and error prone, so we try to avoid that now.
70a42028
DM
332
333Our new LXC based container model is more flexible regarding
334storage. First, you can have more than a single mount point. This
335allows you to choose a suitable storage for each application. For
336example, you can use a relatively slow (and thus cheap) storage for
337the container root file system. Then you can use a second mount point
338to mount a very fast, distributed storage for your database
339application.
340
341The second big improvement is that you can use any storage type
342supported by the {pve} storage library. That means that you can store
8c1189b6
FG
343your containers on local `lvmthin` or `zfs`, shared `iSCSI` storage,
344or even on distributed storage systems like `ceph`. It also enables us
345to use advanced storage features like snapshots and clones. `vzdump`
a8e99754 346can also use the snapshot feature to provide consistent container
70a42028
DM
347backups.
348
349Last but not least, you can also mount local devices directly, or
350mount local directories using bind mounts. That way you can access
351local storage inside containers with zero overhead. Such bind mounts
a8e99754 352also provide an easy way to share data between different containers.
70a42028 353
eeecce95 354
9e44e493
DM
355Mount Points
356~~~~~~~~~~~~
eeecce95 357
01639994
FG
358The root mount point is configured with the `rootfs` property, and you can
359configure up to 10 additional mount points. The corresponding options
360are called `mp0` to `mp9`, and they can contain the following setting:
361
362include::pct-mountpoint-opts.adoc[]
363
9e44e493
DM
364Currently there are basically three types of mount points: storage backed
365mount points, bind mounts and device mounts.
366
4c3b5c77
DM
367.Typical Container `rootfs` configuration
368----
369rootfs: thin1:base-100-disk-1,size=8G
370----
371
372
373Storage backed mount points
374^^^^^^^^^^^^^^^^^^^^^^^^^^^
01639994 375
9e44e493 376Storage backed mount points are managed by the {pve} storage subsystem and come
eeecce95
WB
377in three different flavors:
378
379- Image based: These are raw images containing a single ext4 formatted file
380 system.
381- ZFS Subvolumes: These are technically bind mounts, but with managed storage,
382 and thus allow resizing and snapshotting.
383- Directories: passing `size=0` triggers a special case where instead of a raw
384 image a directory is created.
385
4c3b5c77
DM
386
387Bind mount points
388^^^^^^^^^^^^^^^^^
01639994 389
9baca183
FG
390Bind mounts allow you to access arbitrary directories from your Proxmox VE host
391inside a container. Some potential use cases are:
392
393- Accessing your home directory in the guest
394- Accessing an USB device directory in the guest
acccc49b 395- Accessing an NFS mount from the host in the guest
9baca183 396
eeecce95 397Bind mounts are considered to not be managed by the storage subsystem, so you
9baca183 398cannot make snapshots or deal with quotas from inside the container. With
eeecce95 399unprivileged containers you might run into permission problems caused by the
9baca183
FG
400user mapping and cannot use ACLs.
401
8c1189b6 402NOTE: The contents of bind mount points are not backed up when using `vzdump`.
eeecce95 403
6b707f2c
FG
404WARNING: For security reasons, bind mounts should only be established
405using source directories especially reserved for this purpose, e.g., a
406directory hierarchy under `/mnt/bindmounts`. Never bind mount system
407directories like `/`, `/var` or `/etc` into a container - this poses a
9baca183
FG
408great security risk.
409
410NOTE: The bind mount source path must not contain any symlinks.
411
412For example, to make the directory `/mnt/bindmounts/shared` accessible in the
413container with ID `100` under the path `/shared`, use a configuration line like
8c1189b6
FG
414`mp0: /mnt/bindmounts/shared,mp=/shared` in `/etc/pve/lxc/100.conf`.
415Alternatively, use `pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared` to
9baca183 416achieve the same result.
6b707f2c 417
4c3b5c77
DM
418
419Device mount points
420^^^^^^^^^^^^^^^^^^^
fe154a4f 421
7432d78e
FG
422Device mount points allow to mount block devices of the host directly into the
423container. Similar to bind mounts, device mounts are not managed by {PVE}'s
424storage subsystem, but the `quota` and `acl` options will be honored.
425
426NOTE: Device mount points should only be used under special circumstances. In
427most cases a storage backed mount point offers the same performance and a lot
428more features.
429
8c1189b6 430NOTE: The contents of device mount points are not backed up when using `vzdump`.
01639994 431
4c3b5c77
DM
432
433FUSE mounts
434~~~~~~~~~~~
01639994
FG
435
436WARNING: Because of existing issues in the Linux kernel's freezer
437subsystem the usage of FUSE mounts inside a container is strongly
438advised against, as containers need to be frozen for suspend or
439snapshot mode backups.
fe154a4f 440
01639994
FG
441If FUSE mounts cannot be replaced by other mounting mechanisms or storage
442technologies, it is possible to establish the FUSE mount on the Proxmox host
443and use a bind mount point to make it accessible inside the container.
444
01639994 445
d6ed3622 446Using quotas inside containers
04c569f6 447~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
d6ed3622 448
9e44e493
DM
449Quotas allow to set limits inside a container for the amount of disk
450space that each user can use. This only works on ext4 image based
451storage types and currently does not work with unprivileged
452containers.
d6ed3622 453
9e44e493
DM
454Activating the `quota` option causes the following mount options to be
455used for a mount point:
456`usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0`
d6ed3622 457
9e44e493
DM
458This allows quotas to be used like you would on any other system. You
459can initialize the `/aquota.user` and `/aquota.group` files by running
d6ed3622 460
9e44e493
DM
461----
462quotacheck -cmug /
463quotaon /
464----
d6ed3622 465
166e63d6
FG
466and edit the quotas via the `edquota` command. Refer to the documentation
467of the distribution running inside the container for details.
468
9e44e493
DM
469NOTE: You need to run the above commands for every mount point by passing
470the mount point's path instead of just `/`.
471
d6ed3622 472
6c60aebf 473Using ACLs inside containers
04c569f6 474~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6c60aebf
EK
475
476The standard Posix Access Control Lists are also available inside containers.
477ACLs allow you to set more detailed file ownership than the traditional user/
478group/others model.
d6ed3622 479
04c569f6
DM
480
481Container Network
482-----------------
483
bac8c385 484You can configure up to 10 network interfaces for a single
8c1189b6 485container. The corresponding options are called `net0` to `net9`, and
bac8c385
DM
486they can contain the following setting:
487
488include::pct-network-opts.adoc[]
04c569f6
DM
489
490
51e33128
FG
491Backup and Restore
492------------------
493
2175e37b
FG
494Container Backup
495~~~~~~~~~~~~~~~~
496
8c1189b6
FG
497It is possible to use the `vzdump` tool for container backup. Please
498refer to the `vzdump` manual page for details.
499
51e33128 500
2175e37b
FG
501Restoring Container Backups
502~~~~~~~~~~~~~~~~~~~~~~~~~~~
503
8c1189b6
FG
504Restoring container backups made with `vzdump` is possible using the
505`pct restore` command. By default, `pct restore` will attempt to restore as much
2175e37b
FG
506of the backed up container configuration as possible. It is possible to override
507the backed up configuration by manually setting container options on the command
8c1189b6 508line (see the `pct` manual page for details).
2175e37b 509
8c1189b6 510NOTE: `pvesm extractconfig` can be used to view the backed up configuration
2175e37b
FG
511contained in a vzdump archive.
512
513There are two basic restore modes, only differing by their handling of mount
514points:
515
4c3b5c77 516
8c1189b6
FG
517``Simple'' Restore Mode
518^^^^^^^^^^^^^^^^^^^^^^^
2175e37b
FG
519
520If neither the `rootfs` parameter nor any of the optional `mpX` parameters
521are explicitly set, the mount point configuration from the backed up
522configuration file is restored using the following steps:
523
524. Extract mount points and their options from backup
525. Create volumes for storage backed mount points (on storage provided with the
526`storage` parameter, or default local storage if unset)
527. Extract files from backup archive
528. Add bind and device mount points to restored configuration (limited to root user)
529
530NOTE: Since bind and device mount points are never backed up, no files are
531restored in the last step, but only the configuration options. The assumption
532is that such mount points are either backed up with another mechanism (e.g.,
533NFS space that is bind mounted into many containers), or not intended to be
534backed up at all.
535
536This simple mode is also used by the container restore operations in the web
537interface.
538
4c3b5c77 539
8c1189b6
FG
540``Advanced'' Restore Mode
541^^^^^^^^^^^^^^^^^^^^^^^^^
2175e37b
FG
542
543By setting the `rootfs` parameter (and optionally, any combination of `mpX`
8c1189b6 544parameters), the `pct restore` command is automatically switched into an
2175e37b
FG
545advanced mode. This advanced mode completely ignores the `rootfs` and `mpX`
546configuration options contained in the backup archive, and instead only
547uses the options explicitly provided as parameters.
548
549This mode allows flexible configuration of mount point settings at restore time,
550for example:
551
552* Set target storages, volume sizes and other options for each mount point
553individually
554* Redistribute backed up files according to new mount point scheme
555* Restore to device and/or bind mount points (limited to root user)
556
51e33128 557
8c1189b6 558Managing Containers with `pct`
04c569f6
DM
559------------------------------
560
8c1189b6 561`pct` is the tool to manage Linux Containers on {pve}. You can create
04c569f6
DM
562and destroy containers, and control execution (start, stop, migrate,
563...). You can use pct to set parameters in the associated config file,
564like network configuration or memory limits.
565
566CLI Usage Examples
567~~~~~~~~~~~~~~~~~~
568
569Create a container based on a Debian template (provided you have
570already downloaded the template via the webgui)
571
572 pct create 100 /var/lib/vz/template/cache/debian-8.0-standard_8.0-1_amd64.tar.gz
573
574Start container 100
575
576 pct start 100
577
578Start a login session via getty
579
580 pct console 100
581
582Enter the LXC namespace and run a shell as root user
583
584 pct enter 100
585
586Display the configuration
587
588 pct config 100
589
8c1189b6 590Add a network interface called `eth0`, bridged to the host bridge `vmbr0`,
04c569f6
DM
591set the address and gateway, while it's running
592
593 pct set 100 -net0 name=eth0,bridge=vmbr0,ip=192.168.15.147/24,gw=192.168.15.1
594
595Reduce the memory of the container to 512MB
596
0585f29a
DM
597 pct set 100 -memory 512
598
04c569f6
DM
599
600Files
601------
602
8c1189b6 603`/etc/pve/lxc/<CTID>.conf`::
04c569f6
DM
604
605Configuration file for the container '<CTID>'.
606
607
0c6b782f
DM
608Container Advantages
609--------------------
610
8c1189b6 611* Simple, and fully integrated into {pve}. Setup looks similar to a normal
0c6b782f
DM
612 VM setup.
613
8c1189b6 614** Storage (ZFS, LVM, NFS, Ceph, ...)
0c6b782f 615
8c1189b6 616** Network
0c6b782f 617
8c1189b6 618** Authentication
0c6b782f 619
8c1189b6 620** Cluster
0c6b782f 621
8c1189b6 622* Fast: minimal overhead, as fast as bare metal
0c6b782f 623
8c1189b6 624* High density (perfect for idle workloads)
0c6b782f 625
8c1189b6 626* REST API
0c6b782f 627
8c1189b6 628* Direct hardware access
0c6b782f
DM
629
630
631Technology Overview
632-------------------
633
634- Integrated into {pve} graphical user interface (GUI)
635
636- LXC (https://linuxcontainers.org/)
637
638- cgmanager for cgroup management
639
640- lxcfs to provive containerized /proc file system
641
642- apparmor
643
644- CRIU: for live migration (planned)
645
11f340ff 646- We use latest available kernels (4.4.X)
0c6b782f 647
a8e99754 648- Image based deployment (templates)
0c6b782f
DM
649
650- Container setup from host (Network, DNS, Storage, ...)
651
652
653ifdef::manvolnum[]
654include::pve-copyright.adoc[]
655endif::manvolnum[]
656
657
658
659
660
661
662