1 [[chapter_pve_firewall]]
10 pve-firewall - PVE Firewall Daemon
16 include::pve-firewall.8-synopsis.adoc[]
31 {pve} Firewall provides an easy way to protect your IT
32 infrastructure. You can setup firewall rules for all hosts
33 inside a cluster, or define rules for virtual machines and
34 containers. Features like firewall macros, security groups, IP sets
35 and aliases help to make that task easier.
37 While all configuration is stored on the cluster file system, the
38 `iptables`-based firewall service runs on each cluster node, and thus provides
39 full isolation between virtual machines. The distributed nature of
40 this system also provides much higher bandwidth than a central
43 The firewall has full support for IPv4 and IPv6. IPv6 support is fully
44 transparent, and we filter traffic for both protocols by default. So
45 there is no need to maintain a different set of rules for IPv6.
51 The Proxmox VE firewall groups the network into the following logical zones:
55 Traffic from/to a cluster node
59 Traffic from/to a specific VM
61 For each zone, you can define firewall rules for incoming and/or
68 All firewall related configuration is stored on the proxmox cluster
69 file system. So those files are automatically distributed to all
70 cluster nodes, and the `pve-firewall` service updates the underlying
71 `iptables` rules automatically on changes.
73 You can configure anything using the GUI (i.e. *Datacenter* -> *Firewall*,
74 or on a *Node* -> *Firewall*), or you can edit the configuration files
75 directly using your preferred editor.
77 Firewall configuration files contain sections of key-value
78 pairs. Lines beginning with a `#` and blank lines are considered
79 comments. Sections start with a header line containing the section
80 name enclosed in `[` and `]`.
83 [[pve_firewall_cluster_wide_setup]]
87 The cluster-wide firewall configuration is stored at:
89 /etc/pve/firewall/cluster.fw
91 The configuration can contain the following sections:
95 This is used to set cluster-wide firewall options.
97 include::pve-firewall-cluster-opts.adoc[]
101 This sections contains cluster-wide firewall rules for all nodes.
105 Cluster wide IP set definitions.
109 Cluster wide security group definitions.
113 Cluster wide Alias definitions.
116 Enabling the Firewall
117 ^^^^^^^^^^^^^^^^^^^^^
119 The firewall is completely disabled by default, so you need to
120 set the enable option here:
124 # enable firewall (cluster-wide setting, default is disabled)
128 IMPORTANT: If you enable the firewall, traffic to all hosts is blocked by
129 default. Only exceptions is WebGUI(8006) and ssh(22) from your local
132 If you want to administrate your {pve} hosts from remote, you
133 need to create rules to allow traffic from those remote IPs to the web
134 GUI (port 8006). You may also want to allow ssh (port 22), and maybe
137 TIP: Please open a SSH connection to one of your {PVE} hosts before
138 enabling the firewall. That way you still have access to the host if
139 something goes wrong .
141 To simplify that task, you can instead create an IPSet called
142 ``management'', and add all remote IPs there. This creates all required
143 firewall rules to access the GUI from remote.
146 [[pve_firewall_host_specific_configuration]]
147 Host Specific Configuration
148 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
150 Host related configuration is read from:
152 /etc/pve/nodes/<nodename>/host.fw
154 This is useful if you want to overwrite rules from `cluster.fw`
155 config. You can also increase log verbosity, and set netfilter related
156 options. The configuration can contain the following sections:
160 This is used to set host related firewall options.
162 include::pve-firewall-host-opts.adoc[]
166 This sections contains host specific firewall rules.
168 [[pve_firewall_vm_container_configuration]]
169 VM/Container Configuration
170 ~~~~~~~~~~~~~~~~~~~~~~~~~~
172 VM firewall configuration is read from:
174 /etc/pve/firewall/<VMID>.fw
176 and contains the following data:
180 This is used to set VM/Container related firewall options.
182 include::pve-firewall-vm-opts.adoc[]
186 This sections contains VM/Container firewall rules.
194 IP Alias definitions.
197 Enabling the Firewall for VMs and Containers
198 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
200 Each virtual network device has its own firewall enable flag. So you
201 can selectively enable the firewall for each interface. This is
202 required in addition to the general firewall `enable` option.
208 Firewall rules consists of a direction (`IN` or `OUT`) and an
209 action (`ACCEPT`, `DENY`, `REJECT`). You can also specify a macro
210 name. Macros contain predefined sets of rules and options. Rules can be
211 disabled by prefixing them with `|`.
213 .Firewall rules syntax
217 DIRECTION ACTION [OPTIONS]
218 |DIRECTION ACTION [OPTIONS] # disabled rule
220 DIRECTION MACRO(ACTION) [OPTIONS] # use predefined macro
223 The following options can be used to refine rule matches.
225 include::pve-firewall-rules-opts.adoc[]
227 Here are some examples:
231 IN SSH(ACCEPT) -i net0
232 IN SSH(ACCEPT) -i net0 # a comment
233 IN SSH(ACCEPT) -i net0 -source 192.168.2.192 # only allow SSH from 192.168.2.192
234 IN SSH(ACCEPT) -i net0 -source 10.0.0.1-10.0.0.10 # accept SSH for IP range
235 IN SSH(ACCEPT) -i net0 -source 10.0.0.1,10.0.0.2,10.0.0.3 #accept ssh for IP list
236 IN SSH(ACCEPT) -i net0 -source +mynetgroup # accept ssh for ipset mynetgroup
237 IN SSH(ACCEPT) -i net0 -source myserveralias #accept ssh for alias myserveralias
239 |IN SSH(ACCEPT) -i net0 # disabled rule
241 IN DROP # drop all incoming packages
242 OUT ACCEPT # accept all outgoing packages
246 [[pve_firewall_security_groups]]
250 A security group is a collection of rules, defined at cluster level, which
251 can be used in all VMs' rules. For example you can define a group named
252 ``webserver'' with rules to open the 'http' and 'https' ports.
255 # /etc/pve/firewall/cluster.fw
258 IN ACCEPT -p tcp -dport 80
259 IN ACCEPT -p tcp -dport 443
262 Then, you can add this group to a VM's firewall
265 # /etc/pve/firewall/<VMID>.fw
271 [[pve_firewall_ip_aliases]]
275 IP Aliases allow you to associate IP addresses of networks with a
276 name. You can then refer to those names:
278 * inside IP set definitions
279 * in `source` and `dest` properties of firewall rules
282 Standard IP Alias `local_network`
283 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
285 This alias is automatically defined. Please use the following command
286 to see assigned values:
289 # pve-firewall localnet
290 local hostname: example
291 local IP address: 192.168.2.100
292 network auto detect: 192.168.0.0/20
293 using detected local_network: 192.168.0.0/20
296 The firewall automatically sets up rules to allow everything needed
297 for cluster communication (corosync, API, SSH) using this alias.
299 The user can overwrite these values in the `cluster.fw` alias
300 section. If you use a single host on a public network, it is better to
301 explicitly assign the local IP address
304 # /etc/pve/firewall/cluster.fw
306 local_network 1.2.3.4 # use the single IP address
309 [[pve_firewall_ip_sets]]
313 IP sets can be used to define groups of networks and hosts. You can
314 refer to them with `+name` in the firewall rules' `source` and `dest`
317 The following example allows HTTP traffic from the `management` IP
320 IN HTTP(ACCEPT) -source +management
323 Standard IP set `management`
324 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
326 This IP set applies only to host firewalls (not VM firewalls). Those
327 IPs are allowed to do normal management tasks ({PVE} GUI, VNC, SPICE,
330 The local cluster network is automatically added to this IP set (alias
331 `cluster_network`), to enable inter-host cluster
332 communication. (multicast,ssh,...)
335 # /etc/pve/firewall/cluster.fw
343 Standard IP set `blacklist`
344 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
346 Traffic from these IPs is dropped by every host's and VM's firewall.
349 # /etc/pve/firewall/cluster.fw
357 [[pve_firewall_ipfilter_section]]
358 Standard IP set `ipfilter-net*`
359 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
361 These filters belong to a VM's network interface and are mainly used to prevent
362 IP spoofing. If such a set exists for an interface then any outgoing traffic
363 with a source IP not matching its interface's corresponding ipfilter set will
366 For containers with configured IP addresses these sets, if they exist (or are
367 activated via the general `IP Filter` option in the VM's firewall's *options*
368 tab), implicitly contain the associated IP addresses.
370 For both virtual machines and containers they also implicitly contain the
371 standard MAC-derived IPv6 link-local address in order to allow the neighbor
372 discovery protocol to work.
375 /etc/pve/firewall/<VMID>.fw
377 [IPSET ipfilter-net0] # only allow specified IPs on net0
382 [[pve_firewall_services_commands]]
383 Services and Commands
384 ---------------------
386 The firewall runs two service daemons on each node:
388 * pvefw-logger: NFLOG daemon (ulogd replacement).
389 * pve-firewall: updates iptables rules
391 There is also a CLI command named `pve-firewall`, which can be used to
392 start and stop the firewall service:
397 To get the status use:
399 # pve-firewall status
401 The above command reads and compiles all firewall rules, so you will
402 see warnings if your firewall configuration contains any errors.
404 If you want to see the generated iptables rules you can use:
408 [[pve_firewall_default_rules]]
409 Default firewall rules
410 ----------------------
412 The following traffic is filtered by the default firewall configuration:
414 Datacenter incoming/outgoing DROP/REJECT
415 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
417 If the input or output policy for the firewall is set to DROP or REJECT, the
418 following traffic is still allowed for all {pve} hosts in the cluster:
420 * traffic over the loopback interface
421 * already established connections
422 * traffic using the IGMP protocol
423 * TCP traffic from management hosts to port 8006 in order to allow access to
425 * TCP traffic from management hosts to the port range 5900 to 5999 allowing
426 traffic for the VNC web console
427 * TCP traffic from management hosts to port 3128 for connections to the SPICE
429 * TCP traffic from management hosts to port 22 to allow ssh access
430 * UDP traffic in the cluster network to ports 5405-5412 for corosync
431 * UDP multicast traffic in the cluster network
432 * ICMP traffic type 3 (Destination Unreachable), 4 (congestion control) or 11
435 The following traffic is dropped, but not logged even with logging enabled:
437 * TCP connections with invalid connection state
438 * Broadcast, multicast and anycast traffic not related to corosync, i.e., not
439 coming through ports 5405-5412
440 * TCP traffic to port 43
441 * UDP traffic to ports 135 and 445
442 * UDP traffic to the port range 137 to 139
443 * UDP traffic form source port 137 to port range 1024 to 65535
444 * UDP traffic to port 1900
445 * TCP traffic to port 135, 139 and 445
446 * UDP traffic originating from source port 53
448 The rest of the traffic is dropped or rejected, respectively, and also logged.
449 This may vary depending on the additional options enabled in
450 *Firewall* -> *Options*, such as NDP, SMURFS and TCP flag filtering.
452 [[pve_firewall_iptables_inspect]]
453 Please inspect the output of the
459 system command to see the firewall chains and rules active on your system.
460 This output is also included in a `System Report`, accessible over a node's
461 subscription tab in the web GUI, or through the `pvereport` command-line tool.
463 VM/CT incoming/outgoing DROP/REJECT
464 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
466 This drops or rejects all the traffic to the VMs, with some exceptions for
467 DHCP, NDP, Router Advertisement, MAC and IP filtering depending on the set
468 configuration. The same rules for dropping/rejecting packets are inherited
469 from the datacenter, while the exceptions for accepted incoming/outgoing
470 traffic of the host do not apply.
472 Again, you can use xref:pve_firewall_iptables_inspect[iptables-save (see above)]
473 to inspect all rules and chains applied.
475 Logging of firewall rules
476 -------------------------
478 By default, all logging of traffic filtered by the firewall rules is disabled.
479 To enable logging, the `loglevel` for incoming and/or outgoing traffic has to be
480 set in *Firewall* -> *Options*. This can be done for the host as well as for the
481 VM/CT firewall individually. By this, logging of {PVE}'s standard firewall rules
482 is enabled and the output can be observed in *Firewall* -> *Log*.
483 Further, only some dropped or rejected packets are logged for the standard rules
484 (see xref:pve_firewall_default_rules[default firewall rules]).
486 `loglevel` does not affect how much of the filtered traffic is logged. It
487 changes a `LOGID` appended as prefix to the log output for easier filtering and
490 `loglevel` is one of the following flags:
492 [[pve_firewall_log_levels]]
493 [width="25%", options="header"]
507 A typical firewall log output looks like this:
510 VMID LOGID CHAIN TIMESTAMP POLICY: PACKET_DETAILS
513 In case of the host firewall, `VMID` is equal to 0.
516 Logging of user defined firewall rules
517 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
519 In order to log packets filtered by user-defined firewall rules, it is possible
520 to set a log-level parameter for each rule individually.
521 This allows to log in a fine grained manner and independent of the log-level
522 defined for the standard rules in *Firewall* -> *Options*.
524 While the `loglevel` for each individual rule can be defined or changed easily
525 in the web UI during creation or modification of the rule, it is possible to set
526 this also via the corresponding `pvesh` API calls.
528 Further, the log-level can also be set via the firewall configuration file by
529 appending a `-log <loglevel>` to the selected rule (see
530 xref:pve_firewall_log_levels[possible log-levels]).
532 For example, the following two are identical:
535 IN REJECT -p icmp -log nolog
542 IN REJECT -p icmp -log debug
545 produces a log output flagged with the `debug` level.
554 FTP is an old style protocol which uses port 21 and several other dynamic ports. So you
555 need a rule to accept port 21. In addition, you need to load the `ip_conntrack_ftp` module.
558 modprobe ip_conntrack_ftp
560 and add `ip_conntrack_ftp` to `/etc/modules` (so that it works after a reboot).
563 Suricata IPS integration
564 ~~~~~~~~~~~~~~~~~~~~~~~~
566 If you want to use the https://suricata.io/[Suricata IPS]
567 (Intrusion Prevention System), it's possible.
569 Packets will be forwarded to the IPS only after the firewall ACCEPTed
572 Rejected/Dropped firewall packets don't go to the IPS.
574 Install suricata on proxmox host:
577 # apt-get install suricata
578 # modprobe nfnetlink_queue
581 Don't forget to add `nfnetlink_queue` to `/etc/modules` for next reboot.
583 Then, enable IPS for a specific VM with:
586 # /etc/pve/firewall/<VMID>.fw
593 `ips_queues` will bind a specific cpu queue for this VM.
595 Available queues are defined in
598 # /etc/default/suricata
606 The firewall contains a few IPv6 specific options. One thing to note is that
607 IPv6 does not use the ARP protocol anymore, and instead uses NDP (Neighbor
608 Discovery Protocol) which works on IP level and thus needs IP addresses to
609 succeed. For this purpose link-local addresses derived from the interface's MAC
610 address are used. By default the `NDP` option is enabled on both host and VM
611 level to allow neighbor discovery (NDP) packets to be sent and received.
613 Beside neighbor discovery NDP is also used for a couple of other things, like
614 auto-configuration and advertising routers.
616 By default VMs are allowed to send out router solicitation messages (to query
617 for a router), and to receive router advertisement packets. This allows them to
618 use stateless auto configuration. On the other hand VMs cannot advertise
619 themselves as routers unless the ``Allow Router Advertisement'' (`radv: 1`) option
622 As for the link local addresses required for NDP, there's also an ``IP Filter''
623 (`ipfilter: 1`) option which can be enabled which has the same effect as adding
624 an `ipfilter-net*` ipset for each of the VM's network interfaces containing the
625 corresponding link local addresses. (See the
626 <<pve_firewall_ipfilter_section,Standard IP set `ipfilter-net*`>> section for details.)
632 * Web interface: 8006 (TCP, HTTP/1.1 over TLS)
633 * VNC Web console: 5900-5999 (TCP, WebSocket)
634 * SPICE proxy: 3128 (TCP)
635 * sshd (used for cluster actions): 22 (TCP)
637 * sendmail: 25 (TCP, outgoing)
638 * corosync cluster traffic: 5405-5412 UDP
639 * live migration (VM memory and local-disk data): 60000-60050 (TCP)
645 As an alternative to `pve-firewall` we offer `proxmox-firewall`, which is an
646 implementation of the Proxmox VE firewall based on the newer
647 https://wiki.nftables.org/wiki-nftables/index.php/What_is_nftables%3F[nftables]
648 rather than iptables.
650 WARNING: `proxmox-firewall` is currently in tech preview. There might be bugs or
651 incompatibilies with the original firewall. It is currently not suited for
654 This implementation uses the same configuration files and configuration format,
655 so you can use your old configuration when switching. It provides the exact same
656 functionality with a few exceptions:
658 * REJECT is currently not possible for guest traffic (traffic will instead be
660 * Using the `NDP`, `Router Advertisement` or `DHCP` options will *always* create
661 firewall rules, irregardless of your default policy.
662 * firewall rules for guests are evaluated even for connections that have
663 conntrack table entries.
666 Installation and Usage
667 ~~~~~~~~~~~~~~~~~~~~~~
669 Install the `proxmox-firewall` package:
672 apt install proxmox-firewall
675 Enable the nftables backend via the Web UI on your hosts (Host > Firewall >
676 Options > nftables), or by enabling it in the configuration file for your hosts
677 (`/etc/pve/nodes/<node_name>/host.fw`):
685 NOTE: After enabling/disabling `proxmox-firewall`, all running VMs and
686 containers need to be restarted for the old/new firewall to work properly.
688 After setting the `nftables` configuration key, the new `proxmox-firewall`
689 service will take over. You can check if the new service is working by
690 checking the systemctl status of `proxmox-firewall`:
693 systemctl status proxmox-firewall
696 You can also examine the generated ruleset. You can find more information about
697 this in the section xref:pve_firewall_nft_helpful_commands[Helpful Commands].
698 You should also check whether `pve-firewall` is no longer generating iptables
699 rules, you can find the respective commands in the
700 xref:pve_firewall_services_commands[Services and Commands] section.
702 Switching back to the old firewall can be done by simply setting the
703 configuration value back to 0 / No.
708 `proxmox-firewall` will create two tables that are managed by the
709 `proxmox-firewall` service: `proxmox-firewall` and `proxmox-firewall-guests`. If
710 you want to create custom rules that live outside the Proxmox VE firewall
711 configuration you can create your own tables to manage your custom firewall
712 rules. `proxmox-firewall` will only touch the tables it generates, so you can
713 easily extend and modify the behavior of the `proxmox-firewall` by adding your
716 Instead of using the `pve-firewall` command, the nftables-based firewall uses
717 `proxmox-firewall`. It is a systemd service, so you can start and stop it via
721 systemctl start proxmox-firewall
722 systemctl stop proxmox-firewall
725 Stopping the firewall service will remove all generated rules.
727 To query the status of the firewall, you can query the status of the systemctl
731 systemctl status proxmox-firewall
735 [[pve_firewall_nft_helpful_commands]]
738 You can check the generated ruleset via the following command:
744 If you want to debug `proxmox-firewall` you can simply run the daemon in
745 foreground with the `RUST_LOG` environment variable set to `trace`. This should
746 provide you with detailed debugging output:
749 RUST_LOG=trace /usr/libexec/proxmox/proxmox-firewall
752 You can also edit the systemctl service if you want to have detailed output for
753 your firewall daemon:
756 systemctl edit proxmox-firewall
759 Then you need to add the override for the `RUST_LOG` environment variable:
763 Environment="RUST_LOG=trace"
766 This will generate a large amount of logs very quickly, so only use this for
767 debugging purposes. Other, less verbose, log levels are `info` and `debug`.
769 Running in foreground writes the log output to STDERR, so you can redirect it
770 with the following command (e.g. for submitting logs to the community forum):
773 RUST_LOG=trace /usr/libexec/proxmox/proxmox-firewall 2> firewall_log_$(hostname).txt
776 It can be helpful to trace packet flow through the different chains in order to
777 debug firewall rules. This can be achieved by setting `nftrace` to 1 for packets
778 that you want to track. It is advisable that you do not set this flag for *all*
779 packets, in the example below we only examine ICMP packets.
783 table bridge tracebridge
784 delete table bridge tracebridge
786 table bridge tracebridge {
788 meta l4proto icmp meta nftrace set 1
792 type filter hook prerouting priority -350; policy accept;
797 type filter hook postrouting priority -350; policy accept;
803 Saving this file, making it executable, and then running it once will create the
804 respective tracing chains. You can then inspect the tracing output via the
805 Proxmox VE Web UI (Firewall > Log) or via `nft monitor trace`.
807 The above example traces traffic on all bridges, which is usually where guest
808 traffic flows through. If you want to examine host traffic, create those chains
809 in the `inet` table instead of the `bridge` table.
811 NOTE: Be aware that this can generate a *lot* of log spam and slow down the
812 performance of your networking stack significantly.
814 You can remove the tracing rules via running the following command:
817 nft delete table bridge tracebridge
826 include::pve-firewall-macros.adoc[]
829 include::pve-copyright.adoc[]