Cluster Wide Setup
~~~~~~~~~~~~~~~~~~
-The cluster wide firewall configuration is stored at:
+The cluster-wide firewall configuration is stored at:
/etc/pve/firewall/cluster.fw
`[OPTIONS]`::
-This is used to set cluster wide firewall options.
+This is used to set cluster-wide firewall options.
include::pve-firewall-cluster-opts.adoc[]
`[RULES]`::
-This sections contains cluster wide firewall rules for all nodes.
+This sections contains cluster-wide firewall rules for all nodes.
`[IPSET <name>]`::
----
[OPTIONS]
-# enable firewall (cluster wide setting, default is disabled)
+# enable firewall (cluster-wide setting, default is disabled)
enable: 1
----
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This IP set applies only to host firewalls (not VM firewalls). Those
-IPs are allowed to do normal management tasks (PVE GUI, VNC, SPICE,
+IPs are allowed to do normal management tasks ({PVE} GUI, VNC, SPICE,
SSH).
The local cluster network is automatically added to this IP set (alias
----
+[[pve_firewall_services_commands]]
Services and Commands
---------------------
* TCP traffic from management hosts to port 3128 for connections to the SPICE
proxy
* TCP traffic from management hosts to port 22 to allow ssh access
-* UDP traffic in the cluster network to port 5404 and 5405 for corosync
+* UDP traffic in the cluster network to ports 5405-5412 for corosync
* UDP multicast traffic in the cluster network
* ICMP traffic type 3 (Destination Unreachable), 4 (congestion control) or 11
(Time Exceeded)
* TCP connections with invalid connection state
* Broadcast, multicast and anycast traffic not related to corosync, i.e., not
- coming through port 5404 or 5405
+ coming through ports 5405-5412
* TCP traffic to port 43
* UDP traffic to ports 135 and 445
* UDP traffic to the port range 137 to 139
system command to see the firewall chains and rules active on your system.
This output is also included in a `System Report`, accessible over a node's
-subscription tab in the web GUI, or through the `pvereport` command line tool.
+subscription tab in the web GUI, or through the `pvereport` command-line tool.
VM/CT incoming/outgoing DROP/REJECT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-------------------------
By default, all logging of traffic filtered by the firewall rules is disabled.
-To enable logging, the `loglevel` for incommig and/or outgoing traffic has to be
+To enable logging, the `loglevel` for incoming and/or outgoing traffic has to be
set in *Firewall* -> *Options*. This can be done for the host as well as for the
VM/CT firewall individually. By this, logging of {PVE}'s standard firewall rules
is enabled and the output can be observed in *Firewall* -> *Log*.
defined for the standard rules in *Firewall* -> *Options*.
While the `loglevel` for each individual rule can be defined or changed easily
-in the WebUI during creation or modification of the rule, it is possible to set
+in the web UI during creation or modification of the rule, it is possible to set
this also via the corresponding `pvesh` API calls.
Further, the log-level can also be set via the firewall configuration file by
appending a `-log <loglevel>` to the selected rule (see
xref:pve_firewall_log_levels[possible log-levels]).
-For example, the following two are ident:
+For example, the following two are identical:
----
IN REJECT -p icmp -log nolog
Suricata IPS integration
~~~~~~~~~~~~~~~~~~~~~~~~
-If you want to use the http://suricata-ids.org/[Suricata IPS]
+If you want to use the https://suricata.io/[Suricata IPS]
(Intrusion Prevention System), it's possible.
Packets will be forwarded to the IPS only after the firewall ACCEPTed
Ports used by {pve}
-------------------
-* Web interface: 8006
-* VNC Web console: 5900-5999
-* SPICE proxy: 3128
-* sshd (used for cluster actions): 22
-* rpcbind: 111
-* corosync multicast (if you run a cluster): 5404, 5405 UDP
+* Web interface: 8006 (TCP, HTTP/1.1 over TLS)
+* VNC Web console: 5900-5999 (TCP, WebSocket)
+* SPICE proxy: 3128 (TCP)
+* sshd (used for cluster actions): 22 (TCP)
+* rpcbind: 111 (UDP)
+* sendmail: 25 (TCP, outgoing)
+* corosync cluster traffic: 5405-5412 UDP
+* live migration (VM memory and local-disk data): 60000-60050 (TCP)
+
+
+nftables
+--------
+
+As an alternative to `pve-firewall` we offer `proxmox-firewall`, which is an
+implementation of the Proxmox VE firewall based on the newer
+https://wiki.nftables.org/wiki-nftables/index.php/What_is_nftables%3F[nftables]
+rather than iptables.
+
+WARNING: `proxmox-firewall` is currently in tech preview. There might be bugs or
+incompatibilies with the original firewall. It is currently not suited for
+production use.
+
+This implementation uses the same configuration files and configuration format,
+so you can use your old configuration when switching. It provides the exact same
+functionality with a few exceptions:
+
+* REJECT is currently not possible for guest traffic (traffic will instead be
+ dropped).
+* Using the `NDP`, `Router Advertisement` or `DHCP` options will *always* create
+ firewall rules, irregardless of your default policy.
+* firewall rules for guests are evaluated even for connections that have
+ conntrack table entries.
+
+
+Installation and Usage
+~~~~~~~~~~~~~~~~~~~~~~
+
+Install the `proxmox-firewall` package:
+
+----
+apt install proxmox-firewall
+----
+
+Enable the nftables backend via the Web UI on your hosts (Host > Firewall >
+Options > nftables), or by enabling it in the configuration file for your hosts
+(`/etc/pve/nodes/<node_name>/host.fw`):
+
+----
+[OPTIONS]
+
+nftables: 1
+----
+
+NOTE: After enabling/disabling `proxmox-firewall`, all running VMs and
+containers need to be restarted for the old/new firewall to work properly.
+
+After setting the `nftables` configuration key, the new `proxmox-firewall`
+service will take over. You can check if the new service is working by
+checking the systemctl status of `proxmox-firewall`:
+
+----
+systemctl status proxmox-firewall
+----
+
+You can also examine the generated ruleset. You can find more information about
+this in the section xref:pve_firewall_nft_helpful_commands[Helpful Commands].
+You should also check whether `pve-firewall` is no longer generating iptables
+rules, you can find the respective commands in the
+xref:pve_firewall_services_commands[Services and Commands] section.
+
+Switching back to the old firewall can be done by simply setting the
+configuration value back to 0 / No.
+
+Usage
+~~~~~
+
+`proxmox-firewall` will create two tables that are managed by the
+`proxmox-firewall` service: `proxmox-firewall` and `proxmox-firewall-guests`. If
+you want to create custom rules that live outside the Proxmox VE firewall
+configuration you can create your own tables to manage your custom firewall
+rules. `proxmox-firewall` will only touch the tables it generates, so you can
+easily extend and modify the behavior of the `proxmox-firewall` by adding your
+own tables.
+
+Instead of using the `pve-firewall` command, the nftables-based firewall uses
+`proxmox-firewall`. It is a systemd service, so you can start and stop it via
+`systemctl`:
+
+----
+systemctl start proxmox-firewall
+systemctl stop proxmox-firewall
+----
+
+Stopping the firewall service will remove all generated rules.
+
+To query the status of the firewall, you can query the status of the systemctl
+service:
+
+----
+systemctl status proxmox-firewall
+----
+
+
+[[pve_firewall_nft_helpful_commands]]
+Helpful Commands
+~~~~~~~~~~~~~~~~
+You can check the generated ruleset via the following command:
+
+----
+nft list ruleset
+----
+
+If you want to debug `proxmox-firewall` you can simply run the daemon in
+foreground with the `RUST_LOG` environment variable set to `trace`. This should
+provide you with detailed debugging output:
+
+----
+RUST_LOG=trace /usr/libexec/proxmox/proxmox-firewall
+----
+
+You can also edit the systemctl service if you want to have detailed output for
+your firewall daemon:
+
+----
+systemctl edit proxmox-firewall
+----
+
+Then you need to add the override for the `RUST_LOG` environment variable:
+
+----
+[Service]
+Environment="RUST_LOG=trace"
+----
+
+This will generate a large amount of logs very quickly, so only use this for
+debugging purposes. Other, less verbose, log levels are `info` and `debug`.
+
+Running in foreground writes the log output to STDERR, so you can redirect it
+with the following command (e.g. for submitting logs to the community forum):
+
+----
+RUST_LOG=trace /usr/libexec/proxmox/proxmox-firewall 2> firewall_log_$(hostname).txt
+----
+
+It can be helpful to trace packet flow through the different chains in order to
+debug firewall rules. This can be achieved by setting `nftrace` to 1 for packets
+that you want to track. It is advisable that you do not set this flag for *all*
+packets, in the example below we only examine ICMP packets.
+
+----
+#!/usr/sbin/nft -f
+table bridge tracebridge
+delete table bridge tracebridge
+
+table bridge tracebridge {
+ chain trace {
+ meta l4proto icmp meta nftrace set 1
+ }
+
+ chain prerouting {
+ type filter hook prerouting priority -350; policy accept;
+ jump trace
+ }
+
+ chain postrouting {
+ type filter hook postrouting priority -350; policy accept;
+ jump trace
+ }
+}
+----
+
+Saving this file, making it executable, and then running it once will create the
+respective tracing chains. You can then inspect the tracing output via the
+Proxmox VE Web UI (Firewall > Log) or via `nft monitor trace`.
+
+The above example traces traffic on all bridges, which is usually where guest
+traffic flows through. If you want to examine host traffic, create those chains
+in the `inet` table instead of the `bridge` table.
+
+NOTE: Be aware that this can generate a *lot* of log spam and slow down the
+performance of your networking stack significantly.
+
+You can remove the tracing rules via running the following command:
+
+----
+nft delete table bridge tracebridge
+----
ifdef::manvolnum[]
projects / pve-docs.git / blobdiff