[mirror_ubuntu-jammy-kernel.git] / Documentation / networking / devlink-trap.rst

.. SPDX-License-Identifier: GPL-2.0

============
Devlink Trap
============

Background
==========

Devices capable of offloading the kernel's datapath and perform functions such
as bridging and routing must also be able to send specific packets to the
kernel (i.e., the CPU) for processing.

For example, a device acting as a multicast-aware bridge must be able to send
IGMP membership reports to the kernel for processing by the bridge module.
Without processing such packets, the bridge module could never populate its
MDB.

As another example, consider a device acting as router which has received an IP
packet with a TTL of 1. Upon routing the packet the device must send it to the
kernel so that it will route it as well and generate an ICMP Time Exceeded
error datagram. Without letting the kernel route such packets itself, utilities
such as ``traceroute`` could never work.

The fundamental ability of sending certain packets to the kernel for processing
is called "packet trapping".

Overview
========

The ``devlink-trap`` mechanism allows capable device drivers to register their
supported packet traps with ``devlink`` and report trapped packets to
``devlink`` for further analysis.

Upon receiving trapped packets, ``devlink`` will perform a per-trap packets and
bytes accounting and potentially report the packet to user space via a netlink
event along with all the provided metadata (e.g., trap reason, timestamp, input
port). This is especially useful for drop traps (see :ref:`Trap-Types`)
as it allows users to obtain further visibility into packet drops that would
otherwise be invisible.

The following diagram provides a general overview of ``devlink-trap``::

                                    Netlink event: Packet w/ metadata
                                                   Or a summary of recent drops
                                  ^
                                  |
         Userspace                |
        +---------------------------------------------------+
         Kernel                   |
                                  |
                          +-------+--------+
                          |                |
                          |  drop_monitor  |
                          |                |
                          +-------^--------+
                                  |
                                  |
                                  |
                             +----+----+
                             |         |      Kernel's Rx path
                             | devlink |      (non-drop traps)
                             |         |
                             +----^----+      ^
                                  |           |
                                  +-----------+
                                  |
                          +-------+-------+
                          |               |
                          | Device driver |
                          |               |
                          +-------^-------+
         Kernel                   |
        +---------------------------------------------------+
         Hardware                 |
                                  | Trapped packet
                                  |
                               +--+---+
                               |      |
                               | ASIC |
                               |      |
                               +------+

.. _Trap-Types:

Trap Types
==========

The ``devlink-trap`` mechanism supports the following packet trap types:

  * ``drop``: Trapped packets were dropped by the underlying device. Packets
    are only processed by ``devlink`` and not injected to the kernel's Rx path.
    The trap action (see :ref:`Trap-Actions`) can be changed.
  * ``exception``: Trapped packets were not forwarded as intended by the
    underlying device due to an exception (e.g., TTL error, missing neighbour
    entry) and trapped to the control plane for resolution. Packets are
    processed by ``devlink`` and injected to the kernel's Rx path. Changing the
    action of such traps is not allowed, as it can easily break the control
    plane.

.. _Trap-Actions:

Trap Actions
============

The ``devlink-trap`` mechanism supports the following packet trap actions:

  * ``trap``: The sole copy of the packet is sent to the CPU.
  * ``drop``: The packet is dropped by the underlying device and a copy is not
    sent to the CPU.

Generic Packet Traps
====================

Generic packet traps are used to describe traps that trap well-defined packets
or packets that are trapped due to well-defined conditions (e.g., TTL error).
Such traps can be shared by multiple device drivers and their description must
be added to the following table:

.. list-table:: List of Generic Packet Traps
   :widths: 5 5 90

   * - Name
     - Type
     - Description
   * - ``source_mac_is_multicast``
     - ``drop``
     - Traps incoming packets that the device decided to drop because of a
       multicast source MAC
   * - ``vlan_tag_mismatch``
     - ``drop``
     - Traps incoming packets that the device decided to drop in case of VLAN
       tag mismatch: The ingress bridge port is not configured with a PVID and
       the packet is untagged or prio-tagged
   * - ``ingress_vlan_filter``
     - ``drop``
     - Traps incoming packets that the device decided to drop in case they are
       tagged with a VLAN that is not configured on the ingress bridge port
   * - ``ingress_spanning_tree_filter``
     - ``drop``
     - Traps incoming packets that the device decided to drop in case the STP
       state of the ingress bridge port is not "forwarding"
   * - ``port_list_is_empty``
     - ``drop``
     - Traps packets that the device decided to drop in case they need to be
       flooded (e.g., unknown unicast, unregistered multicast) and there are
       no ports the packets should be flooded to
   * - ``port_loopback_filter``
     - ``drop``
     - Traps packets that the device decided to drop in case after layer 2
       forwarding the only port from which they should be transmitted through
       is the port from which they were received
   * - ``blackhole_route``
     - ``drop``
     - Traps packets that the device decided to drop in case they hit a
       blackhole route
   * - ``ttl_value_is_too_small``
     - ``exception``
     - Traps unicast packets that should be forwarded by the device whose TTL
       was decremented to 0 or less
   * - ``tail_drop``
     - ``drop``
     - Traps packets that the device decided to drop because they could not be
       enqueued to a transmission queue which is full
   * - ``non_ip``
     - ``drop``
     - Traps packets that the device decided to drop because they need to
       undergo a layer 3 lookup, but are not IP or MPLS packets
   * - ``uc_dip_over_mc_dmac``
     - ``drop``
     - Traps packets that the device decided to drop because they need to be
       routed and they have a unicast destination IP and a multicast destination
       MAC
   * - ``dip_is_loopback_address``
     - ``drop``
     - Traps packets that the device decided to drop because they need to be
       routed and their destination IP is the loopback address (i.e., 127.0.0.0/8
       and ::1/128)
   * - ``sip_is_mc``
     - ``drop``
     - Traps packets that the device decided to drop because they need to be
       routed and their source IP is multicast (i.e., 224.0.0.0/8 and ff::/8)
   * - ``sip_is_loopback_address``
     - ``drop``
     - Traps packets that the device decided to drop because they need to be
       routed and their source IP is the loopback address (i.e., 127.0.0.0/8 and ::1/128)
   * - ``ip_header_corrupted``
     - ``drop``
     - Traps packets that the device decided to drop because they need to be
       routed and their IP header is corrupted: wrong checksum, wrong IP version
       or too short Internet Header Length (IHL)
   * - ``ipv4_sip_is_limited_bc``
     - ``drop``
     - Traps packets that the device decided to drop because they need to be
       routed and their source IP is limited broadcast (i.e., 255.255.255.255/32)
   * - ``ipv6_mc_dip_reserved_scope``
     - ``drop``
     - Traps IPv6 packets that the device decided to drop because they need to
       be routed and their IPv6 multicast destination IP has a reserved scope
       (i.e., ffx0::/16)
   * - ``ipv6_mc_dip_interface_local_scope``
     - ``drop``
     - Traps IPv6 packets that the device decided to drop because they need to
       be routed and their IPv6 multicast destination IP has an interface-local scope
       (i.e., ffx1::/16)
   * - ``mtu_value_is_too_small``
     - ``exception``
     - Traps packets that should have been routed by the device, but were bigger
       than the MTU of the egress interface
   * - ``unresolved_neigh``
     - ``exception``
     - Traps packets that did not have a matching IP neighbour after routing
   * - ``mc_reverse_path_forwarding``
     - ``exception``
     - Traps multicast IP packets that failed reverse-path forwarding (RPF)
       check during multicast routing
   * - ``reject_route``
     - ``exception``
     - Traps packets that hit reject routes (i.e., "unreachable", "prohibit")
   * - ``ipv4_lpm_miss``
     - ``exception``
     - Traps unicast IPv4 packets that did not match any route
   * - ``ipv6_lpm_miss``
     - ``exception``
     - Traps unicast IPv6 packets that did not match any route

Driver-specific Packet Traps
============================

Device drivers can register driver-specific packet traps, but these must be
clearly documented. Such traps can correspond to device-specific exceptions and
help debug packet drops caused by these exceptions. The following list includes
links to the description of driver-specific traps registered by various device
drivers:

  * :doc:`/devlink-trap-netdevsim`

Generic Packet Trap Groups
==========================

Generic packet trap groups are used to aggregate logically related packet
traps. These groups allow the user to batch operations such as setting the trap
action of all member traps. In addition, ``devlink-trap`` can report aggregated
per-group packets and bytes statistics, in case per-trap statistics are too
narrow. The description of these groups must be added to the following table:

.. list-table:: List of Generic Packet Trap Groups
   :widths: 10 90

   * - Name
     - Description
   * - ``l2_drops``
     - Contains packet traps for packets that were dropped by the device during
       layer 2 forwarding (i.e., bridge)
   * - ``l3_drops``
     - Contains packet traps for packets that were dropped by the device or hit
       an exception (e.g., TTL error) during layer 3 forwarding
   * - ``buffer_drops``
     - Contains packet traps for packets that were dropped by the device due to
       an enqueue decision

Testing
=======

See ``tools/testing/selftests/drivers/net/netdevsim/devlink_trap.sh`` for a
test covering the core infrastructure. Test cases should be added for any new
functionality.

Device drivers should focus their tests on device-specific functionality, such
as the triggering of supported packet traps.
Commit	Line	Data
f3047ca0 IS	1	.. SPDX-License-Identifier: GPL-2.0
	2
	3	============
	4	Devlink Trap
	5	============
	6
	7	Background
	8	==========
	9
	10	Devices capable of offloading the kernel's datapath and perform functions such
	11	as bridging and routing must also be able to send specific packets to the
	12	kernel (i.e., the CPU) for processing.
	13
	14	For example, a device acting as a multicast-aware bridge must be able to send
	15	IGMP membership reports to the kernel for processing by the bridge module.
	16	Without processing such packets, the bridge module could never populate its
	17	MDB.
	18
	19	As another example, consider a device acting as router which has received an IP
	20	packet with a TTL of 1. Upon routing the packet the device must send it to the
	21	kernel so that it will route it as well and generate an ICMP Time Exceeded
	22	error datagram. Without letting the kernel route such packets itself, utilities
	23	such as ``traceroute`` could never work.
	24
	25	The fundamental ability of sending certain packets to the kernel for processing
	26	is called "packet trapping".
	27
	28	Overview
	29	========
	30
	31	The ``devlink-trap`` mechanism allows capable device drivers to register their
	32	supported packet traps with ``devlink`` and report trapped packets to
	33	``devlink`` for further analysis.
	34
	35	Upon receiving trapped packets, ``devlink`` will perform a per-trap packets and
	36	bytes accounting and potentially report the packet to user space via a netlink
	37	event along with all the provided metadata (e.g., trap reason, timestamp, input
	38	port). This is especially useful for drop traps (see :ref:`Trap-Types`)
	39	as it allows users to obtain further visibility into packet drops that would
	40	otherwise be invisible.
	41
	42	The following diagram provides a general overview of ``devlink-trap``::
	43
	44	Netlink event: Packet w/ metadata
	45	Or a summary of recent drops
	46	^
	47	\|
	48	Userspace \|
	49	+---------------------------------------------------+
	50	Kernel \|
	51	\|
	52	+-------+--------+
	53	\| \|
	54	\| drop_monitor \|
	55	\| \|
	56	+-------^--------+
	57	\|
	58	\|
	59	\|
	60	+----+----+
	61	\| \| Kernel's Rx path
	62	\| devlink \| (non-drop traps)
	63	\| \|
	64	+----^----+ ^
65	\| \|
66	+-----------+
67	\|
68	+-------+-------+
69	\| \|
70	\| Device driver \|
71	\| \|
72	+-------^-------+
73	Kernel \|
74	+---------------------------------------------------+
75	Hardware \|
76	\| Trapped packet
77	\|
78	+--+---+
79	\| \|
80	\| ASIC \|
81	\| \|
82	+------+
83
84	.. _Trap-Types:
85
86	Trap Types
87	==========
88
89	The ``devlink-trap`` mechanism supports the following packet trap types:
90
91	* ``drop``: Trapped packets were dropped by the underlying device. Packets
92	are only processed by ``devlink`` and not injected to the kernel's Rx path.
93	The trap action (see :ref:`Trap-Actions`) can be changed.
94	* ``exception``: Trapped packets were not forwarded as intended by the
95	underlying device due to an exception (e.g., TTL error, missing neighbour
96	entry) and trapped to the control plane for resolution. Packets are
97	processed by ``devlink`` and injected to the kernel's Rx path. Changing the
98	action of such traps is not allowed, as it can easily break the control
99	plane.
100
101	.. _Trap-Actions:
102
103	Trap Actions
104	============
105
106	The ``devlink-trap`` mechanism supports the following packet trap actions:
107
108	* ``trap``: The sole copy of the packet is sent to the CPU.
109	* ``drop``: The packet is dropped by the underlying device and a copy is not
110	sent to the CPU.
111
112	Generic Packet Traps
113	====================
114
115	Generic packet traps are used to describe traps that trap well-defined packets
116	or packets that are trapped due to well-defined conditions (e.g., TTL error).
117	Such traps can be shared by multiple device drivers and their description must
118	be added to the following table:
119
120	.. list-table:: List of Generic Packet Traps
121	:widths: 5 5 90
122
123	* - Name
124	- Type
125	- Description
126	* - ``source_mac_is_multicast``
127	- ``drop``
128	- Traps incoming packets that the device decided to drop because of a
129	multicast source MAC
130	* - ``vlan_tag_mismatch``
131	- ``drop``
132	- Traps incoming packets that the device decided to drop in case of VLAN
133	tag mismatch: The ingress bridge port is not configured with a PVID and
134	the packet is untagged or prio-tagged
135	* - ``ingress_vlan_filter``
136	- ``drop``
137	- Traps incoming packets that the device decided to drop in case they are
138	tagged with a VLAN that is not configured on the ingress bridge port
139	* - ``ingress_spanning_tree_filter``
140	- ``drop``
141	- Traps incoming packets that the device decided to drop in case the STP
142	state of the ingress bridge port is not "forwarding"
143	* - ``port_list_is_empty``
144	- ``drop``
145	- Traps packets that the device decided to drop in case they need to be
44bde514 IS	146	flooded (e.g., unknown unicast, unregistered multicast) and there are
44bde514 IS	147	no ports the packets should be flooded to
f3047ca0 IS	148	* - ``port_loopback_filter``
	149	- ``drop``
	150	- Traps packets that the device decided to drop in case after layer 2
	151	forwarding the only port from which they should be transmitted through
	152	is the port from which they were received
	153	* - ``blackhole_route``
	154	- ``drop``
	155	- Traps packets that the device decided to drop in case they hit a
	156	blackhole route
	157	* - ``ttl_value_is_too_small``
	158	- ``exception``
	159	- Traps unicast packets that should be forwarded by the device whose TTL
	160	was decremented to 0 or less
	161	* - ``tail_drop``
	162	- ``drop``
	163	- Traps packets that the device decided to drop because they could not be
	164	enqueued to a transmission queue which is full
6896cc4d AC	165	* - ``non_ip``
	166	- ``drop``
	167	- Traps packets that the device decided to drop because they need to
	168	undergo a layer 3 lookup, but are not IP or MPLS packets
	169	* - ``uc_dip_over_mc_dmac``
	170	- ``drop``
	171	- Traps packets that the device decided to drop because they need to be
	172	routed and they have a unicast destination IP and a multicast destination
	173	MAC
	174	* - ``dip_is_loopback_address``
	175	- ``drop``
	176	- Traps packets that the device decided to drop because they need to be
	177	routed and their destination IP is the loopback address (i.e., 127.0.0.0/8
	178	and ::1/128)
	179	* - ``sip_is_mc``
	180	- ``drop``
	181	- Traps packets that the device decided to drop because they need to be
	182	routed and their source IP is multicast (i.e., 224.0.0.0/8 and ff::/8)
	183	* - ``sip_is_loopback_address``
	184	- ``drop``
	185	- Traps packets that the device decided to drop because they need to be
	186	routed and their source IP is the loopback address (i.e., 127.0.0.0/8 and ::1/128)
	187	* - ``ip_header_corrupted``
	188	- ``drop``
	189	- Traps packets that the device decided to drop because they need to be
	190	routed and their IP header is corrupted: wrong checksum, wrong IP version
	191	or too short Internet Header Length (IHL)
	192	* - ``ipv4_sip_is_limited_bc``
	193	- ``drop``
	194	- Traps packets that the device decided to drop because they need to be
	195	routed and their source IP is limited broadcast (i.e., 255.255.255.255/32)
	196	* - ``ipv6_mc_dip_reserved_scope``
	197	- ``drop``
	198	- Traps IPv6 packets that the device decided to drop because they need to
	199	be routed and their IPv6 multicast destination IP has a reserved scope
	200	(i.e., ffx0::/16)
	201	* - ``ipv6_mc_dip_interface_local_scope``
	202	- ``drop``
	203	- Traps IPv6 packets that the device decided to drop because they need to
	204	be routed and their IPv6 multicast destination IP has an interface-local scope
	205	(i.e., ffx1::/16)
3b063ae5 AC	206	* - ``mtu_value_is_too_small``
	207	- ``exception``
	208	- Traps packets that should have been routed by the device, but were bigger
	209	than the MTU of the egress interface
	210	* - ``unresolved_neigh``
	211	- ``exception``
	212	- Traps packets that did not have a matching IP neighbour after routing
	213	* - ``mc_reverse_path_forwarding``
	214	- ``exception``
	215	- Traps multicast IP packets that failed reverse-path forwarding (RPF)
	216	check during multicast routing
	217	* - ``reject_route``
	218	- ``exception``
	219	- Traps packets that hit reject routes (i.e., "unreachable", "prohibit")
	220	* - ``ipv4_lpm_miss``
	221	- ``exception``
	222	- Traps unicast IPv4 packets that did not match any route
	223	* - ``ipv6_lpm_miss``
	224	- ``exception``
	225	- Traps unicast IPv6 packets that did not match any route
f3047ca0	226
9e087457 IS	227	Driver-specific Packet Traps
	228	============================
	229
	230	Device drivers can register driver-specific packet traps, but these must be
	231	clearly documented. Such traps can correspond to device-specific exceptions and
	232	help debug packet drops caused by these exceptions. The following list includes
	233	links to the description of driver-specific traps registered by various device
	234	drivers:
	235
	236	* :doc:`/devlink-trap-netdevsim`
	237
f3047ca0 IS	238	Generic Packet Trap Groups
	239	==========================
	240
	241	Generic packet trap groups are used to aggregate logically related packet
	242	traps. These groups allow the user to batch operations such as setting the trap
	243	action of all member traps. In addition, ``devlink-trap`` can report aggregated
	244	per-group packets and bytes statistics, in case per-trap statistics are too
	245	narrow. The description of these groups must be added to the following table:
	246
	247	.. list-table:: List of Generic Packet Trap Groups
	248	:widths: 10 90
	249
	250	* - Name
	251	- Description
	252	* - ``l2_drops``
	253	- Contains packet traps for packets that were dropped by the device during
	254	layer 2 forwarding (i.e., bridge)
	255	* - ``l3_drops``
	256	- Contains packet traps for packets that were dropped by the device or hit
	257	an exception (e.g., TTL error) during layer 3 forwarding
	258	* - ``buffer_drops``
	259	- Contains packet traps for packets that were dropped by the device due to
	260	an enqueue decision
95766451 IS	261
	262	Testing
	263	=======
	264
	265	See ``tools/testing/selftests/drivers/net/netdevsim/devlink_trap.sh`` for a
	266	test covering the core infrastructure. Test cases should be added for any new
	267	functionality.
	268
	269	Device drivers should focus their tests on device-specific functionality, such
	270	as the triggering of supported packet traps.