[mirror_ubuntu-hirsute-kernel.git] / Documentation / PCI / boot-interrupts.rst

.. SPDX-License-Identifier: GPL-2.0

===============
Boot Interrupts
===============

:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>

Overview
========

On PCI Express, interrupts are represented with either MSI or inbound
interrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a
given Core IO converts the legacy interrupt messages from PCI Express to
MSI interrupts.  If the IO-APIC is disabled (via the mask bits in the
IO-APIC table entries), the messages are routed to the legacy PCH. This
in-band interrupt mechanism was traditionally necessary for systems that
did not support the IO-APIC and for boot. Intel in the past has used the
term "boot interrupts" to describe this mechanism. Further, the PCI Express
protocol describes this in-band legacy wire-interrupt INTx mechanism for
I/O devices to signal PCI-style level interrupts. The subsequent paragraphs
describe problems with the Core IO handling of INTx message routing to the
PCH and mitigation within BIOS and the OS.


Issue
=====

When in-band legacy INTx messages are forwarded to the PCH, they in turn
trigger a new interrupt for which the OS likely lacks a handler. When an
interrupt goes unhandled over time, they are tracked by the Linux kernel as
Spurious Interrupts. The IRQ will be disabled by the Linux kernel after it
reaches a specific count with the error "nobody cared". This disabled IRQ
now prevents valid usage by an existing interrupt which may happen to share
the IRQ line::

  irq 19: nobody cared (try booting with the "irqpoll" option)
  CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1
  Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020
  Call Trace:

  <IRQ>
   ? dump_stack+0x46/0x5e
   ? __report_bad_irq+0x2e/0xb0
   ? note_interrupt+0x242/0x290
   ? nNIKAL100_memoryRead16+0x8/0x10 [nikal]
   ? handle_irq_event_percpu+0x55/0x70
   ? handle_irq_event+0x4f/0x80
   ? handle_fasteoi_irq+0x81/0x180
   ? handle_irq+0x1c/0x30
   ? do_IRQ+0x41/0xd0
   ? common_interrupt+0x84/0x84
  </IRQ>

  handlers:
  irq_default_primary_handler threaded usb_hcd_irq
  Disabling IRQ #19


Conditions
==========

The use of threaded interrupts is the most likely condition to trigger
this problem today. Threaded interrupts may not be reenabled after the IRQ
handler wakes. These "one shot" conditions mean that the threaded interrupt
needs to keep the interrupt line masked until the threaded handler has run.
Especially when dealing with high data rate interrupts, the thread needs to
run to completion; otherwise some handlers will end up in stack overflows
since the interrupt of the issuing device is still active.

Affected Chipsets
=================

The legacy interrupt forwarding mechanism exists today in a number of
devices including but not limited to chipsets from AMD/ATI, Broadcom, and
Intel. Changes made through the mitigations below have been applied to
drivers/pci/quirks.c

Starting with ICX there are no longer any IO-APICs in the Core IO's
devices.  IO-APIC is only in the PCH.  Devices connected to the Core IO's
PCIe Root Ports will use native MSI/MSI-X mechanisms.

Mitigations
===========

The mitigations take the form of PCI quirks. The preference has been to
first identify and make use of a means to disable the routing to the PCH.
In such a case a quirk to disable boot interrupt generation can be
added. [1]_

Intel® 6300ESB I/O Controller Hub
  Alternate Base Address Register:
   BIE: Boot Interrupt Enable

	  ==  ===========================
	  0   Boot interrupt is enabled.
	  1   Boot interrupt is disabled.
	  ==  ===========================

Intel® Sandy Bridge through Sky Lake based Xeon servers:
  Coherent Interface Protocol Interrupt Control
   dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:
	  When this bit is set. Local INTx messages received from the
	  Intel® Quick Data DMA/PCI Express ports are not routed to legacy
	  PCH - they are either converted into MSI via the integrated IO-APIC
	  (if the IO-APIC mask bit is clear in the appropriate entries)
	  or cause no further action (when mask bit is set)

In the absence of a way to directly disable the routing, another approach
has been to make use of PCI Interrupt pin to INTx routing tables for
purposes of redirecting the interrupt handler to the rerouted interrupt
line by default.  Therefore, on chipsets where this INTx routing cannot be
disabled, the Linux kernel will reroute the valid interrupt to its legacy
interrupt. This redirection of the handler will prevent the occurrence of
the spurious interrupt detection which would ordinarily disable the IRQ
line due to excessive unhandled counts. [2]_

The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or
disable) the redirection of the interrupt handler to the PCH interrupt
line. The option can be overridden by either pci=ioapicreroute or
pci=noioapicreroute. [3]_


More Documentation
==================

There is an overview of the legacy interrupt handling in several datasheets
(6300ESB and 6700PXH below). While largely the same, it provides insight
into the evolution of its handling with chipsets.

Example of disabling of the boot interrupt
------------------------------------------

      - Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)
	5.7.3 Boot Interrupt
	https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf

      - Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families
	Datasheet - Volume 2: Registers (Document # 330784-003)
	6.6.41 cipintrc Coherent Interface Protocol Interrupt Control
	https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf

Example of handler rerouting
----------------------------

      - Intel® 6700PXH 64-bit PCI Hub (Document # 302628)
	2.15.2 PCI Express Legacy INTx Support and Boot Interrupt
	https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf


If you have any legacy PCI interrupt questions that aren't answered, email me.

Cheers,
    Sean V Kelley
    sean.v.kelley@linux.intel.com

.. [1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/
.. [2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/
.. [3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/
Commit	Line	Data
06320ced SK	1	.. SPDX-License-Identifier: GPL-2.0
	2
	3	===============
	4	Boot Interrupts
	5	===============
	6
	7	:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>
	8
	9	Overview
	10	========
	11
	12	On PCI Express, interrupts are represented with either MSI or inbound
	13	interrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a
	14	given Core IO converts the legacy interrupt messages from PCI Express to
	15	MSI interrupts. If the IO-APIC is disabled (via the mask bits in the
	16	IO-APIC table entries), the messages are routed to the legacy PCH. This
	17	in-band interrupt mechanism was traditionally necessary for systems that
	18	did not support the IO-APIC and for boot. Intel in the past has used the
	19	term "boot interrupts" to describe this mechanism. Further, the PCI Express
	20	protocol describes this in-band legacy wire-interrupt INTx mechanism for
	21	I/O devices to signal PCI-style level interrupts. The subsequent paragraphs
	22	describe problems with the Core IO handling of INTx message routing to the
	23	PCH and mitigation within BIOS and the OS.
	24
	25
	26	Issue
	27	=====
	28
	29	When in-band legacy INTx messages are forwarded to the PCH, they in turn
	30	trigger a new interrupt for which the OS likely lacks a handler. When an
	31	interrupt goes unhandled over time, they are tracked by the Linux kernel as
	32	Spurious Interrupts. The IRQ will be disabled by the Linux kernel after it
	33	reaches a specific count with the error "nobody cared". This disabled IRQ
	34	now prevents valid usage by an existing interrupt which may happen to share
a588332f	35	the IRQ line::
06320ced SK	36
	37	irq 19: nobody cared (try booting with the "irqpoll" option)
	38	CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1
	39	Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020
	40	Call Trace:
a588332f	41
06320ced SK	42	<IRQ>
	43	? dump_stack+0x46/0x5e
	44	? __report_bad_irq+0x2e/0xb0
	45	? note_interrupt+0x242/0x290
	46	? nNIKAL100_memoryRead16+0x8/0x10 [nikal]
	47	? handle_irq_event_percpu+0x55/0x70
	48	? handle_irq_event+0x4f/0x80
	49	? handle_fasteoi_irq+0x81/0x180
	50	? handle_irq+0x1c/0x30
	51	? do_IRQ+0x41/0xd0
	52	? common_interrupt+0x84/0x84
	53	</IRQ>
	54
	55	handlers:
	56	irq_default_primary_handler threaded usb_hcd_irq
	57	Disabling IRQ #19
	58
	59
	60	Conditions
	61	==========
	62
	63	The use of threaded interrupts is the most likely condition to trigger
	64	this problem today. Threaded interrupts may not be reenabled after the IRQ
	65	handler wakes. These "one shot" conditions mean that the threaded interrupt
	66	needs to keep the interrupt line masked until the threaded handler has run.
	67	Especially when dealing with high data rate interrupts, the thread needs to
	68	run to completion; otherwise some handlers will end up in stack overflows
	69	since the interrupt of the issuing device is still active.
	70
	71	Affected Chipsets
	72	=================
	73
	74	The legacy interrupt forwarding mechanism exists today in a number of
	75	devices including but not limited to chipsets from AMD/ATI, Broadcom, and
	76	Intel. Changes made through the mitigations below have been applied to
	77	drivers/pci/quirks.c
	78
	79	Starting with ICX there are no longer any IO-APICs in the Core IO's
	80	devices. IO-APIC is only in the PCH. Devices connected to the Core IO's
	81	PCIe Root Ports will use native MSI/MSI-X mechanisms.
	82
	83	Mitigations
	84	===========
	85
	86	The mitigations take the form of PCI quirks. The preference has been to
	87	first identify and make use of a means to disable the routing to the PCH.
	88	In such a case a quirk to disable boot interrupt generation can be
a588332f	89	added. [1]_
06320ced	90
a588332f	91	Intel® 6300ESB I/O Controller Hub
06320ced SK	92	Alternate Base Address Register:
06320ced SK	93	BIE: Boot Interrupt Enable
06320ced	94
a588332f MCC	95	== ===========================
	96	0 Boot interrupt is enabled.
	97	1 Boot interrupt is disabled.
	98	== ===========================
	99
	100	Intel® Sandy Bridge through Sky Lake based Xeon servers:
06320ced SK	101	Coherent Interface Protocol Interrupt Control
	102	dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:
	103	When this bit is set. Local INTx messages received from the
	104	Intel® Quick Data DMA/PCI Express ports are not routed to legacy
	105	PCH - they are either converted into MSI via the integrated IO-APIC
	106	(if the IO-APIC mask bit is clear in the appropriate entries)
	107	or cause no further action (when mask bit is set)
	108
	109	In the absence of a way to directly disable the routing, another approach
	110	has been to make use of PCI Interrupt pin to INTx routing tables for
	111	purposes of redirecting the interrupt handler to the rerouted interrupt
	112	line by default. Therefore, on chipsets where this INTx routing cannot be
	113	disabled, the Linux kernel will reroute the valid interrupt to its legacy
	114	interrupt. This redirection of the handler will prevent the occurrence of
	115	the spurious interrupt detection which would ordinarily disable the IRQ
a588332f	116	line due to excessive unhandled counts. [2]_
06320ced SK	117
	118	The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or
	119	disable) the redirection of the interrupt handler to the PCH interrupt
	120	line. The option can be overridden by either pci=ioapicreroute or
a588332f	121	pci=noioapicreroute. [3]_
06320ced SK	122
	123
	124	More Documentation
	125	==================
	126
	127	There is an overview of the legacy interrupt handling in several datasheets
	128	(6300ESB and 6700PXH below). While largely the same, it provides insight
	129	into the evolution of its handling with chipsets.
	130
	131	Example of disabling of the boot interrupt
	132	------------------------------------------
	133
a588332f	134	- Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)
06320ced SK	135	5.7.3 Boot Interrupt
	136	https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf
	137
a588332f MCC	138	- Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families
a588332f MCC	139	Datasheet - Volume 2: Registers (Document # 330784-003)
06320ced SK	140	6.6.41 cipintrc Coherent Interface Protocol Interrupt Control
	141	https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf
	142
	143	Example of handler rerouting
	144	----------------------------
	145
a588332f	146	- Intel® 6700PXH 64-bit PCI Hub (Document # 302628)
06320ced SK	147	2.15.2 PCI Express Legacy INTx Support and Boot Interrupt
	148	https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf
	149
	150
	151	If you have any legacy PCI interrupt questions that aren't answered, email me.
	152
	153	Cheers,
	154	Sean V Kelley
	155	sean.v.kelley@linux.intel.com
	156
a588332f MCC	157	.. [1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/
	158	.. [2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/
	159	.. [3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/