docs/specs/ppc-xive.rst

   1 ================================
   2 POWER9 XIVE interrupt controller
   3 ================================
   4
   5 The POWER9 processor comes with a new interrupt controller
   6 architecture, called XIVE as "eXternal Interrupt Virtualization
   7 Engine".
   8
   9 Compared to the previous architecture, the main characteristics of
  10 XIVE are to support a larger number of interrupt sources and to
  11 deliver interrupts directly to virtual processors without hypervisor
  12 assistance. This removes the context switches required for the
  13 delivery process.
  14
  15
  16 XIVE architecture
  17 =================
  18
  19 The XIVE IC is composed of three sub-engines, each taking care of a
  20 processing layer of external interrupts:
  21
  22 - Interrupt Virtualization Source Engine (IVSE), or Source Controller
  23   (SC). These are found in PCI PHBs, in the PSI host bridge
  24   controller, but also inside the main controller for the core IPIs
  25   and other sub-chips (NX, CAP, NPU) of the chip/processor. They are
  26   configured to feed the IVRE with events.
  27 - Interrupt Virtualization Routing Engine (IVRE) or Virtualization
  28   Controller (VC). It handles event coalescing and perform interrupt
  29   routing by matching an event source number with an Event
  30   Notification Descriptor (END).
  31 - Interrupt Virtualization Presentation Engine (IVPE) or Presentation
  32   Controller (PC). It maintains the interrupt context state of each
  33   thread and handles the delivery of the external interrupt to the
  34   thread.
  35
  36 ::
  37
  38                 XIVE Interrupt Controller
  39                 +------------------------------------+      IPIs
  40                 | +---------+ +---------+ +--------+ |    +-------+
  41                 | |IVRE     | |Common Q | |IVPE    |----> | CORES |
  42                 | |     esb | |         | |        |----> |       |
  43                 | |     eas | |  Bridge | |   tctx |----> |       |
  44                 | |SC   end | |         | |    nvt | |    |       |
  45     +------+    | +---------+ +----+----+ +--------+ |    +-+-+-+-+
  46     | RAM  |    +------------------|-----------------+      | | |
  47     |      |                       |                        | | |
  48     |      |                       |                        | | |
  49     |      |  +--------------------v------------------------v-v-v--+    other
  50     |      <--+                     Power Bus                      +--> chips
  51     |  esb |  +---------+-----------------------+------------------+
  52     |  eas |            |                       |
  53     |  end |         +--|------+                |
  54     |  nvt |       +----+----+ |           +----+----+
  55     +------+       |IVSE     | |           |IVSE     |
  56                    |         | |           |         |
  57                    | PQ-bits | |           | PQ-bits |
  58                    | local   |-+           |  in VC  |
  59                    +---------+             +---------+
  60                       PCIe                 NX,NPU,CAPI
  61
  62
  63     PQ-bits: 2 bits source state machine (P:pending Q:queued)
  64     esb: Event State Buffer (Array of PQ bits in an IVSE)
  65     eas: Event Assignment Structure
  66     end: Event Notification Descriptor
  67     nvt: Notification Virtual Target
  68     tctx: Thread interrupt Context registers
  69
  70
  71
  72 XIVE internal tables
  73 --------------------
  74
  75 Each of the sub-engines uses a set of tables to redirect interrupts
  76 from event sources to CPU threads.
  77
  78 ::
  79
  80                                             +-------+
  81     User or O/S                             |  EQ   |
  82         or                          +------>|entries|
  83     Hypervisor                      |       |  ..   |
  84       Memory                        |       +-------+
  85                                     |           ^
  86                                     |           |
  87                +-------------------------------------------------+
  88                                     |           |
  89     Hypervisor      +------+    +---+--+    +---+--+   +------+
  90       Memory        | ESB  |    | EAT  |    | ENDT |   | NVTT |
  91      (skiboot)      +----+-+    +----+-+    +----+-+   +------+
  92                       ^  |        ^  |        ^  |       ^
  93                       |  |        |  |        |  |       |
  94                +-------------------------------------------------+
  95                       |  |        |  |        |  |       |
  96                       |  |        |  |        |  |       |
  97                  +----|--|--------|--|--------|--|-+   +-|-----+    +------+
  98                  |    |  |        |  |        |  | |   | | tctx|    |Thread|
  99      IPI or   ---+    +  v        +  v        +  v |---| +  .. |----->     |
 100     HW events    |                                 |   |       |    |      |
 101                  |             IVRE                |   | IVPE  |    +------+
 102                  +---------------------------------+   +-------+
 103
 104
 105 The IVSE have a 2-bits state machine, P for pending and Q for queued,
 106 for each source that allows events to be triggered. They are stored in
 107 an Event State Buffer (ESB) array and can be controlled by MMIOs.
 108
 109 If the event is let through, the IVRE looks up in the Event Assignment
 110 Structure (EAS) table for an Event Notification Descriptor (END)
 111 configured for the source. Each Event Notification Descriptor defines
 112 a notification path to a CPU and an in-memory Event Queue, in which
 113 will be enqueued an EQ data for the O/S to pull.
 114
 115 The IVPE determines if a Notification Virtual Target (NVT) can handle
 116 the event by scanning the thread contexts of the VCPUs dispatched on
 117 the processor HW threads. It maintains the interrupt context state of
 118 each thread in a NVT table.
 119
 120 XIVE thread interrupt context
 121 -----------------------------
 122
 123 The XIVE presenter can generate four different exceptions to its
 124 HW threads:
 125
 126 - hypervisor exception
 127 - O/S exception
 128 - Event-Based Branch (user level)
 129 - msgsnd (doorbell)
 130
 131 Each exception has a state independent from the others called a Thread
 132 Interrupt Management context. This context is a set of registers which
 133 lets the thread handle priority management and interrupt
 134 acknowledgment among other things. The most important ones being :
 135
 136 - Interrupt Priority Register  (PIPR)
 137 - Interrupt Pending Buffer     (IPB)
 138 - Current Processor Priority   (CPPR)
 139 - Notification Source Register (NSR)
 140
 141 TIMA
 142 ~~~~
 143
 144 The Thread Interrupt Management registers are accessible through a
 145 specific MMIO region, called the Thread Interrupt Management Area
 146 (TIMA), four aligned pages, each exposing a different view of the
 147 registers. First page (page address ending in ``0b00``) gives access
 148 to the entire context and is reserved for the ring 0 view for the
 149 physical thread context. The second (page address ending in ``0b01``)
 150 is for the hypervisor, ring 1 view. The third (page address ending in
 151 ``0b10``) is for the operating system, ring 2 view. The fourth (page
 152 address ending in ``0b11``) is for user level, ring 3 view.
 153
 154 Interrupt flow from an O/S perspective
 155 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 156
 157 After an event data has been enqueued in the O/S Event Queue, the IVPE
 158 raises the bit corresponding to the priority of the pending interrupt
 159 in the register IBP (Interrupt Pending Buffer) to indicate that an
 160 event is pending in one of the 8 priority queues. The Pending
 161 Interrupt Priority Register (PIPR) is also updated using the IPB. This
 162 register represent the priority of the most favored pending
 163 notification.
 164
 165 The PIPR is then compared to the the Current Processor Priority
 166 Register (CPPR). If it is more favored (numerically less than), the
 167 CPU interrupt line is raised and the EO bit of the Notification Source
 168 Register (NSR) is updated to notify the presence of an exception for
 169 the O/S. The O/S acknowledges the interrupt with a special load in the
 170 Thread Interrupt Management Area.
 171
 172 The O/S handles the interrupt and when done, performs an EOI using a
 173 MMIO operation on the ESB management page of the associate source.
 174
 175 Overview of the QEMU models for XIVE
 176 ====================================
 177
 178 The XiveSource models the IVSE in general, internal and external. It
 179 handles the source ESBs and the MMIO interface to control them.
 180
 181 The XiveNotifier is a small helper interface interconnecting the
 182 XiveSource to the XiveRouter.
 183
 184 The XiveRouter is an abstract model acting as a combined IVRE and
 185 IVPE. It routes event notifications using the EAS and END tables to
 186 the IVPE sub-engine which does a CAM scan to find a CPU to deliver the
 187 exception. Storage should be provided by the inheriting classes.
 188
 189 XiveEnDSource is a special source object. It exposes the END ESB MMIOs
 190 of the Event Queues which are used for coalescing event notifications
 191 and for escalation. Not used on the field, only to sync the EQ cache
 192 in OPAL.
 193
 194 Finally, the XiveTCTX contains the interrupt state context of a thread,
 195 four sets of registers, one for each exception that can be delivered
 196 to a CPU. These contexts are scanned by the IVPE to find a matching VP
 197 when a notification is triggered. It also models the Thread Interrupt
 198 Management Area (TIMA), which exposes the thread context registers to
 199 the CPU for interrupt management.