[mirror_ubuntu-artful-kernel.git] / Documentation / x86 / topology.txt

x86 Topology
============

This documents and clarifies the main aspects of x86 topology modelling and
representation in the kernel. Update/change when doing changes to the
respective code.

The architecture-agnostic topology definitions are in
Documentation/cputopology.txt. This file holds x86-specific
differences/specialities which must not necessarily apply to the generic
definitions. Thus, the way to read up on Linux topology on x86 is to start
with the generic one and look at this one in parallel for the x86 specifics.

Needless to say, code should use the generic functions - this file is *only*
here to *document* the inner workings of x86 topology.

Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.

The main aim of the topology facilities is to present adequate interfaces to
code which needs to know/query/use the structure of the running system wrt
threads, cores, packages, etc.

The kernel does not care about the concept of physical sockets because a
socket has no relevance to software. It's an electromechanical component. In
the past a socket always contained a single package (see below), but with the
advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
there might be still references to sockets in the code, but they are of
historical nature and should be cleaned up.

The topology of a system is described in the units of:

    - packages
    - cores
    - threads

* Package:

  Packages contain a number of cores plus shared resources, e.g. DRAM
  controller, shared caches etc.

  AMD nomenclature for package is 'Node'.

  Package-related topology information in the kernel:

  - cpuinfo_x86.x86_max_cores:

    The number of cores in a package. This information is retrieved via CPUID.

  - cpuinfo_x86.phys_proc_id:

    The physical ID of the package. This information is retrieved via CPUID
    and deduced from the APIC IDs of the cores in the package.

  - cpuinfo_x86.logical_id:

    The logical ID of the package. As we do not trust BIOSes to enumerate the
    packages in a consistent way, we introduced the concept of logical package
    ID so we can sanely calculate the number of maximum possible packages in
    the system and have the packages enumerated linearly.

  - topology_max_packages():

    The maximum possible number of packages in the system. Helpful for per
    package facilities to preallocate per package information.

  - cpu_llc_id:

    A per-CPU variable containing:
    - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
    Cache

    - On AMD, the Node ID or Core Complex ID containing the Last Level
    Cache. In general, it is a number identifying an LLC uniquely on the
    system.

* Cores:

  A core consists of 1 or more threads. It does not matter whether the threads
  are SMT- or CMT-type threads.

  AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
  "core".

  Core-related topology information in the kernel:

  - smp_num_siblings:

    The number of threads in a core. The number of threads in a package can be
    calculated by:

	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings


* Threads:

  A thread is a single scheduling unit. It's the equivalent to a logical Linux
  CPU.

  AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
  uses "thread".

  Thread-related topology information in the kernel:

  - topology_core_cpumask():

    The cpumask contains all online threads in the package to which a thread
    belongs.

    The number of online threads is also printed in /proc/cpuinfo "siblings."

  - topology_sibling_mask():

    The cpumask contains all online threads in the core to which a thread
    belongs.

   - topology_logical_package_id():

    The logical package ID to which a thread belongs.

   - topology_physical_package_id():

    The physical package ID to which a thread belongs.

   - topology_core_id();

    The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
    "core_id."


System topology examples

Note:

The alternative Linux CPU enumeration depends on how the BIOS enumerates the
threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
the same whether threads are enabled or not. That's merely an implementation
detail and has no practical impact.

1) Single Package, Single Core

   [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0

2) Single Package, Dual Core

   a) One thread per core

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
		    -> [core 1] -> [thread 0] -> Linux CPU 1

   b) Two threads per core

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 1
		    -> [core 1] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 3

      Alternative enumeration:

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 2
		    -> [core 1] -> [thread 0] -> Linux CPU 1
				-> [thread 1] -> Linux CPU 3

      AMD nomenclature for CMT systems:

	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
				     -> [Compute Unit Core 1] -> Linux CPU 1
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
				     -> [Compute Unit Core 1] -> Linux CPU 3

4) Dual Package, Dual Core

   a) One thread per core

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
		    -> [core 1] -> [thread 0] -> Linux CPU 1

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
		    -> [core 1] -> [thread 0] -> Linux CPU 3

   b) Two threads per core

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 1
		    -> [core 1] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 3

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
				-> [thread 1] -> Linux CPU 5
		    -> [core 1] -> [thread 0] -> Linux CPU 6
				-> [thread 1] -> Linux CPU 7

      Alternative enumeration:

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 4
		    -> [core 1] -> [thread 0] -> Linux CPU 1
				-> [thread 1] -> Linux CPU 5

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 6
		    -> [core 1] -> [thread 0] -> Linux CPU 3
				-> [thread 1] -> Linux CPU 7

      AMD nomenclature for CMT systems:

	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
				     -> [Compute Unit Core 1] -> Linux CPU 1
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
				     -> [Compute Unit Core 1] -> Linux CPU 3

	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
				     -> [Compute Unit Core 1] -> Linux CPU 5
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
				     -> [Compute Unit Core 1] -> Linux CPU 7
Commit	Line	Data
f7be8610 BP	1	x86 Topology
	2	============
	3
	4	This documents and clarifies the main aspects of x86 topology modelling and
	5	representation in the kernel. Update/change when doing changes to the
	6	respective code.
	7
	8	The architecture-agnostic topology definitions are in
	9	Documentation/cputopology.txt. This file holds x86-specific
	10	differences/specialities which must not necessarily apply to the generic
	11	definitions. Thus, the way to read up on Linux topology on x86 is to start
	12	with the generic one and look at this one in parallel for the x86 specifics.
	13
	14	Needless to say, code should use the generic functions - this file is only
	15	here to document the inner workings of x86 topology.
	16
	17	Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
	18
	19	The main aim of the topology facilities is to present adequate interfaces to
	20	code which needs to know/query/use the structure of the running system wrt
	21	threads, cores, packages, etc.
	22
	23	The kernel does not care about the concept of physical sockets because a
	24	socket has no relevance to software. It's an electromechanical component. In
	25	the past a socket always contained a single package (see below), but with the
	26	advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
	27	there might be still references to sockets in the code, but they are of
	28	historical nature and should be cleaned up.
	29
	30	The topology of a system is described in the units of:
	31
	32	- packages
	33	- cores
	34	- threads
	35
	36	* Package:
	37
	38	Packages contain a number of cores plus shared resources, e.g. DRAM
	39	controller, shared caches etc.
	40
	41	AMD nomenclature for package is 'Node'.
	42
	43	Package-related topology information in the kernel:
	44
	45	- cpuinfo_x86.x86_max_cores:
	46
	47	The number of cores in a package. This information is retrieved via CPUID.
	48
	49	- cpuinfo_x86.phys_proc_id:
	50
	51	The physical ID of the package. This information is retrieved via CPUID
	52	and deduced from the APIC IDs of the cores in the package.
	53
	54	- cpuinfo_x86.logical_id:
	55
	56	The logical ID of the package. As we do not trust BIOSes to enumerate the
	57	packages in a consistent way, we introduced the concept of logical package
	58	ID so we can sanely calculate the number of maximum possible packages in
	59	the system and have the packages enumerated linearly.
	60
	61	- topology_max_packages():
	62
	63	The maximum possible number of packages in the system. Helpful for per
	64	package facilities to preallocate per package information.
65
a268b5f1 BP	66	- cpu_llc_id:
	67
	68	A per-CPU variable containing:
	69	- On Intel, the first APIC ID of the list of CPUs sharing the Last Level
	70	Cache
	71
	72	- On AMD, the Node ID or Core Complex ID containing the Last Level
	73	Cache. In general, it is a number identifying an LLC uniquely on the
	74	system.
f7be8610 BP	75
	76	* Cores:
	77
	78	A core consists of 1 or more threads. It does not matter whether the threads
	79	are SMT- or CMT-type threads.
	80
	81	AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
	82	"core".
	83
	84	Core-related topology information in the kernel:
	85
	86	- smp_num_siblings:
	87
	88	The number of threads in a core. The number of threads in a package can be
	89	calculated by:
	90
	91	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
	92
	93
	94	* Threads:
	95
	96	A thread is a single scheduling unit. It's the equivalent to a logical Linux
	97	CPU.
	98
	99	AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
	100	uses "thread".
	101
	102	Thread-related topology information in the kernel:
	103
	104	- topology_core_cpumask():
	105
	106	The cpumask contains all online threads in the package to which a thread
	107	belongs.
	108
	109	The number of online threads is also printed in /proc/cpuinfo "siblings."
	110
	111	- topology_sibling_mask():
	112
	113	The cpumask contains all online threads in the core to which a thread
	114	belongs.
	115
	116	- topology_logical_package_id():
	117
	118	The logical package ID to which a thread belongs.
	119
	120	- topology_physical_package_id():
	121
	122	The physical package ID to which a thread belongs.
	123
	124	- topology_core_id();
	125
	126	The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
	127	"core_id."
	128
	129
	130
	131	System topology examples
	132
	133	Note:
	134
	135	The alternative Linux CPU enumeration depends on how the BIOS enumerates the
	136	threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
	137	That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
	138	the same whether threads are enabled or not. That's merely an implementation
139	detail and has no practical impact.
140
141	1) Single Package, Single Core
142
143	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
144
145	2) Single Package, Dual Core
146
147	a) One thread per core
148
149	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
150	-> [core 1] -> [thread 0] -> Linux CPU 1
151
152	b) Two threads per core
153
154	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
155	-> [thread 1] -> Linux CPU 1
156	-> [core 1] -> [thread 0] -> Linux CPU 2
157	-> [thread 1] -> Linux CPU 3
158
159	Alternative enumeration:
160
161	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
162	-> [thread 1] -> Linux CPU 2
163	-> [core 1] -> [thread 0] -> Linux CPU 1
164	-> [thread 1] -> Linux CPU 3
165
166	AMD nomenclature for CMT systems:
167
168	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
169	-> [Compute Unit Core 1] -> Linux CPU 1
170	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
171	-> [Compute Unit Core 1] -> Linux CPU 3
172
173	4) Dual Package, Dual Core
174
175	a) One thread per core
176
177	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
178	-> [core 1] -> [thread 0] -> Linux CPU 1
179
180	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
181	-> [core 1] -> [thread 0] -> Linux CPU 3
182
183	b) Two threads per core
184
185	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
186	-> [thread 1] -> Linux CPU 1
187	-> [core 1] -> [thread 0] -> Linux CPU 2
188	-> [thread 1] -> Linux CPU 3
189
190	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
191	-> [thread 1] -> Linux CPU 5
192	-> [core 1] -> [thread 0] -> Linux CPU 6
193	-> [thread 1] -> Linux CPU 7
194
195	Alternative enumeration:
196
197	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
198	-> [thread 1] -> Linux CPU 4
199	-> [core 1] -> [thread 0] -> Linux CPU 1
200	-> [thread 1] -> Linux CPU 5
201
202	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
203	-> [thread 1] -> Linux CPU 6
204	-> [core 1] -> [thread 0] -> Linux CPU 3
205	-> [thread 1] -> Linux CPU 7
206
207	AMD nomenclature for CMT systems:
208
209	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
210	-> [Compute Unit Core 1] -> Linux CPU 1
211	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
212	-> [Compute Unit Core 1] -> Linux CPU 3
213
214	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
215	-> [Compute Unit Core 1] -> Linux CPU 5
216	-> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
217	-> [Compute Unit Core 1] -> Linux CPU 7