]>
Commit | Line | Data |
---|---|---|
e15b4d68 AB |
1 | The Linux RapidIO Subsystem |
2 | ||
3 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
4 | ||
5 | The RapidIO standard is a packet-based fabric interconnect standard designed for | |
6 | use in embedded systems. Development of the RapidIO standard is directed by the | |
7 | RapidIO Trade Association (RTA). The current version of the RapidIO specification | |
8 | is publicly available for download from the RTA web-site [1]. | |
9 | ||
10 | This document describes the basics of the Linux RapidIO subsystem and provides | |
11 | information on its major components. | |
12 | ||
13 | 1 Overview | |
14 | ---------- | |
15 | ||
16 | Because the RapidIO subsystem follows the Linux device model it is integrated | |
17 | into the kernel similarly to other buses by defining RapidIO-specific device and | |
18 | bus types and registering them within the device model. | |
19 | ||
20 | The Linux RapidIO subsystem is architecture independent and therefore defines | |
21 | architecture-specific interfaces that provide support for common RapidIO | |
22 | subsystem operations. | |
23 | ||
24 | 2. Core Components | |
25 | ------------------ | |
26 | ||
27 | A typical RapidIO network is a combination of endpoints and switches. | |
28 | Each of these components is represented in the subsystem by an associated data | |
29 | structure. The core logical components of the RapidIO subsystem are defined | |
30 | in include/linux/rio.h file. | |
31 | ||
32 | 2.1 Master Port | |
33 | ||
34 | A master port (or mport) is a RapidIO interface controller that is local to the | |
35 | processor executing the Linux code. A master port generates and receives RapidIO | |
36 | packets (transactions). In the RapidIO subsystem each master port is represented | |
37 | by a rio_mport data structure. This structure contains master port specific | |
38 | resources such as mailboxes and doorbells. The rio_mport also includes a unique | |
39 | host device ID that is valid when a master port is configured as an enumerating | |
40 | host. | |
41 | ||
42 | RapidIO master ports are serviced by subsystem specific mport device drivers | |
43 | that provide functionality defined for this subsystem. To provide a hardware | |
44 | independent interface for RapidIO subsystem operations, rio_mport structure | |
45 | includes rio_ops data structure which contains pointers to hardware specific | |
46 | implementations of RapidIO functions. | |
47 | ||
48 | 2.2 Device | |
49 | ||
50 | A RapidIO device is any endpoint (other than mport) or switch in the network. | |
51 | All devices are presented in the RapidIO subsystem by corresponding rio_dev data | |
52 | structure. Devices form one global device list and per-network device lists | |
53 | (depending on number of available mports and networks). | |
54 | ||
55 | 2.3 Switch | |
56 | ||
57 | A RapidIO switch is a special class of device that routes packets between its | |
58 | ports towards their final destination. The packet destination port within a | |
59 | switch is defined by an internal routing table. A switch is presented in the | |
60 | RapidIO subsystem by rio_dev data structure expanded by additional rio_switch | |
61 | data structure, which contains switch specific information such as copy of the | |
62 | routing table and pointers to switch specific functions. | |
63 | ||
64 | The RapidIO subsystem defines the format and initialization method for subsystem | |
65 | specific switch drivers that are designed to provide hardware-specific | |
66 | implementation of common switch management routines. | |
67 | ||
68 | 2.4 Network | |
69 | ||
70 | A RapidIO network is a combination of interconnected endpoint and switch devices. | |
71 | Each RapidIO network known to the system is represented by corresponding rio_net | |
72 | data structure. This structure includes lists of all devices and local master | |
73 | ports that form the same network. It also contains a pointer to the default | |
74 | master port that is used to communicate with devices within the network. | |
75 | ||
ed5edee2 AB |
76 | 2.5 Device Drivers |
77 | ||
78 | RapidIO device-specific drivers follow Linux Kernel Driver Model and are | |
79 | intended to support specific RapidIO devices attached to the RapidIO network. | |
80 | ||
81 | 2.6 Subsystem Interfaces | |
82 | ||
83 | RapidIO interconnect specification defines features that may be used to provide | |
84 | one or more common service layers for all participating RapidIO devices. These | |
85 | common services may act separately from device-specific drivers or be used by | |
86 | device-specific drivers. Example of such service provider is the RIONET driver | |
87 | which implements Ethernet-over-RapidIO interface. Because only one driver can be | |
88 | registered for a device, all common RapidIO services have to be registered as | |
89 | subsystem interfaces. This allows to have multiple common services attached to | |
90 | the same device without blocking attachment of a device-specific driver. | |
91 | ||
e15b4d68 AB |
92 | 3. Subsystem Initialization |
93 | --------------------------- | |
94 | ||
95 | In order to initialize the RapidIO subsystem, a platform must initialize and | |
96 | register at least one master port within the RapidIO network. To register mport | |
ed5edee2 | 97 | within the subsystem controller driver's initialization code calls function |
5eeb9293 | 98 | rio_register_mport() for each available master port. |
e15b4d68 | 99 | |
5eeb9293 AB |
100 | After all active master ports are registered with a RapidIO subsystem, |
101 | an enumeration and/or discovery routine may be called automatically or | |
102 | by user-space command. | |
e15b4d68 | 103 | |
ed5edee2 AB |
104 | RapidIO subsystem can be configured to be built as a statically linked or |
105 | modular component of the kernel (see details below). | |
106 | ||
e15b4d68 AB |
107 | 4. Enumeration and Discovery |
108 | ---------------------------- | |
109 | ||
5eeb9293 AB |
110 | 4.1 Overview |
111 | ------------ | |
112 | ||
ed5edee2 | 113 | RapidIO subsystem configuration options allow users to build enumeration and |
5eeb9293 AB |
114 | discovery methods as statically linked components or loadable modules. |
115 | An enumeration/discovery method implementation and available input parameters | |
116 | define how any given method can be attached to available RapidIO mports: | |
117 | simply to all available mports OR individually to the specified mport device. | |
118 | ||
119 | Depending on selected enumeration/discovery build configuration, there are | |
120 | several methods to initiate an enumeration and/or discovery process: | |
121 | ||
122 | (a) Statically linked enumeration and discovery process can be started | |
123 | automatically during kernel initialization time using corresponding module | |
124 | parameters. This was the original method used since introduction of RapidIO | |
125 | subsystem. Now this method relies on enumerator module parameter which is | |
126 | 'rio-scan.scan' for existing basic enumeration/discovery method. | |
127 | When automatic start of enumeration/discovery is used a user has to ensure | |
128 | that all discovering endpoints are started before the enumerating endpoint | |
129 | and are waiting for enumeration to be completed. | |
130 | Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering | |
131 | endpoint waits for enumeration to be completed. If the specified timeout | |
132 | expires the discovery process is terminated without obtaining RapidIO network | |
133 | information. NOTE: a timed out discovery process may be restarted later using | |
ed5edee2 AB |
134 | a user-space command as it is described below (if the given endpoint was |
135 | enumerated successfully). | |
5eeb9293 AB |
136 | |
137 | (b) Statically linked enumeration and discovery process can be started by | |
138 | a command from user space. This initiation method provides more flexibility | |
139 | for a system startup compared to the option (a) above. After all participating | |
140 | endpoints have been successfully booted, an enumeration process shall be | |
141 | started first by issuing a user-space command, after an enumeration is | |
142 | completed a discovery process can be started on all remaining endpoints. | |
143 | ||
144 | (c) Modular enumeration and discovery process can be started by a command from | |
145 | user space. After an enumeration/discovery module is loaded, a network scan | |
146 | process can be started by issuing a user-space command. | |
147 | Similar to the option (b) above, an enumerator has to be started first. | |
148 | ||
149 | (d) Modular enumeration and discovery process can be started by a module | |
150 | initialization routine. In this case an enumerating module shall be loaded | |
151 | first. | |
152 | ||
153 | When a network scan process is started it calls an enumeration or discovery | |
154 | routine depending on the configured role of a master port: host or agent. | |
e15b4d68 AB |
155 | |
156 | Enumeration is performed by a master port if it is configured as a host port by | |
ed5edee2 AB |
157 | assigning a host destination ID greater than or equal to zero. The host |
158 | destination ID can be assigned to a master port using various methods depending | |
159 | on RapidIO subsystem build configuration: | |
160 | ||
161 | (a) For a statically linked RapidIO subsystem core use command line parameter | |
162 | "rapidio.hdid=" with a list of destination ID assignments in order of mport | |
163 | device registration. For example, in a system with two RapidIO controllers | |
164 | the command line parameter "rapidio.hdid=-1,7" will result in assignment of | |
165 | the host destination ID=7 to the second RapidIO controller, while the first | |
166 | one will be assigned destination ID=-1. | |
167 | ||
168 | (b) If the RapidIO subsystem core is built as a loadable module, in addition | |
169 | to the method shown above, the host destination ID(s) can be specified using | |
170 | traditional methods of passing module parameter "hdid=" during its loading: | |
171 | - from command line: "modprobe rapidio hdid=-1,7", or | |
172 | - from modprobe configuration file using configuration command "options", | |
173 | like in this example: "options rapidio hdid=-1,7". An example of modprobe | |
174 | configuration file is provided in the section below. | |
175 | ||
176 | NOTES: | |
177 | (i) if "hdid=" parameter is omitted all available mport will be assigned | |
178 | destination ID = -1; | |
179 | (ii) the "hdid=" parameter in systems with multiple mports can have | |
180 | destination ID assignments omitted from the end of list (default = -1). | |
181 | ||
182 | If the host device ID for a specific master port is set to -1, the discovery | |
183 | process will be performed for it. | |
e15b4d68 AB |
184 | |
185 | The enumeration and discovery routines use RapidIO maintenance transactions | |
186 | to access the configuration space of devices. | |
187 | ||
ed5edee2 AB |
188 | NOTE: If RapidIO switch-specific device drivers are built as loadable modules |
189 | they must be loaded before enumeration/discovery process starts. | |
190 | This requirement is cased by the fact that enumeration/discovery methods invoke | |
191 | vendor-specific callbacks on early stages. | |
192 | ||
5eeb9293 AB |
193 | 4.2 Automatic Start of Enumeration and Discovery |
194 | ------------------------------------------------ | |
195 | ||
196 | Automatic enumeration/discovery start method is applicable only to built-in | |
197 | enumeration/discovery RapidIO configuration selection. To enable automatic | |
198 | enumeration/discovery start by existing basic enumerator method set use boot | |
199 | command line parameter "rio-scan.scan=1". | |
200 | ||
201 | This configuration requires synchronized start of all RapidIO endpoints that | |
202 | form a network which will be enumerated/discovered. Discovering endpoints have | |
203 | to be started before an enumeration starts to ensure that all RapidIO | |
204 | controllers have been initialized and are ready to be discovered. Configuration | |
205 | parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which | |
206 | a discovering endpoint will wait for enumeration to be completed. | |
207 | ||
208 | When automatic enumeration/discovery start is selected, basic method's | |
209 | initialization routine calls rio_init_mports() to perform enumeration or | |
210 | discovery for all known mport devices. | |
211 | ||
212 | Depending on RapidIO network size and configuration this automatic | |
213 | enumeration/discovery start method may be difficult to use due to the | |
214 | requirement for synchronized start of all endpoints. | |
215 | ||
216 | 4.3 User-space Start of Enumeration and Discovery | |
217 | ------------------------------------------------- | |
218 | ||
219 | User-space start of enumeration and discovery can be used with built-in and | |
220 | modular build configurations. For user-space controlled start RapidIO subsystem | |
221 | creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate | |
222 | an enumeration or discovery process on specific mport device, a user needs to | |
223 | write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a | |
224 | sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device | |
225 | registration. For example for machine with single RapidIO controller, mport_ID | |
226 | for that controller always will be 0. | |
227 | ||
228 | To initiate RapidIO enumeration/discovery on all available mports a user may | |
229 | write '-1' (or RIO_MPORT_ANY) into the scan attribute file. | |
230 | ||
231 | 4.4 Basic Enumeration Method | |
232 | ---------------------------- | |
233 | ||
234 | This is an original enumeration/discovery method which is available since | |
235 | first release of RapidIO subsystem code. The enumeration process is | |
236 | implemented according to the enumeration algorithm outlined in the RapidIO | |
237 | Interconnect Specification: Annex I [1]. | |
238 | ||
239 | This method can be configured as statically linked or loadable module. | |
240 | The method's single parameter "scan" allows to trigger the enumeration/discovery | |
241 | process from module initialization routine. | |
242 | ||
243 | This enumeration/discovery method can be started only once and does not support | |
244 | unloading if it is built as a module. | |
e15b4d68 AB |
245 | |
246 | The enumeration process traverses the network using a recursive depth-first | |
247 | algorithm. When a new device is found, the enumerator takes ownership of that | |
248 | device by writing into the Host Device ID Lock CSR. It does this to ensure that | |
249 | the enumerator has exclusive right to enumerate the device. If device ownership | |
250 | is successfully acquired, the enumerator allocates a new rio_dev structure and | |
251 | initializes it according to device capabilities. | |
252 | ||
253 | If the device is an endpoint, a unique device ID is assigned to it and its value | |
254 | is written into the device's Base Device ID CSR. | |
255 | ||
256 | If the device is a switch, the enumerator allocates an additional rio_switch | |
257 | structure to store switch specific information. Then the switch's vendor ID and | |
258 | device ID are queried against a table of known RapidIO switches. Each switch | |
259 | table entry contains a pointer to a switch-specific initialization routine that | |
260 | initializes pointers to the rest of switch specific operations, and performs | |
261 | hardware initialization if necessary. A RapidIO switch does not have a unique | |
262 | device ID; it relies on hopcount and routing for device ID of an attached | |
263 | endpoint if access to its configuration registers is required. If a switch (or | |
264 | chain of switches) does not have any endpoint (except enumerator) attached to | |
265 | it, a fake device ID will be assigned to configure a route to that switch. | |
266 | In the case of a chain of switches without endpoint, one fake device ID is used | |
267 | to configure a route through the entire chain and switches are differentiated by | |
268 | their hopcount value. | |
269 | ||
270 | For both endpoints and switches the enumerator writes a unique component tag | |
271 | into device's Component Tag CSR. That unique value is used by the error | |
272 | management notification mechanism to identify a device that is reporting an | |
273 | error management event. | |
274 | ||
275 | Enumeration beyond a switch is completed by iterating over each active egress | |
276 | port of that switch. For each active link, a route to a default device ID | |
277 | (0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written | |
278 | into the routing table. The algorithm recurs by calling itself with hopcount + 1 | |
279 | and the default device ID in order to access the device on the active port. | |
280 | ||
281 | After the host has completed enumeration of the entire network it releases | |
282 | devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint | |
088024b1 | 283 | in the system, it sets the Discovered bit in the Port General Control CSR |
e15b4d68 AB |
284 | to indicate that enumeration is completed and agents are allowed to execute |
285 | passive discovery of the network. | |
286 | ||
287 | The discovery process is performed by agents and is similar to the enumeration | |
288 | process that is described above. However, the discovery process is performed | |
289 | without changes to the existing routing because agents only gather information | |
290 | about RapidIO network structure and are building an internal map of discovered | |
291 | devices. This way each Linux-based component of the RapidIO subsystem has | |
292 | a complete view of the network. The discovery process can be performed | |
293 | simultaneously by several agents. After initializing its RapidIO master port | |
294 | each agent waits for enumeration completion by the host for the configured wait | |
295 | time period. If this wait time period expires before enumeration is completed, | |
296 | an agent skips RapidIO discovery and continues with remaining kernel | |
297 | initialization. | |
298 | ||
5eeb9293 AB |
299 | 4.5 Adding New Enumeration/Discovery Method |
300 | ------------------------------------------- | |
301 | ||
302 | RapidIO subsystem code organization allows addition of new enumeration/discovery | |
303 | methods as new configuration options without significant impact to to the core | |
304 | RapidIO code. | |
305 | ||
306 | A new enumeration/discovery method has to be attached to one or more mport | |
307 | devices before an enumeration/discovery process can be started. Normally, | |
308 | method's module initialization routine calls rio_register_scan() to attach | |
309 | an enumerator to a specified mport device (or devices). The basic enumerator | |
310 | implementation demonstrates this process. | |
311 | ||
ed5edee2 AB |
312 | 4.6 Using Loadable RapidIO Switch Drivers |
313 | ----------------------------------------- | |
314 | ||
315 | In the case when RapidIO switch drivers are built as loadable modules a user | |
316 | must ensure that they are loaded before the enumeration/discovery starts. | |
317 | This process can be automated by specifying pre- or post- dependencies in the | |
318 | RapidIO-specific modprobe configuration file as shown in the example below. | |
319 | ||
320 | File /etc/modprobe.d/rapidio.conf: | |
321 | ---------------------------------- | |
322 | ||
323 | # Configure RapidIO subsystem modules | |
324 | ||
325 | # Set enumerator host destination ID (overrides kernel command line option) | |
326 | options rapidio hdid=-1,2 | |
327 | ||
328 | # Load RapidIO switch drivers immediately after rapidio core module was loaded | |
329 | softdep rapidio post: idt_gen2 idtcps tsi57x | |
330 | ||
331 | # OR : | |
332 | ||
333 | # Load RapidIO switch drivers just before rio-scan enumerator module is loaded | |
334 | softdep rio-scan pre: idt_gen2 idtcps tsi57x | |
335 | ||
336 | -------------------------- | |
337 | ||
338 | NOTE: In the example above, one of "softdep" commands must be removed or | |
339 | commented out to keep required module loading sequence. | |
340 | ||
341 | A. References | |
e15b4d68 AB |
342 | ------------- |
343 | ||
344 | [1] RapidIO Trade Association. RapidIO Interconnect Specifications. | |
345 | http://www.rapidio.org. | |
346 | [2] Rapidio TA. Technology Comparisons. | |
347 | http://www.rapidio.org/education/technology_comparisons/ | |
348 | [3] RapidIO support for Linux. | |
349 | http://lwn.net/Articles/139118/ | |
350 | [4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005 | |
351 | http://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf |