]>
Commit | Line | Data |
---|---|---|
97162a1e MCC |
1 | =========================== |
2 | InfiniBand Midlayer Locking | |
3 | =========================== | |
617b586b HR |
4 | |
5 | This guide is an attempt to make explicit the locking assumptions | |
6 | made by the InfiniBand midlayer. It describes the requirements on | |
7 | both low-level drivers that sit below the midlayer and upper level | |
8 | protocols that use the midlayer. | |
9 | ||
10 | Sleeping and interrupt context | |
97162a1e | 11 | ============================== |
617b586b HR |
12 | |
13 | With the following exceptions, a low-level driver implementation of | |
14 | all of the methods in struct ib_device may sleep. The exceptions | |
15 | are any methods from the list: | |
16 | ||
97162a1e MCC |
17 | - create_ah |
18 | - modify_ah | |
19 | - query_ah | |
20 | - destroy_ah | |
21 | - post_send | |
22 | - post_recv | |
23 | - poll_cq | |
24 | - req_notify_cq | |
617b586b HR |
25 | |
26 | which may not sleep and must be callable from any context. | |
27 | ||
28 | The corresponding functions exported to upper level protocol | |
29 | consumers: | |
30 | ||
bd0abfa8 GJ |
31 | - rdma_create_ah |
32 | - rdma_modify_ah | |
33 | - rdma_query_ah | |
34 | - rdma_destroy_ah | |
97162a1e MCC |
35 | - ib_post_send |
36 | - ib_post_recv | |
37 | - ib_req_notify_cq | |
617b586b HR |
38 | |
39 | are therefore safe to call from any context. | |
40 | ||
41 | In addition, the function | |
42 | ||
97162a1e | 43 | - ib_dispatch_event |
617b586b HR |
44 | |
45 | used by low-level drivers to dispatch asynchronous events through | |
46 | the midlayer is also safe to call from any context. | |
47 | ||
48 | Reentrancy | |
97162a1e | 49 | ---------- |
617b586b HR |
50 | |
51 | All of the methods in struct ib_device exported by a low-level | |
52 | driver must be fully reentrant. The low-level driver is required to | |
53 | perform all synchronization necessary to maintain consistency, even | |
54 | if multiple function calls using the same object are run | |
55 | simultaneously. | |
56 | ||
57 | The IB midlayer does not perform any serialization of function calls. | |
58 | ||
59 | Because low-level drivers are reentrant, upper level protocol | |
60 | consumers are not required to perform any serialization. However, | |
61 | some serialization may be required to get sensible results. For | |
62 | example, a consumer may safely call ib_poll_cq() on multiple CPUs | |
63 | simultaneously. However, the ordering of the work completion | |
64 | information between different calls of ib_poll_cq() is not defined. | |
65 | ||
66 | Callbacks | |
97162a1e | 67 | --------- |
617b586b HR |
68 | |
69 | A low-level driver must not perform a callback directly from the | |
70 | same callchain as an ib_device method call. For example, it is not | |
71 | allowed for a low-level driver to call a consumer's completion event | |
72 | handler directly from its post_send method. Instead, the low-level | |
73 | driver should defer this callback by, for example, scheduling a | |
74 | tasklet to perform the callback. | |
75 | ||
76 | The low-level driver is responsible for ensuring that multiple | |
77 | completion event handlers for the same CQ are not called | |
78 | simultaneously. The driver must guarantee that only one CQ event | |
79 | handler for a given CQ is running at a time. In other words, the | |
97162a1e | 80 | following situation is not allowed:: |
617b586b | 81 | |
97162a1e | 82 | CPU1 CPU2 |
617b586b | 83 | |
97162a1e MCC |
84 | low-level driver -> |
85 | consumer CQ event callback: | |
86 | /* ... */ | |
87 | ib_req_notify_cq(cq, ...); | |
88 | low-level driver -> | |
89 | /* ... */ consumer CQ event callback: | |
90 | /* ... */ | |
91 | return from CQ event handler | |
617b586b HR |
92 | |
93 | The context in which completion event and asynchronous event | |
94 | callbacks run is not defined. Depending on the low-level driver, it | |
95 | may be process context, softirq context, or interrupt context. | |
96 | Upper level protocol consumers may not sleep in a callback. | |
97 | ||
98 | Hot-plug | |
97162a1e | 99 | -------- |
617b586b HR |
100 | |
101 | A low-level driver announces that a device is ready for use by | |
102 | consumers when it calls ib_register_device(), all initialization | |
103 | must be complete before this call. The device must remain usable | |
104 | until the driver's call to ib_unregister_device() has returned. | |
105 | ||
106 | A low-level driver must call ib_register_device() and | |
107 | ib_unregister_device() from process context. It must not hold any | |
108 | semaphores that could cause deadlock if a consumer calls back into | |
109 | the driver across these calls. | |
110 | ||
111 | An upper level protocol consumer may begin using an IB device as | |
112 | soon as the add method of its struct ib_client is called for that | |
113 | device. A consumer must finish all cleanup and free all resources | |
114 | relating to a device before returning from the remove method. | |
115 | ||
116 | A consumer is permitted to sleep in its add and remove methods. |