]>
Commit | Line | Data |
---|---|---|
83144186 RW |
1 | Freezing of tasks |
2 | (C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL | |
3 | ||
4 | I. What is the freezing of tasks? | |
5 | ||
6 | The freezing of tasks is a mechanism by which user space processes and some | |
7 | kernel threads are controlled during hibernation or system-wide suspend (on some | |
8 | architectures). | |
9 | ||
10 | II. How does it work? | |
11 | ||
12 | There are four per-task flags used for that, PF_NOFREEZE, PF_FROZEN, TIF_FREEZE | |
13 | and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have | |
14 | PF_NOFREEZE unset (all user space processes and some kernel threads) are | |
15 | regarded as 'freezable' and treated in a special way before the system enters a | |
16 | suspend state as well as before a hibernation image is created (in what follows | |
17 | we only consider hibernation, but the description also applies to suspend). | |
18 | ||
19 | Namely, as the first step of the hibernation procedure the function | |
20 | freeze_processes() (defined in kernel/power/process.c) is called. It executes | |
21 | try_to_freeze_tasks() that sets TIF_FREEZE for all of the freezable tasks and | |
d5d8c597 RW |
22 | either wakes them up, if they are kernel threads, or sends fake signals to them, |
23 | if they are user space processes. A task that has TIF_FREEZE set, should react | |
a0acae0e | 24 | to it by calling the function called __refrigerator() (defined in |
e9db50b8 | 25 | kernel/freezer.c), which sets the task's PF_FROZEN flag, changes its state |
d5d8c597 RW |
26 | to TASK_UNINTERRUPTIBLE and makes it loop until PF_FROZEN is cleared for it. |
27 | Then, we say that the task is 'frozen' and therefore the set of functions | |
28 | handling this mechanism is referred to as 'the freezer' (these functions are | |
e9db50b8 SB |
29 | defined in kernel/power/process.c, kernel/freezer.c & include/linux/freezer.h). |
30 | User space processes are generally frozen before kernel threads. | |
83144186 | 31 | |
a0acae0e TH |
32 | __refrigerator() must not be called directly. Instead, use the |
33 | try_to_freeze() function (defined in include/linux/freezer.h), that checks | |
34 | the task's TIF_FREEZE flag and makes the task enter __refrigerator() if the | |
35 | flag is set. | |
83144186 RW |
36 | |
37 | For user space processes try_to_freeze() is called automatically from the | |
38 | signal-handling code, but the freezable kernel threads need to call it | |
d5d8c597 RW |
39 | explicitly in suitable places or use the wait_event_freezable() or |
40 | wait_event_freezable_timeout() macros (defined in include/linux/freezer.h) | |
41 | that combine interruptible sleep with checking if TIF_FREEZE is set and calling | |
42 | try_to_freeze(). The main loop of a freezable kernel thread may look like the | |
43 | following one: | |
83144186 | 44 | |
d5d8c597 | 45 | set_freezable(); |
83144186 RW |
46 | do { |
47 | hub_events(); | |
d5d8c597 RW |
48 | wait_event_freezable(khubd_wait, |
49 | !list_empty(&hub_event_list) || | |
50 | kthread_should_stop()); | |
51 | } while (!kthread_should_stop() || !list_empty(&hub_event_list)); | |
83144186 RW |
52 | |
53 | (from drivers/usb/core/hub.c::hub_thread()). | |
54 | ||
55 | If a freezable kernel thread fails to call try_to_freeze() after the freezer has | |
56 | set TIF_FREEZE for it, the freezing of tasks will fail and the entire | |
57 | hibernation operation will be cancelled. For this reason, freezable kernel | |
d5d8c597 RW |
58 | threads must call try_to_freeze() somewhere or use one of the |
59 | wait_event_freezable() and wait_event_freezable_timeout() macros. | |
83144186 RW |
60 | |
61 | After the system memory state has been restored from a hibernation image and | |
62 | devices have been reinitialized, the function thaw_processes() is called in | |
63 | order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that | |
a0acae0e | 64 | have been frozen leave __refrigerator() and continue running. |
83144186 | 65 | |
9045a050 SB |
66 | |
67 | Rationale behind the functions dealing with freezing and thawing of tasks: | |
68 | ------------------------------------------------------------------------- | |
69 | ||
70 | freeze_processes(): | |
71 | - freezes only userspace tasks | |
72 | ||
73 | freeze_kernel_threads(): | |
74 | - freezes all tasks (including kernel threads) because we can't freeze | |
75 | kernel threads without freezing userspace tasks | |
76 | ||
77 | thaw_kernel_threads(): | |
78 | - thaws only kernel threads; this is particularly useful if we need to do | |
79 | anything special in between thawing of kernel threads and thawing of | |
80 | userspace tasks, or if we want to postpone the thawing of userspace tasks | |
81 | ||
82 | thaw_processes(): | |
83 | - thaws all tasks (including kernel threads) because we can't thaw userspace | |
84 | tasks without thawing kernel threads | |
85 | ||
86 | ||
83144186 RW |
87 | III. Which kernel threads are freezable? |
88 | ||
89 | Kernel threads are not freezable by default. However, a kernel thread may clear | |
90 | PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE | |
3a7cbd50 | 91 | directly is not allowed). From this point it is regarded as freezable |
83144186 RW |
92 | and must call try_to_freeze() in a suitable place. |
93 | ||
94 | IV. Why do we do that? | |
95 | ||
96 | Generally speaking, there is a couple of reasons to use the freezing of tasks: | |
97 | ||
98 | 1. The principal reason is to prevent filesystems from being damaged after | |
99 | hibernation. At the moment we have no simple means of checkpointing | |
100 | filesystems, so if there are any modifications made to filesystem data and/or | |
101 | metadata on disks, we cannot bring them back to the state from before the | |
102 | modifications. At the same time each hibernation image contains some | |
103 | filesystem-related information that must be consistent with the state of the | |
104 | on-disk data and metadata after the system memory state has been restored from | |
105 | the image (otherwise the filesystems will be damaged in a nasty way, usually | |
106 | making them almost impossible to repair). We therefore freeze tasks that might | |
107 | cause the on-disk filesystems' data and metadata to be modified after the | |
108 | hibernation image has been created and before the system is finally powered off. | |
109 | The majority of these are user space processes, but if any of the kernel threads | |
110 | may cause something like this to happen, they have to be freezable. | |
111 | ||
27763653 RW |
112 | 2. Next, to create the hibernation image we need to free a sufficient amount of |
113 | memory (approximately 50% of available RAM) and we need to do that before | |
114 | devices are deactivated, because we generally need them for swapping out. Then, | |
115 | after the memory for the image has been freed, we don't want tasks to allocate | |
116 | additional memory and we prevent them from doing that by freezing them earlier. | |
117 | [Of course, this also means that device drivers should not allocate substantial | |
118 | amounts of memory from their .suspend() callbacks before hibernation, but this | |
e9db50b8 | 119 | is a separate issue.] |
27763653 RW |
120 | |
121 | 3. The third reason is to prevent user space processes and some kernel threads | |
83144186 RW |
122 | from interfering with the suspending and resuming of devices. A user space |
123 | process running on a second CPU while we are suspending devices may, for | |
124 | example, be troublesome and without the freezing of tasks we would need some | |
125 | safeguards against race conditions that might occur in such a case. | |
126 | ||
127 | Although Linus Torvalds doesn't like the freezing of tasks, he said this in one | |
128 | of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608): | |
129 | ||
130 | "RJW:> Why we freeze tasks at all or why we freeze kernel threads? | |
131 | ||
132 | Linus: In many ways, 'at all'. | |
133 | ||
134 | I _do_ realize the IO request queue issues, and that we cannot actually do | |
135 | s2ram with some devices in the middle of a DMA. So we want to be able to | |
136 | avoid *that*, there's no question about that. And I suspect that stopping | |
137 | user threads and then waiting for a sync is practically one of the easier | |
138 | ways to do so. | |
139 | ||
140 | So in practice, the 'at all' may become a 'why freeze kernel threads?' and | |
141 | freezing user threads I don't find really objectionable." | |
142 | ||
143 | Still, there are kernel threads that may want to be freezable. For example, if | |
5eb6f9ad VK |
144 | a kernel thread that belongs to a device driver accesses the device directly, it |
145 | in principle needs to know when the device is suspended, so that it doesn't try | |
146 | to access it at that time. However, if the kernel thread is freezable, it will | |
147 | be frozen before the driver's .suspend() callback is executed and it will be | |
83144186 RW |
148 | thawed after the driver's .resume() callback has run, so it won't be accessing |
149 | the device while it's suspended. | |
150 | ||
27763653 | 151 | 4. Another reason for freezing tasks is to prevent user space processes from |
83144186 RW |
152 | realizing that hibernation (or suspend) operation takes place. Ideally, user |
153 | space processes should not notice that such a system-wide operation has occurred | |
154 | and should continue running without any problems after the restore (or resume | |
155 | from suspend). Unfortunately, in the most general case this is quite difficult | |
156 | to achieve without the freezing of tasks. Consider, for example, a process | |
157 | that depends on all CPUs being online while it's running. Since we need to | |
158 | disable nonboot CPUs during the hibernation, if this process is not frozen, it | |
159 | may notice that the number of CPUs has changed and may start to work incorrectly | |
160 | because of that. | |
161 | ||
162 | V. Are there any problems related to the freezing of tasks? | |
163 | ||
164 | Yes, there are. | |
165 | ||
166 | First of all, the freezing of kernel threads may be tricky if they depend one | |
167 | on another. For example, if kernel thread A waits for a completion (in the | |
168 | TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B | |
169 | and B is frozen in the meantime, then A will be blocked until B is thawed, which | |
170 | may be undesirable. That's why kernel threads are not freezable by default. | |
171 | ||
172 | Second, there are the following two problems related to the freezing of user | |
173 | space processes: | |
174 | 1. Putting processes into an uninterruptible sleep distorts the load average. | |
175 | 2. Now that we have FUSE, plus the framework for doing device drivers in | |
176 | userspace, it gets even more complicated because some userspace processes are | |
177 | now doing the sorts of things that kernel threads do | |
178 | (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html). | |
179 | ||
180 | The problem 1. seems to be fixable, although it hasn't been fixed so far. The | |
181 | other one is more serious, but it seems that we can work around it by using | |
182 | hibernation (and suspend) notifiers (in that case, though, we won't be able to | |
183 | avoid the realization by the user space processes that the hibernation is taking | |
184 | place). | |
185 | ||
186 | There are also problems that the freezing of tasks tends to expose, although | |
187 | they are not directly related to it. For example, if request_firmware() is | |
188 | called from a device driver's .resume() routine, it will timeout and eventually | |
189 | fail, because the user land process that should respond to the request is frozen | |
190 | at this point. So, seemingly, the failure is due to the freezing of tasks. | |
191 | Suppose, however, that the firmware file is located on a filesystem accessible | |
192 | only through another device that hasn't been resumed yet. In that case, | |
193 | request_firmware() will fail regardless of whether or not the freezing of tasks | |
194 | is used. Consequently, the problem is not really related to the freezing of | |
fccdb5ae ON |
195 | tasks, since it generally exists anyway. |
196 | ||
197 | A driver must have all firmwares it may need in RAM before suspend() is called. | |
198 | If keeping them is not practical, for example due to their size, they must be | |
199 | requested early enough using the suspend notifier API described in notifiers.txt. | |
cba3176e SB |
200 | |
201 | VI. Are there any precautions to be taken to prevent freezing failures? | |
202 | ||
203 | Yes, there are. | |
204 | ||
205 | First of all, grabbing the 'pm_mutex' lock to mutually exclude a piece of code | |
206 | from system-wide sleep such as suspend/hibernation is not encouraged. | |
207 | If possible, that piece of code must instead hook onto the suspend/hibernation | |
208 | notifiers to achieve mutual exclusion. Look at the CPU-Hotplug code | |
209 | (kernel/cpu.c) for an example. | |
210 | ||
211 | However, if that is not feasible, and grabbing 'pm_mutex' is deemed necessary, | |
212 | it is strongly discouraged to directly call mutex_[un]lock(&pm_mutex) since | |
213 | that could lead to freezing failures, because if the suspend/hibernate code | |
214 | successfully acquired the 'pm_mutex' lock, and hence that other entity failed | |
215 | to acquire the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE | |
216 | state. As a consequence, the freezer would not be able to freeze that task, | |
217 | leading to freezing failure. | |
218 | ||
219 | However, the [un]lock_system_sleep() APIs are safe to use in this scenario, | |
220 | since they ask the freezer to skip freezing this task, since it is anyway | |
221 | "frozen enough" as it is blocked on 'pm_mutex', which will be released | |
222 | only after the entire suspend/hibernation sequence is complete. | |
223 | So, to summarize, use [un]lock_system_sleep() instead of directly using | |
224 | mutex_[un]lock(&pm_mutex). That would prevent freezing failures. |