]>
Commit | Line | Data |
---|---|---|
1 | ============================== | |
2 | UNEVICTABLE LRU INFRASTRUCTURE | |
3 | ============================== | |
4 | ||
5 | ======== | |
6 | CONTENTS | |
7 | ======== | |
8 | ||
9 | (*) The Unevictable LRU | |
10 | ||
11 | - The unevictable page list. | |
12 | - Memory control group interaction. | |
13 | - Marking address spaces unevictable. | |
14 | - Detecting Unevictable Pages. | |
15 | - vmscan's handling of unevictable pages. | |
16 | ||
17 | (*) mlock()'d pages. | |
18 | ||
19 | - History. | |
20 | - Basic management. | |
21 | - mlock()/mlockall() system call handling. | |
22 | - Filtering special vmas. | |
23 | - munlock()/munlockall() system call handling. | |
24 | - Migrating mlocked pages. | |
25 | - Compacting mlocked pages. | |
26 | - mmap(MAP_LOCKED) system call handling. | |
27 | - munmap()/exit()/exec() system call handling. | |
28 | - try_to_unmap(). | |
29 | - try_to_munlock() reverse map scan. | |
30 | - Page reclaim in shrink_*_list(). | |
31 | ||
32 | ||
33 | ============ | |
34 | INTRODUCTION | |
35 | ============ | |
36 | ||
37 | This document describes the Linux memory manager's "Unevictable LRU" | |
38 | infrastructure and the use of this to manage several types of "unevictable" | |
39 | pages. | |
40 | ||
41 | The document attempts to provide the overall rationale behind this mechanism | |
42 | and the rationale for some of the design decisions that drove the | |
43 | implementation. The latter design rationale is discussed in the context of an | |
44 | implementation description. Admittedly, one can obtain the implementation | |
45 | details - the "what does it do?" - by reading the code. One hopes that the | |
46 | descriptions below add value by provide the answer to "why does it do that?". | |
47 | ||
48 | ||
49 | =================== | |
50 | THE UNEVICTABLE LRU | |
51 | =================== | |
52 | ||
53 | The Unevictable LRU facility adds an additional LRU list to track unevictable | |
54 | pages and to hide these pages from vmscan. This mechanism is based on a patch | |
55 | by Larry Woodman of Red Hat to address several scalability problems with page | |
56 | reclaim in Linux. The problems have been observed at customer sites on large | |
57 | memory x86_64 systems. | |
58 | ||
59 | To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of | |
60 | main memory will have over 32 million 4k pages in a single zone. When a large | |
61 | fraction of these pages are not evictable for any reason [see below], vmscan | |
62 | will spend a lot of time scanning the LRU lists looking for the small fraction | |
63 | of pages that are evictable. This can result in a situation where all CPUs are | |
64 | spending 100% of their time in vmscan for hours or days on end, with the system | |
65 | completely unresponsive. | |
66 | ||
67 | The unevictable list addresses the following classes of unevictable pages: | |
68 | ||
69 | (*) Those owned by ramfs. | |
70 | ||
71 | (*) Those mapped into SHM_LOCK'd shared memory regions. | |
72 | ||
73 | (*) Those mapped into VM_LOCKED [mlock()ed] VMAs. | |
74 | ||
75 | The infrastructure may also be able to handle other conditions that make pages | |
76 | unevictable, either by definition or by circumstance, in the future. | |
77 | ||
78 | ||
79 | THE UNEVICTABLE PAGE LIST | |
80 | ------------------------- | |
81 | ||
82 | The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list | |
83 | called the "unevictable" list and an associated page flag, PG_unevictable, to | |
84 | indicate that the page is being managed on the unevictable list. | |
85 | ||
86 | The PG_unevictable flag is analogous to, and mutually exclusive with, the | |
87 | PG_active flag in that it indicates on which LRU list a page resides when | |
88 | PG_lru is set. | |
89 | ||
90 | The Unevictable LRU infrastructure maintains unevictable pages on an additional | |
91 | LRU list for a few reasons: | |
92 | ||
93 | (1) We get to "treat unevictable pages just like we treat other pages in the | |
94 | system - which means we get to use the same code to manipulate them, the | |
95 | same code to isolate them (for migrate, etc.), the same code to keep track | |
96 | of the statistics, etc..." [Rik van Riel] | |
97 | ||
98 | (2) We want to be able to migrate unevictable pages between nodes for memory | |
99 | defragmentation, workload management and memory hotplug. The linux kernel | |
100 | can only migrate pages that it can successfully isolate from the LRU | |
101 | lists. If we were to maintain pages elsewhere than on an LRU-like list, | |
102 | where they can be found by isolate_lru_page(), we would prevent their | |
103 | migration, unless we reworked migration code to find the unevictable pages | |
104 | itself. | |
105 | ||
106 | ||
107 | The unevictable list does not differentiate between file-backed and anonymous, | |
108 | swap-backed pages. This differentiation is only important while the pages are, | |
109 | in fact, evictable. | |
110 | ||
111 | The unevictable list benefits from the "arrayification" of the per-zone LRU | |
112 | lists and statistics originally proposed and posted by Christoph Lameter. | |
113 | ||
114 | The unevictable list does not use the LRU pagevec mechanism. Rather, | |
115 | unevictable pages are placed directly on the page's zone's unevictable list | |
116 | under the zone lru_lock. This allows us to prevent the stranding of pages on | |
117 | the unevictable list when one task has the page isolated from the LRU and other | |
118 | tasks are changing the "evictability" state of the page. | |
119 | ||
120 | ||
121 | MEMORY CONTROL GROUP INTERACTION | |
122 | -------------------------------- | |
123 | ||
124 | The unevictable LRU facility interacts with the memory control group [aka | |
125 | memory controller; see Documentation/cgroups/memory.txt] by extending the | |
126 | lru_list enum. | |
127 | ||
128 | The memory controller data structure automatically gets a per-zone unevictable | |
129 | list as a result of the "arrayification" of the per-zone LRU lists (one per | |
130 | lru_list enum element). The memory controller tracks the movement of pages to | |
131 | and from the unevictable list. | |
132 | ||
133 | When a memory control group comes under memory pressure, the controller will | |
134 | not attempt to reclaim pages on the unevictable list. This has a couple of | |
135 | effects: | |
136 | ||
137 | (1) Because the pages are "hidden" from reclaim on the unevictable list, the | |
138 | reclaim process can be more efficient, dealing only with pages that have a | |
139 | chance of being reclaimed. | |
140 | ||
141 | (2) On the other hand, if too many of the pages charged to the control group | |
142 | are unevictable, the evictable portion of the working set of the tasks in | |
143 | the control group may not fit into the available memory. This can cause | |
144 | the control group to thrash or to OOM-kill tasks. | |
145 | ||
146 | ||
147 | MARKING ADDRESS SPACES UNEVICTABLE | |
148 | ---------------------------------- | |
149 | ||
150 | For facilities such as ramfs none of the pages attached to the address space | |
151 | may be evicted. To prevent eviction of any such pages, the AS_UNEVICTABLE | |
152 | address space flag is provided, and this can be manipulated by a filesystem | |
153 | using a number of wrapper functions: | |
154 | ||
155 | (*) void mapping_set_unevictable(struct address_space *mapping); | |
156 | ||
157 | Mark the address space as being completely unevictable. | |
158 | ||
159 | (*) void mapping_clear_unevictable(struct address_space *mapping); | |
160 | ||
161 | Mark the address space as being evictable. | |
162 | ||
163 | (*) int mapping_unevictable(struct address_space *mapping); | |
164 | ||
165 | Query the address space, and return true if it is completely | |
166 | unevictable. | |
167 | ||
168 | These are currently used in two places in the kernel: | |
169 | ||
170 | (1) By ramfs to mark the address spaces of its inodes when they are created, | |
171 | and this mark remains for the life of the inode. | |
172 | ||
173 | (2) By SYSV SHM to mark SHM_LOCK'd address spaces until SHM_UNLOCK is called. | |
174 | ||
175 | Note that SHM_LOCK is not required to page in the locked pages if they're | |
176 | swapped out; the application must touch the pages manually if it wants to | |
177 | ensure they're in memory. | |
178 | ||
179 | ||
180 | DETECTING UNEVICTABLE PAGES | |
181 | --------------------------- | |
182 | ||
183 | The function page_evictable() in vmscan.c determines whether a page is | |
184 | evictable or not using the query function outlined above [see section "Marking | |
185 | address spaces unevictable"] to check the AS_UNEVICTABLE flag. | |
186 | ||
187 | For address spaces that are so marked after being populated (as SHM regions | |
188 | might be), the lock action (eg: SHM_LOCK) can be lazy, and need not populate | |
189 | the page tables for the region as does, for example, mlock(), nor need it make | |
190 | any special effort to push any pages in the SHM_LOCK'd area to the unevictable | |
191 | list. Instead, vmscan will do this if and when it encounters the pages during | |
192 | a reclamation scan. | |
193 | ||
194 | On an unlock action (such as SHM_UNLOCK), the unlocker (eg: shmctl()) must scan | |
195 | the pages in the region and "rescue" them from the unevictable list if no other | |
196 | condition is keeping them unevictable. If an unevictable region is destroyed, | |
197 | the pages are also "rescued" from the unevictable list in the process of | |
198 | freeing them. | |
199 | ||
200 | page_evictable() also checks for mlocked pages by testing an additional page | |
201 | flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is | |
202 | faulted into a VM_LOCKED vma, or found in a vma being VM_LOCKED. | |
203 | ||
204 | ||
205 | VMSCAN'S HANDLING OF UNEVICTABLE PAGES | |
206 | -------------------------------------- | |
207 | ||
208 | If unevictable pages are culled in the fault path, or moved to the unevictable | |
209 | list at mlock() or mmap() time, vmscan will not encounter the pages until they | |
210 | have become evictable again (via munlock() for example) and have been "rescued" | |
211 | from the unevictable list. However, there may be situations where we decide, | |
212 | for the sake of expediency, to leave a unevictable page on one of the regular | |
213 | active/inactive LRU lists for vmscan to deal with. vmscan checks for such | |
214 | pages in all of the shrink_{active|inactive|page}_list() functions and will | |
215 | "cull" such pages that it encounters: that is, it diverts those pages to the | |
216 | unevictable list for the zone being scanned. | |
217 | ||
218 | There may be situations where a page is mapped into a VM_LOCKED VMA, but the | |
219 | page is not marked as PG_mlocked. Such pages will make it all the way to | |
220 | shrink_page_list() where they will be detected when vmscan walks the reverse | |
221 | map in try_to_unmap(). If try_to_unmap() returns SWAP_MLOCK, | |
222 | shrink_page_list() will cull the page at that point. | |
223 | ||
224 | To "cull" an unevictable page, vmscan simply puts the page back on the LRU list | |
225 | using putback_lru_page() - the inverse operation to isolate_lru_page() - after | |
226 | dropping the page lock. Because the condition which makes the page unevictable | |
227 | may change once the page is unlocked, putback_lru_page() will recheck the | |
228 | unevictable state of a page that it places on the unevictable list. If the | |
229 | page has become unevictable, putback_lru_page() removes it from the list and | |
230 | retries, including the page_unevictable() test. Because such a race is a rare | |
231 | event and movement of pages onto the unevictable list should be rare, these | |
232 | extra evictabilty checks should not occur in the majority of calls to | |
233 | putback_lru_page(). | |
234 | ||
235 | ||
236 | ============= | |
237 | MLOCKED PAGES | |
238 | ============= | |
239 | ||
240 | The unevictable page list is also useful for mlock(), in addition to ramfs and | |
241 | SYSV SHM. Note that mlock() is only available in CONFIG_MMU=y situations; in | |
242 | NOMMU situations, all mappings are effectively mlocked. | |
243 | ||
244 | ||
245 | HISTORY | |
246 | ------- | |
247 | ||
248 | The "Unevictable mlocked Pages" infrastructure is based on work originally | |
249 | posted by Nick Piggin in an RFC patch entitled "mm: mlocked pages off LRU". | |
250 | Nick posted his patch as an alternative to a patch posted by Christoph Lameter | |
251 | to achieve the same objective: hiding mlocked pages from vmscan. | |
252 | ||
253 | In Nick's patch, he used one of the struct page LRU list link fields as a count | |
254 | of VM_LOCKED VMAs that map the page. This use of the link field for a count | |
255 | prevented the management of the pages on an LRU list, and thus mlocked pages | |
256 | were not migratable as isolate_lru_page() could not find them, and the LRU list | |
257 | link field was not available to the migration subsystem. | |
258 | ||
259 | Nick resolved this by putting mlocked pages back on the lru list before | |
260 | attempting to isolate them, thus abandoning the count of VM_LOCKED VMAs. When | |
261 | Nick's patch was integrated with the Unevictable LRU work, the count was | |
262 | replaced by walking the reverse map to determine whether any VM_LOCKED VMAs | |
263 | mapped the page. More on this below. | |
264 | ||
265 | ||
266 | BASIC MANAGEMENT | |
267 | ---------------- | |
268 | ||
269 | mlocked pages - pages mapped into a VM_LOCKED VMA - are a class of unevictable | |
270 | pages. When such a page has been "noticed" by the memory management subsystem, | |
271 | the page is marked with the PG_mlocked flag. This can be manipulated using the | |
272 | PageMlocked() functions. | |
273 | ||
274 | A PG_mlocked page will be placed on the unevictable list when it is added to | |
275 | the LRU. Such pages can be "noticed" by memory management in several places: | |
276 | ||
277 | (1) in the mlock()/mlockall() system call handlers; | |
278 | ||
279 | (2) in the mmap() system call handler when mmapping a region with the | |
280 | MAP_LOCKED flag; | |
281 | ||
282 | (3) mmapping a region in a task that has called mlockall() with the MCL_FUTURE | |
283 | flag | |
284 | ||
285 | (4) in the fault path, if mlocked pages are "culled" in the fault path, | |
286 | and when a VM_LOCKED stack segment is expanded; or | |
287 | ||
288 | (5) as mentioned above, in vmscan:shrink_page_list() when attempting to | |
289 | reclaim a page in a VM_LOCKED VMA via try_to_unmap() | |
290 | ||
291 | all of which result in the VM_LOCKED flag being set for the VMA if it doesn't | |
292 | already have it set. | |
293 | ||
294 | mlocked pages become unlocked and rescued from the unevictable list when: | |
295 | ||
296 | (1) mapped in a range unlocked via the munlock()/munlockall() system calls; | |
297 | ||
298 | (2) munmap()'d out of the last VM_LOCKED VMA that maps the page, including | |
299 | unmapping at task exit; | |
300 | ||
301 | (3) when the page is truncated from the last VM_LOCKED VMA of an mmapped file; | |
302 | or | |
303 | ||
304 | (4) before a page is COW'd in a VM_LOCKED VMA. | |
305 | ||
306 | ||
307 | mlock()/mlockall() SYSTEM CALL HANDLING | |
308 | --------------------------------------- | |
309 | ||
310 | Both [do_]mlock() and [do_]mlockall() system call handlers call mlock_fixup() | |
311 | for each VMA in the range specified by the call. In the case of mlockall(), | |
312 | this is the entire active address space of the task. Note that mlock_fixup() | |
313 | is used for both mlocking and munlocking a range of memory. A call to mlock() | |
314 | an already VM_LOCKED VMA, or to munlock() a VMA that is not VM_LOCKED is | |
315 | treated as a no-op, and mlock_fixup() simply returns. | |
316 | ||
317 | If the VMA passes some filtering as described in "Filtering Special Vmas" | |
318 | below, mlock_fixup() will attempt to merge the VMA with its neighbors or split | |
319 | off a subset of the VMA if the range does not cover the entire VMA. Once the | |
320 | VMA has been merged or split or neither, mlock_fixup() will call | |
321 | populate_vma_page_range() to fault in the pages via get_user_pages() and to | |
322 | mark the pages as mlocked via mlock_vma_page(). | |
323 | ||
324 | Note that the VMA being mlocked might be mapped with PROT_NONE. In this case, | |
325 | get_user_pages() will be unable to fault in the pages. That's okay. If pages | |
326 | do end up getting faulted into this VM_LOCKED VMA, we'll handle them in the | |
327 | fault path or in vmscan. | |
328 | ||
329 | Also note that a page returned by get_user_pages() could be truncated or | |
330 | migrated out from under us, while we're trying to mlock it. To detect this, | |
331 | populate_vma_page_range() checks page_mapping() after acquiring the page lock. | |
332 | If the page is still associated with its mapping, we'll go ahead and call | |
333 | mlock_vma_page(). If the mapping is gone, we just unlock the page and move on. | |
334 | In the worst case, this will result in a page mapped in a VM_LOCKED VMA | |
335 | remaining on a normal LRU list without being PageMlocked(). Again, vmscan will | |
336 | detect and cull such pages. | |
337 | ||
338 | mlock_vma_page() will call TestSetPageMlocked() for each page returned by | |
339 | get_user_pages(). We use TestSetPageMlocked() because the page might already | |
340 | be mlocked by another task/VMA and we don't want to do extra work. We | |
341 | especially do not want to count an mlocked page more than once in the | |
342 | statistics. If the page was already mlocked, mlock_vma_page() need do nothing | |
343 | more. | |
344 | ||
345 | If the page was NOT already mlocked, mlock_vma_page() attempts to isolate the | |
346 | page from the LRU, as it is likely on the appropriate active or inactive list | |
347 | at that time. If the isolate_lru_page() succeeds, mlock_vma_page() will put | |
348 | back the page - by calling putback_lru_page() - which will notice that the page | |
349 | is now mlocked and divert the page to the zone's unevictable list. If | |
350 | mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle | |
351 | it later if and when it attempts to reclaim the page. | |
352 | ||
353 | ||
354 | FILTERING SPECIAL VMAS | |
355 | ---------------------- | |
356 | ||
357 | mlock_fixup() filters several classes of "special" VMAs: | |
358 | ||
359 | 1) VMAs with VM_IO or VM_PFNMAP set are skipped entirely. The pages behind | |
360 | these mappings are inherently pinned, so we don't need to mark them as | |
361 | mlocked. In any case, most of the pages have no struct page in which to so | |
362 | mark the page. Because of this, get_user_pages() will fail for these VMAs, | |
363 | so there is no sense in attempting to visit them. | |
364 | ||
365 | 2) VMAs mapping hugetlbfs page are already effectively pinned into memory. We | |
366 | neither need nor want to mlock() these pages. However, to preserve the | |
367 | prior behavior of mlock() - before the unevictable/mlock changes - | |
368 | mlock_fixup() will call make_pages_present() in the hugetlbfs VMA range to | |
369 | allocate the huge pages and populate the ptes. | |
370 | ||
371 | 3) VMAs with VM_DONTEXPAND are generally userspace mappings of kernel pages, | |
372 | such as the VDSO page, relay channel pages, etc. These pages | |
373 | are inherently unevictable and are not managed on the LRU lists. | |
374 | mlock_fixup() treats these VMAs the same as hugetlbfs VMAs. It calls | |
375 | make_pages_present() to populate the ptes. | |
376 | ||
377 | Note that for all of these special VMAs, mlock_fixup() does not set the | |
378 | VM_LOCKED flag. Therefore, we won't have to deal with them later during | |
379 | munlock(), munmap() or task exit. Neither does mlock_fixup() account these | |
380 | VMAs against the task's "locked_vm". | |
381 | ||
382 | ||
383 | munlock()/munlockall() SYSTEM CALL HANDLING | |
384 | ------------------------------------------- | |
385 | ||
386 | The munlock() and munlockall() system calls are handled by the same functions - | |
387 | do_mlock[all]() - as the mlock() and mlockall() system calls with the unlock vs | |
388 | lock operation indicated by an argument. So, these system calls are also | |
389 | handled by mlock_fixup(). Again, if called for an already munlocked VMA, | |
390 | mlock_fixup() simply returns. Because of the VMA filtering discussed above, | |
391 | VM_LOCKED will not be set in any "special" VMAs. So, these VMAs will be | |
392 | ignored for munlock. | |
393 | ||
394 | If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the | |
395 | specified range. The range is then munlocked via the function | |
396 | populate_vma_page_range() - the same function used to mlock a VMA range - | |
397 | passing a flag to indicate that munlock() is being performed. | |
398 | ||
399 | Because the VMA access protections could have been changed to PROT_NONE after | |
400 | faulting in and mlocking pages, get_user_pages() was unreliable for visiting | |
401 | these pages for munlocking. Because we don't want to leave pages mlocked, | |
402 | get_user_pages() was enhanced to accept a flag to ignore the permissions when | |
403 | fetching the pages - all of which should be resident as a result of previous | |
404 | mlocking. | |
405 | ||
406 | For munlock(), populate_vma_page_range() unlocks individual pages by calling | |
407 | munlock_vma_page(). munlock_vma_page() unconditionally clears the PG_mlocked | |
408 | flag using TestClearPageMlocked(). As with mlock_vma_page(), | |
409 | munlock_vma_page() use the Test*PageMlocked() function to handle the case where | |
410 | the page might have already been unlocked by another task. If the page was | |
411 | mlocked, munlock_vma_page() updates that zone statistics for the number of | |
412 | mlocked pages. Note, however, that at this point we haven't checked whether | |
413 | the page is mapped by other VM_LOCKED VMAs. | |
414 | ||
415 | We can't call try_to_munlock(), the function that walks the reverse map to | |
416 | check for other VM_LOCKED VMAs, without first isolating the page from the LRU. | |
417 | try_to_munlock() is a variant of try_to_unmap() and thus requires that the page | |
418 | not be on an LRU list [more on these below]. However, the call to | |
419 | isolate_lru_page() could fail, in which case we couldn't try_to_munlock(). So, | |
420 | we go ahead and clear PG_mlocked up front, as this might be the only chance we | |
421 | have. If we can successfully isolate the page, we go ahead and | |
422 | try_to_munlock(), which will restore the PG_mlocked flag and update the zone | |
423 | page statistics if it finds another VMA holding the page mlocked. If we fail | |
424 | to isolate the page, we'll have left a potentially mlocked page on the LRU. | |
425 | This is fine, because we'll catch it later if and if vmscan tries to reclaim | |
426 | the page. This should be relatively rare. | |
427 | ||
428 | ||
429 | MIGRATING MLOCKED PAGES | |
430 | ----------------------- | |
431 | ||
432 | A page that is being migrated has been isolated from the LRU lists and is held | |
433 | locked across unmapping of the page, updating the page's address space entry | |
434 | and copying the contents and state, until the page table entry has been | |
435 | replaced with an entry that refers to the new page. Linux supports migration | |
436 | of mlocked pages and other unevictable pages. This involves simply moving the | |
437 | PG_mlocked and PG_unevictable states from the old page to the new page. | |
438 | ||
439 | Note that page migration can race with mlocking or munlocking of the same page. | |
440 | This has been discussed from the mlock/munlock perspective in the respective | |
441 | sections above. Both processes (migration and m[un]locking) hold the page | |
442 | locked. This provides the first level of synchronization. Page migration | |
443 | zeros out the page_mapping of the old page before unlocking it, so m[un]lock | |
444 | can skip these pages by testing the page mapping under page lock. | |
445 | ||
446 | To complete page migration, we place the new and old pages back onto the LRU | |
447 | after dropping the page lock. The "unneeded" page - old page on success, new | |
448 | page on failure - will be freed when the reference count held by the migration | |
449 | process is released. To ensure that we don't strand pages on the unevictable | |
450 | list because of a race between munlock and migration, page migration uses the | |
451 | putback_lru_page() function to add migrated pages back to the LRU. | |
452 | ||
453 | ||
454 | COMPACTING MLOCKED PAGES | |
455 | ------------------------ | |
456 | ||
457 | The unevictable LRU can be scanned for compactable regions and the default | |
458 | behavior is to do so. /proc/sys/vm/compact_unevictable_allowed controls | |
459 | this behavior (see Documentation/sysctl/vm.txt). Once scanning of the | |
460 | unevictable LRU is enabled, the work of compaction is mostly handled by | |
461 | the page migration code and the same work flow as described in MIGRATING | |
462 | MLOCKED PAGES will apply. | |
463 | ||
464 | ||
465 | mmap(MAP_LOCKED) SYSTEM CALL HANDLING | |
466 | ------------------------------------- | |
467 | ||
468 | In addition the mlock()/mlockall() system calls, an application can request | |
469 | that a region of memory be mlocked supplying the MAP_LOCKED flag to the mmap() | |
470 | call. There is one important and subtle difference here, though. mmap() + mlock() | |
471 | will fail if the range cannot be faulted in (e.g. because mm_populate fails) | |
472 | and returns with ENOMEM while mmap(MAP_LOCKED) will not fail. The mmaped | |
473 | area will still have properties of the locked area - aka. pages will not get | |
474 | swapped out - but major page faults to fault memory in might still happen. | |
475 | ||
476 | Furthermore, any mmap() call or brk() call that expands the heap by a | |
477 | task that has previously called mlockall() with the MCL_FUTURE flag will result | |
478 | in the newly mapped memory being mlocked. Before the unevictable/mlock | |
479 | changes, the kernel simply called make_pages_present() to allocate pages and | |
480 | populate the page table. | |
481 | ||
482 | To mlock a range of memory under the unevictable/mlock infrastructure, the | |
483 | mmap() handler and task address space expansion functions call | |
484 | populate_vma_page_range() specifying the vma and the address range to mlock. | |
485 | ||
486 | The callers of populate_vma_page_range() will have already added the memory range | |
487 | to be mlocked to the task's "locked_vm". To account for filtered VMAs, | |
488 | populate_vma_page_range() returns the number of pages NOT mlocked. All of the | |
489 | callers then subtract a non-negative return value from the task's locked_vm. A | |
490 | negative return value represent an error - for example, from get_user_pages() | |
491 | attempting to fault in a VMA with PROT_NONE access. In this case, we leave the | |
492 | memory range accounted as locked_vm, as the protections could be changed later | |
493 | and pages allocated into that region. | |
494 | ||
495 | ||
496 | munmap()/exit()/exec() SYSTEM CALL HANDLING | |
497 | ------------------------------------------- | |
498 | ||
499 | When unmapping an mlocked region of memory, whether by an explicit call to | |
500 | munmap() or via an internal unmap from exit() or exec() processing, we must | |
501 | munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. | |
502 | Before the unevictable/mlock changes, mlocking did not mark the pages in any | |
503 | way, so unmapping them required no processing. | |
504 | ||
505 | To munlock a range of memory under the unevictable/mlock infrastructure, the | |
506 | munmap() handler and task address space call tear down function | |
507 | munlock_vma_pages_all(). The name reflects the observation that one always | |
508 | specifies the entire VMA range when munlock()ing during unmap of a region. | |
509 | Because of the VMA filtering when mlocking() regions, only "normal" VMAs that | |
510 | actually contain mlocked pages will be passed to munlock_vma_pages_all(). | |
511 | ||
512 | munlock_vma_pages_all() clears the VM_LOCKED VMA flag and, like mlock_fixup() | |
513 | for the munlock case, calls __munlock_vma_pages_range() to walk the page table | |
514 | for the VMA's memory range and munlock_vma_page() each resident page mapped by | |
515 | the VMA. This effectively munlocks the page, only if this is the last | |
516 | VM_LOCKED VMA that maps the page. | |
517 | ||
518 | ||
519 | try_to_unmap() | |
520 | -------------- | |
521 | ||
522 | Pages can, of course, be mapped into multiple VMAs. Some of these VMAs may | |
523 | have VM_LOCKED flag set. It is possible for a page mapped into one or more | |
524 | VM_LOCKED VMAs not to have the PG_mlocked flag set and therefore reside on one | |
525 | of the active or inactive LRU lists. This could happen if, for example, a task | |
526 | in the process of munlocking the page could not isolate the page from the LRU. | |
527 | As a result, vmscan/shrink_page_list() might encounter such a page as described | |
528 | in section "vmscan's handling of unevictable pages". To handle this situation, | |
529 | try_to_unmap() checks for VM_LOCKED VMAs while it is walking a page's reverse | |
530 | map. | |
531 | ||
532 | try_to_unmap() is always called, by either vmscan for reclaim or for page | |
533 | migration, with the argument page locked and isolated from the LRU. Separate | |
534 | functions handle anonymous and mapped file and KSM pages, as these types of | |
535 | pages have different reverse map lookup mechanisms, with different locking. | |
536 | In each case, whether rmap_walk_anon() or rmap_walk_file() or rmap_walk_ksm(), | |
537 | it will call try_to_unmap_one() for every VMA which might contain the page. | |
538 | ||
539 | When trying to reclaim, if try_to_unmap_one() finds the page in a VM_LOCKED | |
540 | VMA, it will then mlock the page via mlock_vma_page() instead of unmapping it, | |
541 | and return SWAP_MLOCK to indicate that the page is unevictable: and the scan | |
542 | stops there. | |
543 | ||
544 | mlock_vma_page() is called while holding the page table's lock (in addition | |
545 | to the page lock, and the rmap lock): to serialize against concurrent mlock or | |
546 | munlock or munmap system calls, mm teardown (munlock_vma_pages_all), reclaim, | |
547 | holepunching, and truncation of file pages and their anonymous COWed pages. | |
548 | ||
549 | ||
550 | try_to_munlock() REVERSE MAP SCAN | |
551 | --------------------------------- | |
552 | ||
553 | [!] TODO/FIXME: a better name might be page_mlocked() - analogous to the | |
554 | page_referenced() reverse map walker. | |
555 | ||
556 | When munlock_vma_page() [see section "munlock()/munlockall() System Call | |
557 | Handling" above] tries to munlock a page, it needs to determine whether or not | |
558 | the page is mapped by any VM_LOCKED VMA without actually attempting to unmap | |
559 | all PTEs from the page. For this purpose, the unevictable/mlock infrastructure | |
560 | introduced a variant of try_to_unmap() called try_to_munlock(). | |
561 | ||
562 | try_to_munlock() calls the same functions as try_to_unmap() for anonymous and | |
563 | mapped file and KSM pages with a flag argument specifying unlock versus unmap | |
564 | processing. Again, these functions walk the respective reverse maps looking | |
565 | for VM_LOCKED VMAs. When such a VMA is found, as in the try_to_unmap() case, | |
566 | the functions mlock the page via mlock_vma_page() and return SWAP_MLOCK. This | |
567 | undoes the pre-clearing of the page's PG_mlocked done by munlock_vma_page. | |
568 | ||
569 | Note that try_to_munlock()'s reverse map walk must visit every VMA in a page's | |
570 | reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA. | |
571 | However, the scan can terminate when it encounters a VM_LOCKED VMA. | |
572 | Although try_to_munlock() might be called a great many times when munlocking a | |
573 | large region or tearing down a large address space that has been mlocked via | |
574 | mlockall(), overall this is a fairly rare event. | |
575 | ||
576 | ||
577 | PAGE RECLAIM IN shrink_*_list() | |
578 | ------------------------------- | |
579 | ||
580 | shrink_active_list() culls any obviously unevictable pages - i.e. | |
581 | !page_evictable(page) - diverting these to the unevictable list. | |
582 | However, shrink_active_list() only sees unevictable pages that made it onto the | |
583 | active/inactive lru lists. Note that these pages do not have PageUnevictable | |
584 | set - otherwise they would be on the unevictable list and shrink_active_list | |
585 | would never see them. | |
586 | ||
587 | Some examples of these unevictable pages on the LRU lists are: | |
588 | ||
589 | (1) ramfs pages that have been placed on the LRU lists when first allocated. | |
590 | ||
591 | (2) SHM_LOCK'd shared memory pages. shmctl(SHM_LOCK) does not attempt to | |
592 | allocate or fault in the pages in the shared memory region. This happens | |
593 | when an application accesses the page the first time after SHM_LOCK'ing | |
594 | the segment. | |
595 | ||
596 | (3) mlocked pages that could not be isolated from the LRU and moved to the | |
597 | unevictable list in mlock_vma_page(). | |
598 | ||
599 | shrink_inactive_list() also diverts any unevictable pages that it finds on the | |
600 | inactive lists to the appropriate zone's unevictable list. | |
601 | ||
602 | shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd | |
603 | after shrink_active_list() had moved them to the inactive list, or pages mapped | |
604 | into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to | |
605 | recheck via try_to_munlock(). shrink_inactive_list() won't notice the latter, | |
606 | but will pass on to shrink_page_list(). | |
607 | ||
608 | shrink_page_list() again culls obviously unevictable pages that it could | |
609 | encounter for similar reason to shrink_inactive_list(). Pages mapped into | |
610 | VM_LOCKED VMAs but without PG_mlocked set will make it all the way to | |
611 | try_to_unmap(). shrink_page_list() will divert them to the unevictable list | |
612 | when try_to_unmap() returns SWAP_MLOCK, as discussed above. |