]>
Commit | Line | Data |
---|---|---|
44f380fe | 1 | .. _transhuge: |
1c9bf22c | 2 | |
44f380fe MR |
3 | ============================ |
4 | Transparent Hugepage Support | |
5 | ============================ | |
6 | ||
41f0a954 RC |
7 | This document describes design principles for Transparent Hugepage (THP) |
8 | support and its interaction with other parts of the memory management | |
9 | system. | |
07a83038 MR |
10 | |
11 | Design principles | |
12 | ================= | |
13 | ||
14 | - "graceful fallback": mm components which don't have transparent hugepage | |
15 | knowledge fall back to breaking huge pmd mapping into table of ptes and, | |
16 | if necessary, split a transparent hugepage. Therefore these components | |
17 | can continue working on the regular pages or regular pte mappings. | |
18 | ||
19 | - if a hugepage allocation fails because of memory fragmentation, | |
20 | regular pages should be gracefully allocated instead and mixed in | |
21 | the same vma without any failure or significant delay and without | |
22 | userland noticing | |
23 | ||
24 | - if some task quits and more hugepages become available (either | |
25 | immediately in the buddy or through the VM), guest physical memory | |
26 | backed by regular pages should be relocated on hugepages | |
27 | automatically (with khugepaged) | |
28 | ||
29 | - it doesn't require memory reservation and in turn it uses hugepages | |
30 | whenever possible (the only possible reservation here is kernelcore= | |
31 | to avoid unmovable pages to fragment all the memory but such a tweak | |
32 | is not specific to transparent hugepage support and it's a generic | |
33 | feature that applies to all dynamic high order allocations in the | |
34 | kernel) | |
35 | ||
44f380fe MR |
36 | get_user_pages and follow_page |
37 | ============================== | |
1c9bf22c AA |
38 | |
39 | get_user_pages and follow_page if run on a hugepage, will return the | |
40 | head or tail pages as usual (exactly as they would do on | |
41f0a954 | 41 | hugetlbfs). Most GUP users will only care about the actual physical |
1c9bf22c AA |
42 | address of the page and its temporary pinning to release after the I/O |
43 | is complete, so they won't ever notice the fact the page is huge. But | |
44 | if any driver is going to mangle over the page structure of the tail | |
45 | page (like for checking page->mapping or other bits that are relevant | |
46 | for the head page and not the tail page), it should be updated to jump | |
41f0a954 RC |
47 | to check head page instead. Taking a reference on any head/tail page would |
48 | prevent the page from being split by anyone. | |
1c9bf22c | 49 | |
44f380fe MR |
50 | .. note:: |
51 | these aren't new constraints to the GUP API, and they match the | |
41f0a954 | 52 | same constraints that apply to hugetlbfs too, so any driver capable |
44f380fe MR |
53 | of handling GUP on hugetlbfs will also work fine on transparent |
54 | hugepage backed mappings. | |
1c9bf22c | 55 | |
44f380fe MR |
56 | Graceful fallback |
57 | ================= | |
1c9bf22c | 58 | |
89474d50 | 59 | Code walking pagetables but unaware about huge pmds can simply call |
a46e6376 | 60 | split_huge_pmd(vma, pmd, addr) where the pmd is the one returned by |
1c9bf22c | 61 | pmd_offset. It's trivial to make the code transparent hugepage aware |
a46e6376 | 62 | by just grepping for "pmd_offset" and adding split_huge_pmd where |
1c9bf22c AA |
63 | missing after pmd_offset returns the pmd. Thanks to the graceful |
64 | fallback design, with a one liner change, you can avoid to write | |
41f0a954 | 65 | hundreds if not thousands of lines of complex code to make your code |
1c9bf22c AA |
66 | hugepage aware. |
67 | ||
68 | If you're not walking pagetables but you run into a physical hugepage | |
41f0a954 | 69 | that you can't handle natively in your code, you can split it by |
1c9bf22c | 70 | calling split_huge_page(page). This is what the Linux VM does before |
a46e6376 KS |
71 | it tries to swapout the hugepage for example. split_huge_page() can fail |
72 | if the page is pinned and you must handle this correctly. | |
1c9bf22c AA |
73 | |
74 | Example to make mremap.c transparent hugepage aware with a one liner | |
44f380fe | 75 | change:: |
1c9bf22c | 76 | |
44f380fe MR |
77 | diff --git a/mm/mremap.c b/mm/mremap.c |
78 | --- a/mm/mremap.c | |
79 | +++ b/mm/mremap.c | |
80 | @@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru | |
81 | return NULL; | |
1c9bf22c | 82 | |
44f380fe MR |
83 | pmd = pmd_offset(pud, addr); |
84 | + split_huge_pmd(vma, pmd, addr); | |
85 | if (pmd_none_or_clear_bad(pmd)) | |
86 | return NULL; | |
1c9bf22c | 87 | |
44f380fe MR |
88 | Locking in hugepage aware code |
89 | ============================== | |
1c9bf22c AA |
90 | |
91 | We want as much code as possible hugepage aware, as calling | |
a46e6376 | 92 | split_huge_page() or split_huge_pmd() has a cost. |
1c9bf22c AA |
93 | |
94 | To make pagetable walks huge pmd aware, all you need to do is to call | |
95 | pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the | |
c1e8d7c6 | 96 | mmap_lock in read (or write) mode to be sure a huge pmd cannot be |
1c9bf22c | 97 | created from under you by khugepaged (khugepaged collapse_huge_page |
c1e8d7c6 | 98 | takes the mmap_lock in write mode in addition to the anon_vma lock). If |
1c9bf22c AA |
99 | pmd_trans_huge returns false, you just fallback in the old code |
100 | paths. If instead pmd_trans_huge returns true, you have to take the | |
a46e6376 | 101 | page table lock (pmd_lock()) and re-run pmd_trans_huge. Taking the |
41f0a954 | 102 | page table lock will prevent the huge pmd being converted into a |
a46e6376 | 103 | regular pmd from under you (split_huge_pmd can run in parallel to the |
1c9bf22c | 104 | pagetable walk). If the second pmd_trans_huge returns false, you |
a46e6376 | 105 | should just drop the page table lock and fallback to the old code as |
41f0a954 RC |
106 | before. Otherwise, you can proceed to process the huge pmd and the |
107 | hugepage natively. Once finished, you can drop the page table lock. | |
a46e6376 | 108 | |
44f380fe MR |
109 | Refcounts and transparent huge pages |
110 | ==================================== | |
a46e6376 KS |
111 | |
112 | Refcounting on THP is mostly consistent with refcounting on other compound | |
113 | pages: | |
114 | ||
41f0a954 | 115 | - get_page()/put_page() and GUP operate on head page's ->_refcount. |
a46e6376 | 116 | |
0139aa7b | 117 | - ->_refcount in tail pages is always zero: get_page_unless_zero() never |
41f0a954 | 118 | succeeds on tail pages. |
a46e6376 | 119 | |
9bd3155e | 120 | - map/unmap of PMD entry for the whole compound page increment/decrement |
4b51634c HD |
121 | ->compound_mapcount, stored in the first tail page of the compound page; |
122 | and also increment/decrement ->subpages_mapcount (also in the first tail) | |
123 | by COMPOUND_MAPPED when compound_mapcount goes from -1 to 0 or 0 to -1. | |
9bd3155e HD |
124 | |
125 | - map/unmap of sub-pages with PTE entry increment/decrement ->_mapcount | |
126 | on relevant sub-page of the compound page, and also increment/decrement | |
be5ef2d9 HD |
127 | ->subpages_mapcount, stored in first tail page of the compound page, when |
128 | _mapcount goes from -1 to 0 or 0 to -1: counting sub-pages mapped by PTE. | |
a46e6376 | 129 | |
1c9bf22c | 130 | split_huge_page internally has to distribute the refcounts in the head |
a46e6376 KS |
131 | page to the tail pages before clearing all PG_head/tail bits from the page |
132 | structures. It can be done easily for refcounts taken by page table | |
41f0a954 | 133 | entries, but we don't have enough information on how to distribute any |
a46e6376 | 134 | additional pins (i.e. from get_user_pages). split_huge_page() fails any |
41f0a954 RC |
135 | requests to split pinned huge pages: it expects page count to be equal to |
136 | the sum of mapcount of all sub-pages plus one (split_huge_page caller must | |
137 | have a reference to the head page). | |
a46e6376 | 138 | |
0139aa7b | 139 | split_huge_page uses migration entries to stabilize page->_refcount and |
41f0a954 | 140 | page->_mapcount of anonymous pages. File pages just get unmapped. |
a46e6376 | 141 | |
41f0a954 RC |
142 | We are safe against physical memory scanners too: the only legitimate way |
143 | a scanner can get a reference to a page is get_page_unless_zero(). | |
a46e6376 | 144 | |
89474d50 EE |
145 | All tail pages have zero ->_refcount until atomic_add(). This prevents the |
146 | scanner from getting a reference to the tail page up to that point. After the | |
41f0a954 | 147 | atomic_add() we don't care about the ->_refcount value. We already know how |
89474d50 | 148 | many references should be uncharged from the head page. |
a46e6376 KS |
149 | |
150 | For head page get_page_unless_zero() will succeed and we don't mind. It's | |
41f0a954 | 151 | clear where references should go after split: it will stay on the head page. |
a46e6376 | 152 | |
41f0a954 | 153 | Note that split_huge_pmd() doesn't have any limitations on refcounting: |
a46e6376 KS |
154 | pmd can be split at any point and never fails. |
155 | ||
44f380fe MR |
156 | Partial unmap and deferred_split_huge_page() |
157 | ============================================ | |
a46e6376 KS |
158 | |
159 | Unmapping part of THP (with munmap() or other way) is not going to free | |
160 | memory immediately. Instead, we detect that a subpage of THP is not in use | |
161 | in page_remove_rmap() and queue the THP for splitting if memory pressure | |
162 | comes. Splitting will free up unused subpages. | |
163 | ||
164 | Splitting the page right away is not an option due to locking context in | |
41f0a954 | 165 | the place where we can detect partial unmap. It also might be |
929f9d28 SP |
166 | counterproductive since in many cases partial unmap happens during exit(2) if |
167 | a THP crosses a VMA boundary. | |
a46e6376 | 168 | |
41f0a954 | 169 | The function deferred_split_huge_page() is used to queue a page for splitting. |
a46e6376 KS |
170 | The splitting itself will happen when we get memory pressure via shrinker |
171 | interface. |