]>
Commit | Line | Data |
---|---|---|
1 | .. _transhuge: | |
2 | ||
3 | ============================ | |
4 | Transparent Hugepage Support | |
5 | ============================ | |
6 | ||
7 | This document describes design principles for Transparent Hugepage (THP) | |
8 | support and its interaction with other parts of the memory management | |
9 | system. | |
10 | ||
11 | Design principles | |
12 | ================= | |
13 | ||
14 | - "graceful fallback": mm components which don't have transparent hugepage | |
15 | knowledge fall back to breaking huge pmd mapping into table of ptes and, | |
16 | if necessary, split a transparent hugepage. Therefore these components | |
17 | can continue working on the regular pages or regular pte mappings. | |
18 | ||
19 | - if a hugepage allocation fails because of memory fragmentation, | |
20 | regular pages should be gracefully allocated instead and mixed in | |
21 | the same vma without any failure or significant delay and without | |
22 | userland noticing | |
23 | ||
24 | - if some task quits and more hugepages become available (either | |
25 | immediately in the buddy or through the VM), guest physical memory | |
26 | backed by regular pages should be relocated on hugepages | |
27 | automatically (with khugepaged) | |
28 | ||
29 | - it doesn't require memory reservation and in turn it uses hugepages | |
30 | whenever possible (the only possible reservation here is kernelcore= | |
31 | to avoid unmovable pages to fragment all the memory but such a tweak | |
32 | is not specific to transparent hugepage support and it's a generic | |
33 | feature that applies to all dynamic high order allocations in the | |
34 | kernel) | |
35 | ||
36 | get_user_pages and follow_page | |
37 | ============================== | |
38 | ||
39 | get_user_pages and follow_page if run on a hugepage, will return the | |
40 | head or tail pages as usual (exactly as they would do on | |
41 | hugetlbfs). Most GUP users will only care about the actual physical | |
42 | address of the page and its temporary pinning to release after the I/O | |
43 | is complete, so they won't ever notice the fact the page is huge. But | |
44 | if any driver is going to mangle over the page structure of the tail | |
45 | page (like for checking page->mapping or other bits that are relevant | |
46 | for the head page and not the tail page), it should be updated to jump | |
47 | to check head page instead. Taking a reference on any head/tail page would | |
48 | prevent the page from being split by anyone. | |
49 | ||
50 | .. note:: | |
51 | these aren't new constraints to the GUP API, and they match the | |
52 | same constraints that apply to hugetlbfs too, so any driver capable | |
53 | of handling GUP on hugetlbfs will also work fine on transparent | |
54 | hugepage backed mappings. | |
55 | ||
56 | In case you can't handle compound pages if they're returned by | |
57 | follow_page, the FOLL_SPLIT bit can be specified as a parameter to | |
58 | follow_page, so that it will split the hugepages before returning | |
59 | them. | |
60 | ||
61 | Graceful fallback | |
62 | ================= | |
63 | ||
64 | Code walking pagetables but unaware about huge pmds can simply call | |
65 | split_huge_pmd(vma, pmd, addr) where the pmd is the one returned by | |
66 | pmd_offset. It's trivial to make the code transparent hugepage aware | |
67 | by just grepping for "pmd_offset" and adding split_huge_pmd where | |
68 | missing after pmd_offset returns the pmd. Thanks to the graceful | |
69 | fallback design, with a one liner change, you can avoid to write | |
70 | hundreds if not thousands of lines of complex code to make your code | |
71 | hugepage aware. | |
72 | ||
73 | If you're not walking pagetables but you run into a physical hugepage | |
74 | that you can't handle natively in your code, you can split it by | |
75 | calling split_huge_page(page). This is what the Linux VM does before | |
76 | it tries to swapout the hugepage for example. split_huge_page() can fail | |
77 | if the page is pinned and you must handle this correctly. | |
78 | ||
79 | Example to make mremap.c transparent hugepage aware with a one liner | |
80 | change:: | |
81 | ||
82 | diff --git a/mm/mremap.c b/mm/mremap.c | |
83 | --- a/mm/mremap.c | |
84 | +++ b/mm/mremap.c | |
85 | @@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru | |
86 | return NULL; | |
87 | ||
88 | pmd = pmd_offset(pud, addr); | |
89 | + split_huge_pmd(vma, pmd, addr); | |
90 | if (pmd_none_or_clear_bad(pmd)) | |
91 | return NULL; | |
92 | ||
93 | Locking in hugepage aware code | |
94 | ============================== | |
95 | ||
96 | We want as much code as possible hugepage aware, as calling | |
97 | split_huge_page() or split_huge_pmd() has a cost. | |
98 | ||
99 | To make pagetable walks huge pmd aware, all you need to do is to call | |
100 | pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the | |
101 | mmap_lock in read (or write) mode to be sure a huge pmd cannot be | |
102 | created from under you by khugepaged (khugepaged collapse_huge_page | |
103 | takes the mmap_lock in write mode in addition to the anon_vma lock). If | |
104 | pmd_trans_huge returns false, you just fallback in the old code | |
105 | paths. If instead pmd_trans_huge returns true, you have to take the | |
106 | page table lock (pmd_lock()) and re-run pmd_trans_huge. Taking the | |
107 | page table lock will prevent the huge pmd being converted into a | |
108 | regular pmd from under you (split_huge_pmd can run in parallel to the | |
109 | pagetable walk). If the second pmd_trans_huge returns false, you | |
110 | should just drop the page table lock and fallback to the old code as | |
111 | before. Otherwise, you can proceed to process the huge pmd and the | |
112 | hugepage natively. Once finished, you can drop the page table lock. | |
113 | ||
114 | Refcounts and transparent huge pages | |
115 | ==================================== | |
116 | ||
117 | Refcounting on THP is mostly consistent with refcounting on other compound | |
118 | pages: | |
119 | ||
120 | - get_page()/put_page() and GUP operate on head page's ->_refcount. | |
121 | ||
122 | - ->_refcount in tail pages is always zero: get_page_unless_zero() never | |
123 | succeeds on tail pages. | |
124 | ||
125 | - map/unmap of the pages with PTE entry increment/decrement ->_mapcount | |
126 | on relevant sub-page of the compound page. | |
127 | ||
128 | - map/unmap of the whole compound page is accounted for in compound_mapcount | |
129 | (stored in first tail page). For file huge pages, we also increment | |
130 | ->_mapcount of all sub-pages in order to have race-free detection of | |
131 | last unmap of subpages. | |
132 | ||
133 | PageDoubleMap() indicates that the page is *possibly* mapped with PTEs. | |
134 | ||
135 | For anonymous pages, PageDoubleMap() also indicates ->_mapcount in all | |
136 | subpages is offset up by one. This additional reference is required to | |
137 | get race-free detection of unmap of subpages when we have them mapped with | |
138 | both PMDs and PTEs. | |
139 | ||
140 | This optimization is required to lower the overhead of per-subpage mapcount | |
141 | tracking. The alternative is to alter ->_mapcount in all subpages on each | |
142 | map/unmap of the whole compound page. | |
143 | ||
144 | For anonymous pages, we set PG_double_map when a PMD of the page is split | |
145 | for the first time, but still have a PMD mapping. The additional references | |
146 | go away with the last compound_mapcount. | |
147 | ||
148 | File pages get PG_double_map set on the first map of the page with PTE and | |
149 | goes away when the page gets evicted from the page cache. | |
150 | ||
151 | split_huge_page internally has to distribute the refcounts in the head | |
152 | page to the tail pages before clearing all PG_head/tail bits from the page | |
153 | structures. It can be done easily for refcounts taken by page table | |
154 | entries, but we don't have enough information on how to distribute any | |
155 | additional pins (i.e. from get_user_pages). split_huge_page() fails any | |
156 | requests to split pinned huge pages: it expects page count to be equal to | |
157 | the sum of mapcount of all sub-pages plus one (split_huge_page caller must | |
158 | have a reference to the head page). | |
159 | ||
160 | split_huge_page uses migration entries to stabilize page->_refcount and | |
161 | page->_mapcount of anonymous pages. File pages just get unmapped. | |
162 | ||
163 | We are safe against physical memory scanners too: the only legitimate way | |
164 | a scanner can get a reference to a page is get_page_unless_zero(). | |
165 | ||
166 | All tail pages have zero ->_refcount until atomic_add(). This prevents the | |
167 | scanner from getting a reference to the tail page up to that point. After the | |
168 | atomic_add() we don't care about the ->_refcount value. We already know how | |
169 | many references should be uncharged from the head page. | |
170 | ||
171 | For head page get_page_unless_zero() will succeed and we don't mind. It's | |
172 | clear where references should go after split: it will stay on the head page. | |
173 | ||
174 | Note that split_huge_pmd() doesn't have any limitations on refcounting: | |
175 | pmd can be split at any point and never fails. | |
176 | ||
177 | Partial unmap and deferred_split_huge_page() | |
178 | ============================================ | |
179 | ||
180 | Unmapping part of THP (with munmap() or other way) is not going to free | |
181 | memory immediately. Instead, we detect that a subpage of THP is not in use | |
182 | in page_remove_rmap() and queue the THP for splitting if memory pressure | |
183 | comes. Splitting will free up unused subpages. | |
184 | ||
185 | Splitting the page right away is not an option due to locking context in | |
186 | the place where we can detect partial unmap. It also might be | |
187 | counterproductive since in many cases partial unmap happens during exit(2) if | |
188 | a THP crosses a VMA boundary. | |
189 | ||
190 | The function deferred_split_huge_page() is used to queue a page for splitting. | |
191 | The splitting itself will happen when we get memory pressure via shrinker | |
192 | interface. |