]>
Commit | Line | Data |
---|---|---|
c6b4fcba JT |
1 | Introduction |
2 | ============ | |
3 | ||
4 | dm-cache is a device mapper target written by Joe Thornber, Heinz | |
5 | Mauelshagen, and Mike Snitzer. | |
6 | ||
7 | It aims to improve performance of a block device (eg, a spindle) by | |
8 | dynamically migrating some of its data to a faster, smaller device | |
9 | (eg, an SSD). | |
10 | ||
11 | This device-mapper solution allows us to insert this caching at | |
12 | different levels of the dm stack, for instance above the data device for | |
13 | a thin-provisioning pool. Caching solutions that are integrated more | |
14 | closely with the virtual memory system should give better performance. | |
15 | ||
16 | The target reuses the metadata library used in the thin-provisioning | |
17 | library. | |
18 | ||
19 | The decision as to what data to migrate and when is left to a plug-in | |
20 | policy module. Several of these have been written as we experiment, | |
21 | and we hope other people will contribute others for specific io | |
22 | scenarios (eg. a vm image server). | |
23 | ||
24 | Glossary | |
25 | ======== | |
26 | ||
27 | Migration - Movement of the primary copy of a logical block from one | |
28 | device to the other. | |
29 | Promotion - Migration from slow device to fast device. | |
30 | Demotion - Migration from fast device to slow device. | |
31 | ||
32 | The origin device always contains a copy of the logical block, which | |
33 | may be out of date or kept in sync with the copy on the cache device | |
34 | (depending on policy). | |
35 | ||
36 | Design | |
37 | ====== | |
38 | ||
39 | Sub-devices | |
40 | ----------- | |
41 | ||
42 | The target is constructed by passing three devices to it (along with | |
43 | other parameters detailed later): | |
44 | ||
45 | 1. An origin device - the big, slow one. | |
46 | ||
47 | 2. A cache device - the small, fast one. | |
48 | ||
49 | 3. A small metadata device - records which blocks are in the cache, | |
50 | which are dirty, and extra hints for use by the policy object. | |
51 | This information could be put on the cache device, but having it | |
52 | separate allows the volume manager to configure it differently, | |
66bb2644 MS |
53 | e.g. as a mirror for extra robustness. This metadata device may only |
54 | be used by a single cache device. | |
c6b4fcba JT |
55 | |
56 | Fixed block size | |
57 | ---------------- | |
58 | ||
59 | The origin is divided up into blocks of a fixed size. This block size | |
60 | is configurable when you first create the cache. Typically we've been | |
05473044 MS |
61 | using block sizes of 256KB - 1024KB. The block size must be between 64 |
62 | (32KB) and 2097152 (1GB) and a multiple of 64 (32KB). | |
c6b4fcba JT |
63 | |
64 | Having a fixed block size simplifies the target a lot. But it is | |
65 | something of a compromise. For instance, a small part of a block may be | |
66 | getting hit a lot, yet the whole block will be promoted to the cache. | |
67 | So large block sizes are bad because they waste cache space. And small | |
68 | block sizes are bad because they increase the amount of metadata (both | |
69 | in core and on disk). | |
70 | ||
2ee57d58 JT |
71 | Cache operating modes |
72 | --------------------- | |
c6b4fcba | 73 | |
2ee57d58 JT |
74 | The cache has three operating modes: writeback, writethrough and |
75 | passthrough. | |
c6b4fcba JT |
76 | |
77 | If writeback, the default, is selected then a write to a block that is | |
78 | cached will go only to the cache and the block will be marked dirty in | |
79 | the metadata. | |
80 | ||
81 | If writethrough is selected then a write to a cached block will not | |
82 | complete until it has hit both the origin and cache devices. Clean | |
83 | blocks should remain clean. | |
84 | ||
2ee57d58 JT |
85 | If passthrough is selected, useful when the cache contents are not known |
86 | to be coherent with the origin device, then all reads are served from | |
87 | the origin device (all reads miss the cache) and all writes are | |
88 | forwarded to the origin device; additionally, write hits cause cache | |
7b6b2bc9 MS |
89 | block invalidates. To enable passthrough mode the cache must be clean. |
90 | Passthrough mode allows a cache device to be activated without having to | |
91 | worry about coherency. Coherency that exists is maintained, although | |
92 | the cache will gradually cool as writes take place. If the coherency of | |
93 | the cache can later be verified, or established through use of the | |
94 | "invalidate_cblocks" message, the cache device can be transitioned to | |
95 | writethrough or writeback mode while still warm. Otherwise, the cache | |
96 | contents can be discarded prior to transitioning to the desired | |
97 | operating mode. | |
2ee57d58 | 98 | |
c6b4fcba | 99 | A simple cleaner policy is provided, which will clean (write back) all |
7b6b2bc9 MS |
100 | dirty blocks in a cache. Useful for decommissioning a cache or when |
101 | shrinking a cache. Shrinking the cache's fast device requires all cache | |
102 | blocks, in the area of the cache being removed, to be clean. If the | |
103 | area being removed from the cache still contains dirty blocks the resize | |
104 | will fail. Care must be taken to never reduce the volume used for the | |
105 | cache's fast device until the cache is clean. This is of particular | |
106 | importance if writeback mode is used. Writethrough and passthrough | |
107 | modes already maintain a clean cache. Future support to partially clean | |
108 | the cache, above a specified threshold, will allow for keeping the cache | |
109 | warm and in writeback mode during resize. | |
c6b4fcba JT |
110 | |
111 | Migration throttling | |
112 | -------------------- | |
113 | ||
114 | Migrating data between the origin and cache device uses bandwidth. | |
115 | The user can set a throttle to prevent more than a certain amount of | |
f884ab15 | 116 | migration occurring at any one time. Currently we're not taking any |
c6b4fcba JT |
117 | account of normal io traffic going to the devices. More work needs |
118 | doing here to avoid migrating during those peak io moments. | |
119 | ||
120 | For the time being, a message "migration_threshold <#sectors>" | |
121 | can be used to set the maximum number of sectors being migrated, | |
122 | the default being 204800 sectors (or 100MB). | |
123 | ||
124 | Updating on-disk metadata | |
125 | ------------------------- | |
126 | ||
127 | On-disk metadata is committed every time a REQ_SYNC or REQ_FUA bio is | |
128 | written. If no such requests are made then commits will occur every | |
129 | second. This means the cache behaves like a physical disk that has a | |
130 | write cache (the same is true of the thin-provisioning target). If | |
131 | power is lost you may lose some recent writes. The metadata should | |
132 | always be consistent in spite of any crash. | |
133 | ||
134 | The 'dirty' state for a cache block changes far too frequently for us | |
135 | to keep updating it on the fly. So we treat it as a hint. In normal | |
136 | operation it will be written when the dm device is suspended. If the | |
137 | system crashes all cache blocks will be assumed dirty when restarted. | |
138 | ||
139 | Per-block policy hints | |
140 | ---------------------- | |
141 | ||
142 | Policy plug-ins can store a chunk of data per cache block. It's up to | |
143 | the policy how big this chunk is, but it should be kept small. Like the | |
144 | dirty flags this data is lost if there's a crash so a safe fallback | |
145 | value should always be possible. | |
146 | ||
147 | For instance, the 'mq' policy, which is currently the default policy, | |
148 | uses this facility to store the hit count of the cache blocks. If | |
149 | there's a crash this information will be lost, which means the cache | |
150 | may be less efficient until those hit counts are regenerated. | |
151 | ||
152 | Policy hints affect performance, not correctness. | |
153 | ||
154 | Policy messaging | |
155 | ---------------- | |
156 | ||
157 | Policies will have different tunables, specific to each one, so we | |
158 | need a generic way of getting and setting these. Device-mapper | |
159 | messages are used. Refer to cache-policies.txt. | |
160 | ||
161 | Discard bitset resolution | |
162 | ------------------------- | |
163 | ||
164 | We can avoid copying data during migration if we know the block has | |
165 | been discarded. A prime example of this is when mkfs discards the | |
166 | whole block device. We store a bitset tracking the discard state of | |
167 | blocks. However, we allow this bitset to have a different block size | |
168 | from the cache blocks. This is because we need to track the discard | |
169 | state for all of the origin device (compare with the dirty bitset | |
170 | which is just for the smaller cache device). | |
171 | ||
172 | Target interface | |
173 | ================ | |
174 | ||
175 | Constructor | |
176 | ----------- | |
177 | ||
178 | cache <metadata dev> <cache dev> <origin dev> <block size> | |
179 | <#feature args> [<feature arg>]* | |
180 | <policy> <#policy args> [policy args]* | |
181 | ||
182 | metadata dev : fast device holding the persistent metadata | |
183 | cache dev : fast device holding cached data blocks | |
184 | origin dev : slow device holding original data blocks | |
185 | block size : cache unit size in sectors | |
186 | ||
187 | #feature args : number of feature arguments passed | |
7b6b2bc9 | 188 | feature args : writethrough or passthrough (The default is writeback.) |
c6b4fcba JT |
189 | |
190 | policy : the replacement policy to use | |
191 | #policy args : an even number of arguments corresponding to | |
192 | key/value pairs passed to the policy | |
193 | policy args : key/value pairs passed to the policy | |
194 | E.g. 'sequential_threshold 1024' | |
195 | See cache-policies.txt for details. | |
196 | ||
197 | Optional feature arguments are: | |
198 | writethrough : write through caching that prohibits cache block | |
199 | content from being different from origin block content. | |
200 | Without this argument, the default behaviour is to write | |
201 | back cache block contents later for performance reasons, | |
202 | so they may differ from the corresponding origin blocks. | |
203 | ||
7b6b2bc9 MS |
204 | passthrough : a degraded mode useful for various cache coherency |
205 | situations (e.g., rolling back snapshots of | |
206 | underlying storage). Reads and writes always go to | |
207 | the origin. If a write goes to a cached origin | |
208 | block, then the cache block is invalidated. | |
209 | To enable passthrough mode the cache must be clean. | |
210 | ||
c6b4fcba JT |
211 | A policy called 'default' is always registered. This is an alias for |
212 | the policy we currently think is giving best all round performance. | |
213 | ||
214 | As the default policy could vary between kernels, if you are relying on | |
215 | the characteristics of a specific policy, always request it by name. | |
216 | ||
217 | Status | |
218 | ------ | |
219 | ||
220 | <#used metadata blocks>/<#total metadata blocks> <#read hits> <#read misses> | |
221 | <#write hits> <#write misses> <#demotions> <#promotions> <#blocks in cache> | |
222 | <#dirty> <#features> <features>* <#core args> <core args>* <#policy args> | |
223 | <policy args>* | |
224 | ||
225 | #used metadata blocks : Number of metadata blocks used | |
226 | #total metadata blocks : Total number of metadata blocks | |
227 | #read hits : Number of times a READ bio has been mapped | |
228 | to the cache | |
229 | #read misses : Number of times a READ bio has been mapped | |
230 | to the origin | |
231 | #write hits : Number of times a WRITE bio has been mapped | |
232 | to the cache | |
233 | #write misses : Number of times a WRITE bio has been | |
234 | mapped to the origin | |
235 | #demotions : Number of times a block has been removed | |
236 | from the cache | |
237 | #promotions : Number of times a block has been moved to | |
238 | the cache | |
239 | #blocks in cache : Number of blocks resident in the cache | |
240 | #dirty : Number of blocks in the cache that differ | |
241 | from the origin | |
242 | #feature args : Number of feature args to follow | |
243 | feature args : 'writethrough' (optional) | |
244 | #core args : Number of core arguments (must be even) | |
245 | core args : Key/value pairs for tuning the core | |
246 | e.g. migration_threshold | |
247 | #policy args : Number of policy arguments to follow (must be even) | |
248 | policy args : Key/value pairs | |
249 | e.g. 'sequential_threshold 1024 | |
250 | ||
251 | Messages | |
252 | -------- | |
253 | ||
254 | Policies will have different tunables, specific to each one, so we | |
255 | need a generic way of getting and setting these. Device-mapper | |
256 | messages are used. (A sysfs interface would also be possible.) | |
257 | ||
258 | The message format is: | |
259 | ||
260 | <key> <value> | |
261 | ||
262 | E.g. | |
263 | dmsetup message my_cache 0 sequential_threshold 1024 | |
264 | ||
65790ff9 JT |
265 | |
266 | Invalidation is removing an entry from the cache without writing it | |
267 | back. Cache blocks can be invalidated via the invalidate_cblocks | |
7b6b2bc9 | 268 | message, which takes an arbitrary number of cblock ranges. Each cblock |
83f539e1 MS |
269 | range's end value is "one past the end", meaning 5-10 expresses a range |
270 | of values from 5 to 9. Each cblock must be expressed as a decimal | |
271 | value, in the future a variant message that takes cblock ranges | |
272 | expressed in hexidecimal may be needed to better support efficient | |
273 | invalidation of larger caches. The cache must be in passthrough mode | |
274 | when invalidate_cblocks is used. | |
65790ff9 JT |
275 | |
276 | invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]* | |
277 | ||
278 | E.g. | |
279 | dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789 | |
280 | ||
c6b4fcba JT |
281 | Examples |
282 | ======== | |
283 | ||
284 | The test suite can be found here: | |
285 | ||
65790ff9 | 286 | https://github.com/jthornber/device-mapper-test-suite |
c6b4fcba JT |
287 | |
288 | dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ | |
289 | /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0' | |
290 | dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ | |
291 | /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \ | |
292 | mq 4 sequential_threshold 1024 random_threshold 8' |