]>
Commit | Line | Data |
---|---|---|
cafe5635 KO |
1 | Say you've got a big slow raid 6, and an X-25E or three. Wouldn't it be |
2 | nice if you could use them as cache... Hence bcache. | |
3 | ||
4 | Wiki and git repositories are at: | |
5 | http://bcache.evilpiepirate.org | |
6 | http://evilpiepirate.org/git/linux-bcache.git | |
7 | http://evilpiepirate.org/git/bcache-tools.git | |
8 | ||
9 | It's designed around the performance characteristics of SSDs - it only allocates | |
10 | in erase block sized buckets, and it uses a hybrid btree/log to track cached | |
11 | extants (which can be anywhere from a single sector to the bucket size). It's | |
12 | designed to avoid random writes at all costs; it fills up an erase block | |
13 | sequentially, then issues a discard before reusing it. | |
14 | ||
15 | Both writethrough and writeback caching are supported. Writeback defaults to | |
16 | off, but can be switched on and off arbitrarily at runtime. Bcache goes to | |
17 | great lengths to protect your data - it reliably handles unclean shutdown. (It | |
18 | doesn't even have a notion of a clean shutdown; bcache simply doesn't return | |
19 | writes as completed until they're on stable storage). | |
20 | ||
21 | Writeback caching can use most of the cache for buffering writes - writing | |
22 | dirty data to the backing device is always done sequentially, scanning from the | |
23 | start to the end of the index. | |
24 | ||
25 | Since random IO is what SSDs excel at, there generally won't be much benefit | |
26 | to caching large sequential IO. Bcache detects sequential IO and skips it; | |
27 | it also keeps a rolling average of the IO sizes per task, and as long as the | |
28 | average is above the cutoff it will skip all IO from that task - instead of | |
29 | caching the first 512k after every seek. Backups and large file copies should | |
30 | thus entirely bypass the cache. | |
31 | ||
32 | In the event of a data IO error on the flash it will try to recover by reading | |
33 | from disk or invalidating cache entries. For unrecoverable errors (meta data | |
34 | or dirty data), caching is automatically disabled; if dirty data was present | |
35 | in the cache it first disables writeback caching and waits for all dirty data | |
36 | to be flushed. | |
37 | ||
38 | Getting started: | |
39 | You'll need make-bcache from the bcache-tools repository. Both the cache device | |
40 | and backing device must be formatted before use. | |
41 | make-bcache -B /dev/sdb | |
42 | make-bcache -C /dev/sdc | |
43 | ||
44 | make-bcache has the ability to format multiple devices at the same time - if | |
45 | you format your backing devices and cache device at the same time, you won't | |
46 | have to manually attach: | |
47 | make-bcache -B /dev/sda /dev/sdb -C /dev/sdc | |
48 | ||
49 | To make bcache devices known to the kernel, echo them to /sys/fs/bcache/register: | |
50 | ||
51 | echo /dev/sdb > /sys/fs/bcache/register | |
52 | echo /dev/sdc > /sys/fs/bcache/register | |
53 | ||
54 | To register your bcache devices automatically, you could add something like | |
55 | this to an init script: | |
56 | ||
57 | echo /dev/sd* > /sys/fs/bcache/register_quiet | |
58 | ||
59 | It'll look for bcache superblocks and ignore everything that doesn't have one. | |
60 | ||
61 | Registering the backing device makes the bcache show up in /dev; you can now | |
62 | format it and use it as normal. But the first time using a new bcache device, | |
63 | it'll be running in passthrough mode until you attach it to a cache. See the | |
64 | section on attaching. | |
65 | ||
66 | The devices show up at /dev/bcacheN, and can be controlled via sysfs from | |
67 | /sys/block/bcacheN/bcache: | |
68 | ||
69 | mkfs.ext4 /dev/bcache0 | |
70 | mount /dev/bcache0 /mnt | |
71 | ||
72 | Cache devices are managed as sets; multiple caches per set isn't supported yet | |
73 | but will allow for mirroring of metadata and dirty data in the future. Your new | |
74 | cache set shows up as /sys/fs/bcache/<UUID> | |
75 | ||
76 | ATTACHING: | |
77 | ||
78 | After your cache device and backing device are registered, the backing device | |
79 | must be attached to your cache set to enable caching. Attaching a backing | |
80 | device to a cache set is done thusly, with the UUID of the cache set in | |
81 | /sys/fs/bcache: | |
82 | ||
83 | echo <UUID> > /sys/block/bcache0/bcache/attach | |
84 | ||
85 | This only has to be done once. The next time you reboot, just reregister all | |
86 | your bcache devices. If a backing device has data in a cache somewhere, the | |
87 | /dev/bcache# device won't be created until the cache shows up - particularly | |
88 | important if you have writeback caching turned on. | |
89 | ||
90 | If you're booting up and your cache device is gone and never coming back, you | |
91 | can force run the backing device: | |
92 | ||
93 | echo 1 > /sys/block/sdb/bcache/running | |
94 | ||
95 | (You need to use /sys/block/sdb (or whatever your backing device is called), not | |
96 | /sys/block/bcache0, because bcache0 doesn't exist yet. If you're using a | |
97 | partition, the bcache directory would be at /sys/block/sdb/sdb2/bcache) | |
98 | ||
99 | The backing device will still use that cache set if it shows up in the future, | |
100 | but all the cached data will be invalidated. If there was dirty data in the | |
101 | cache, don't expect the filesystem to be recoverable - you will have massive | |
102 | filesystem corruption, though ext4's fsck does work miracles. | |
103 | ||
104 | SYSFS - BACKING DEVICE: | |
105 | ||
106 | attach | |
107 | Echo the UUID of a cache set to this file to enable caching. | |
108 | ||
109 | cache_mode | |
110 | Can be one of either writethrough, writeback, writearound or none. | |
111 | ||
112 | clear_stats | |
113 | Writing to this file resets the running total stats (not the day/hour/5 minute | |
114 | decaying versions). | |
115 | ||
116 | detach | |
117 | Write to this file to detach from a cache set. If there is dirty data in the | |
118 | cache, it will be flushed first. | |
119 | ||
120 | dirty_data | |
121 | Amount of dirty data for this backing device in the cache. Continuously | |
122 | updated unlike the cache set's version, but may be slightly off. | |
123 | ||
124 | label | |
125 | Name of underlying device. | |
126 | ||
127 | readahead | |
128 | Size of readahead that should be performed. Defaults to 0. If set to e.g. | |
129 | 1M, it will round cache miss reads up to that size, but without overlapping | |
130 | existing cache entries. | |
131 | ||
132 | running | |
133 | 1 if bcache is running (i.e. whether the /dev/bcache device exists, whether | |
134 | it's in passthrough mode or caching). | |
135 | ||
136 | sequential_cutoff | |
137 | A sequential IO will bypass the cache once it passes this threshhold; the | |
138 | most recent 128 IOs are tracked so sequential IO can be detected even when | |
139 | it isn't all done at once. | |
140 | ||
141 | sequential_merge | |
142 | If non zero, bcache keeps a list of the last 128 requests submitted to compare | |
143 | against all new requests to determine which new requests are sequential | |
144 | continuations of previous requests for the purpose of determining sequential | |
145 | cutoff. This is necessary if the sequential cutoff value is greater than the | |
146 | maximum acceptable sequential size for any single request. | |
147 | ||
148 | state | |
149 | The backing device can be in one of four different states: | |
150 | ||
151 | no cache: Has never been attached to a cache set. | |
152 | ||
153 | clean: Part of a cache set, and there is no cached dirty data. | |
154 | ||
155 | dirty: Part of a cache set, and there is cached dirty data. | |
156 | ||
157 | inconsistent: The backing device was forcibly run by the user when there was | |
158 | dirty data cached but the cache set was unavailable; whatever data was on the | |
159 | backing device has likely been corrupted. | |
160 | ||
161 | stop | |
162 | Write to this file to shut down the bcache device and close the backing | |
163 | device. | |
164 | ||
165 | writeback_delay | |
166 | When dirty data is written to the cache and it previously did not contain | |
167 | any, waits some number of seconds before initiating writeback. Defaults to | |
168 | 30. | |
169 | ||
170 | writeback_percent | |
171 | If nonzero, bcache tries to keep around this percentage of the cache dirty by | |
172 | throttling background writeback and using a PD controller to smoothly adjust | |
173 | the rate. | |
174 | ||
175 | writeback_rate | |
176 | Rate in sectors per second - if writeback_percent is nonzero, background | |
177 | writeback is throttled to this rate. Continuously adjusted by bcache but may | |
178 | also be set by the user. | |
179 | ||
180 | writeback_running | |
181 | If off, writeback of dirty data will not take place at all. Dirty data will | |
182 | still be added to the cache until it is mostly full; only meant for | |
183 | benchmarking. Defaults to on. | |
184 | ||
185 | SYSFS - BACKING DEVICE STATS: | |
186 | ||
187 | There are directories with these numbers for a running total, as well as | |
188 | versions that decay over the past day, hour and 5 minutes; they're also | |
189 | aggregated in the cache set directory as well. | |
190 | ||
191 | bypassed | |
192 | Amount of IO (both reads and writes) that has bypassed the cache | |
193 | ||
194 | cache_hits | |
195 | cache_misses | |
196 | cache_hit_ratio | |
197 | Hits and misses are counted per individual IO as bcache sees them; a | |
198 | partial hit is counted as a miss. | |
199 | ||
200 | cache_bypass_hits | |
201 | cache_bypass_misses | |
202 | Hits and misses for IO that is intended to skip the cache are still counted, | |
203 | but broken out here. | |
204 | ||
205 | cache_miss_collisions | |
206 | Counts instances where data was going to be inserted into the cache from a | |
207 | cache miss, but raced with a write and data was already present (usually 0 | |
208 | since the synchronization for cache misses was rewritten) | |
209 | ||
210 | cache_readaheads | |
211 | Count of times readahead occured. | |
212 | ||
213 | SYSFS - CACHE SET: | |
214 | ||
215 | average_key_size | |
216 | Average data per key in the btree. | |
217 | ||
218 | bdev<0..n> | |
219 | Symlink to each of the attached backing devices. | |
220 | ||
221 | block_size | |
222 | Block size of the cache devices. | |
223 | ||
224 | btree_cache_size | |
225 | Amount of memory currently used by the btree cache | |
226 | ||
227 | bucket_size | |
228 | Size of buckets | |
229 | ||
230 | cache<0..n> | |
231 | Symlink to each of the cache devices comprising this cache set. | |
232 | ||
233 | cache_available_percent | |
234 | Percentage of cache device free. | |
235 | ||
236 | clear_stats | |
237 | Clears the statistics associated with this cache | |
238 | ||
239 | dirty_data | |
240 | Amount of dirty data is in the cache (updated when garbage collection runs). | |
241 | ||
242 | flash_vol_create | |
243 | Echoing a size to this file (in human readable units, k/M/G) creates a thinly | |
244 | provisioned volume backed by the cache set. | |
245 | ||
246 | io_error_halflife | |
247 | io_error_limit | |
248 | These determines how many errors we accept before disabling the cache. | |
249 | Each error is decayed by the half life (in # ios). If the decaying count | |
250 | reaches io_error_limit dirty data is written out and the cache is disabled. | |
251 | ||
252 | journal_delay_ms | |
253 | Journal writes will delay for up to this many milliseconds, unless a cache | |
254 | flush happens sooner. Defaults to 100. | |
255 | ||
256 | root_usage_percent | |
257 | Percentage of the root btree node in use. If this gets too high the node | |
258 | will split, increasing the tree depth. | |
259 | ||
260 | stop | |
261 | Write to this file to shut down the cache set - waits until all attached | |
262 | backing devices have been shut down. | |
263 | ||
264 | tree_depth | |
265 | Depth of the btree (A single node btree has depth 0). | |
266 | ||
267 | unregister | |
268 | Detaches all backing devices and closes the cache devices; if dirty data is | |
269 | present it will disable writeback caching and wait for it to be flushed. | |
270 | ||
271 | SYSFS - CACHE SET INTERNAL: | |
272 | ||
273 | This directory also exposes timings for a number of internal operations, with | |
274 | separate files for average duration, average frequency, last occurence and max | |
275 | duration: garbage collection, btree read, btree node sorts and btree splits. | |
276 | ||
277 | active_journal_entries | |
278 | Number of journal entries that are newer than the index. | |
279 | ||
280 | btree_nodes | |
281 | Total nodes in the btree. | |
282 | ||
283 | btree_used_percent | |
284 | Average fraction of btree in use. | |
285 | ||
286 | bset_tree_stats | |
287 | Statistics about the auxiliary search trees | |
288 | ||
289 | btree_cache_max_chain | |
290 | Longest chain in the btree node cache's hash table | |
291 | ||
292 | cache_read_races | |
293 | Counts instances where while data was being read from the cache, the bucket | |
294 | was reused and invalidated - i.e. where the pointer was stale after the read | |
295 | completed. When this occurs the data is reread from the backing device. | |
296 | ||
297 | trigger_gc | |
298 | Writing to this file forces garbage collection to run. | |
299 | ||
300 | SYSFS - CACHE DEVICE: | |
301 | ||
302 | block_size | |
303 | Minimum granularity of writes - should match hardware sector size. | |
304 | ||
305 | btree_written | |
306 | Sum of all btree writes, in (kilo/mega/giga) bytes | |
307 | ||
308 | bucket_size | |
309 | Size of buckets | |
310 | ||
311 | cache_replacement_policy | |
312 | One of either lru, fifo or random. | |
313 | ||
314 | discard | |
315 | Boolean; if on a discard/TRIM will be issued to each bucket before it is | |
316 | reused. Defaults to off, since SATA TRIM is an unqueued command (and thus | |
317 | slow). | |
318 | ||
319 | freelist_percent | |
320 | Size of the freelist as a percentage of nbuckets. Can be written to to | |
321 | increase the number of buckets kept on the freelist, which lets you | |
322 | artificially reduce the size of the cache at runtime. Mostly for testing | |
323 | purposes (i.e. testing how different size caches affect your hit rate), but | |
324 | since buckets are discarded when they move on to the freelist will also make | |
325 | the SSD's garbage collection easier by effectively giving it more reserved | |
326 | space. | |
327 | ||
328 | io_errors | |
329 | Number of errors that have occured, decayed by io_error_halflife. | |
330 | ||
331 | metadata_written | |
332 | Sum of all non data writes (btree writes and all other metadata). | |
333 | ||
334 | nbuckets | |
335 | Total buckets in this cache | |
336 | ||
337 | priority_stats | |
338 | Statistics about how recently data in the cache has been accessed. This can | |
339 | reveal your working set size. | |
340 | ||
341 | written | |
342 | Sum of all data that has been written to the cache; comparison with | |
343 | btree_written gives the amount of write inflation in bcache. |