]>
Commit | Line | Data |
---|---|---|
1 | ============= | |
2 | OSD Throttles | |
3 | ============= | |
4 | ||
5 | There are three significant throttles in the filestore: wbthrottle, | |
6 | op_queue_throttle, and a throttle based on journal usage. | |
7 | ||
8 | WBThrottle | |
9 | ---------- | |
10 | The WBThrottle is defined in src/os/filestore/WBThrottle.[h,cc] and | |
11 | included in FileStore as FileStore::wbthrottle. The intention is to | |
12 | bound the amount of outstanding IO we need to do to flush the journal. | |
13 | At the same time, we don't want to necessarily do it inline in case we | |
14 | might be able to combine several IOs on the same object close together | |
15 | in time. Thus, in FileStore::_write, we queue the fd for asyncronous | |
16 | flushing and block in FileStore::_do_op if we have exceeded any hard | |
17 | limits until the background flusher catches up. | |
18 | ||
19 | The relevant config options are filestore_wbthrottle*. There are | |
20 | different defaults for xfs and btrfs. Each set has hard and soft | |
21 | limits on bytes (total dirty bytes), ios (total dirty ios), and | |
22 | inodes (total dirty fds). The WBThrottle will begin flushing | |
23 | when any of these hits the soft limit and will block in throttle() | |
24 | while any has exceeded the hard limit. | |
25 | ||
26 | Tighter soft limits will cause writeback to happen more quickly, | |
27 | but may cause the OSD to miss oportunities for write coalescing. | |
28 | Tighter hard limits may cause a reduction in latency variance by | |
29 | reducing time spent flushing the journal, but may reduce writeback | |
30 | parallelism. | |
31 | ||
32 | op_queue_throttle | |
33 | ----------------- | |
34 | The op queue throttle is intended to bound the amount of queued but | |
35 | uncompleted work in the filestore by delaying threads calling | |
36 | queue_transactions more and more based on how many ops and bytes are | |
37 | currently queued. The throttle is taken in queue_transactions and | |
38 | released when the op is applied to the filesystem. This period | |
39 | includes time spent in the journal queue, time spent writing to the | |
40 | journal, time spent in the actual op queue, time spent waiting for the | |
41 | wbthrottle to open up (thus, the wbthrottle can push back indirectly | |
42 | on the queue_transactions caller), and time spent actually applying | |
43 | the op to the filesystem. A BackoffThrottle is used to gradually | |
44 | delay the queueing thread after each throttle becomes more than | |
45 | filestore_queue_low_threshhold full (a ratio of | |
46 | filestore_queue_max_(bytes|ops)). The throttles will block once the | |
47 | max value is reached (filestore_queue_max_(bytes|ops)). | |
48 | ||
49 | The significant config options are: | |
50 | filestore_queue_low_threshhold | |
51 | filestore_queue_high_threshhold | |
52 | filestore_expected_throughput_ops | |
53 | filestore_expected_throughput_bytes | |
54 | filestore_queue_high_delay_multiple | |
55 | filestore_queue_max_delay_multiple | |
56 | ||
57 | While each throttle is at less than low_threshhold of the max, | |
58 | no delay happens. Between low and high, the throttle will | |
59 | inject a per-op delay (per op or byte) ramping from 0 at low to | |
60 | high_delay_multiple/expected_throughput at high. From high to | |
61 | 1, the delay will ramp from high_delay_multiple/expected_throughput | |
62 | to max_delay_multiple/expected_throughput. | |
63 | ||
64 | filestore_queue_high_delay_multiple and | |
65 | filestore_queue_max_delay_multiple probably do not need to be | |
66 | changed. | |
67 | ||
68 | Setting these properly should help to smooth out op latencies by | |
69 | mostly avoiding the hard limit. | |
70 | ||
71 | See FileStore::throttle_ops and FileSTore::thottle_bytes. | |
72 | ||
73 | journal usage throttle | |
74 | ---------------------- | |
75 | See src/os/filestore/JournalThrottle.h/cc | |
76 | ||
77 | The intention of the journal usage throttle is to gradually slow | |
78 | down queue_transactions callers as the journal fills up in order | |
79 | to smooth out hiccup during filestore syncs. JournalThrottle | |
80 | wraps a BackoffThrottle and tracks journaled but not flushed | |
81 | journal entries so that the throttle can be released when the | |
82 | journal is flushed. The configs work very similarly to the | |
83 | op_queue_throttle. | |
84 | ||
85 | The significant config options are: | |
86 | journal_throttle_low_threshhold | |
87 | journal_throttle_high_threshhold | |
88 | filestore_expected_throughput_ops | |
89 | filestore_expected_throughput_bytes | |
90 | journal_throttle_high_multiple | |
91 | journal_throttle_max_multiple | |
92 | ||
93 | .. literalinclude:: osd_throttles.txt |