]>
Commit | Line | Data |
---|---|---|
f67539c2 TL |
1 | =============================== |
2 | RBD Persistent Read-only Cache | |
3 | =============================== | |
4 | ||
5 | .. index:: Ceph Block Device; Persistent Read-only Cache | |
6 | ||
7 | Shared, Read-only Parent Image Cache | |
8 | ==================================== | |
9 | ||
10 | `Cloned RBD images`_ usually modify only a small fraction of the parent | |
11 | image. For example, in a VDI use-case, VMs are cloned from the same | |
12 | base image and initially differ only by hostname and IP address. During | |
13 | booting, all of these VMs read portions of the same parent | |
14 | image data. If we have a local cache of the parent | |
15 | image, this speeds up reads on the caching host. We also achieve | |
16 | reduction of client-to-cluster network traffic. | |
17 | RBD cache must be explicitly enabled in | |
18 | ``ceph.conf``. The ``ceph-immutable-object-cache`` daemon is responsible for | |
19 | caching the parent content on the local disk, and future reads on that data | |
20 | will be serviced from the local cache. | |
21 | ||
22 | .. note:: RBD shared read-only parent image cache requires the Ceph Nautilus release or later. | |
23 | ||
24 | .. ditaa:: | |
25 | ||
26 | +--------------------------------------------------------+ | |
27 | | QEMU | | |
28 | +--------------------------------------------------------+ | |
29 | | librbd (cloned images) | | |
30 | +-------------------+-+----------------------------------+ | |
31 | | librados | | ceph--immutable--object--cache | | |
32 | +-------------------+ +----------------------------------+ | |
33 | | OSDs/Mons | | local cached parent image | | |
34 | +-------------------+ +----------------------------------+ | |
35 | ||
36 | ||
37 | Enable RBD Shared Read-only Parent Image Cache | |
38 | ---------------------------------------------- | |
39 | ||
40 | To enable RBD shared read-only parent image cache, the following Ceph settings | |
41 | need to added in the ``[client]`` `section`_ of your ``ceph.conf`` file:: | |
42 | ||
43 | rbd parent cache enabled = true | |
44 | rbd plugins = parent_cache | |
45 | ||
46 | Immutable Object Cache Daemon | |
47 | ============================= | |
48 | ||
49 | Introduction and Generic Settings | |
50 | --------------------------------- | |
51 | ||
52 | The ``ceph-immutable-object-cache`` daemon is responsible for caching parent | |
39ae355f TL |
53 | image content within its local caching directory. Using SSDs as the underlying |
54 | storage is recommended because doing so provides better performance. | |
f67539c2 TL |
55 | |
56 | The key components of the daemon are: | |
57 | ||
39ae355f TL |
58 | #. **Domain socket based IPC:** The daemon listens on a local domain socket at |
59 | startup and waits for connections from librbd clients. | |
60 | ||
61 | #. **LRU based promotion/demotion policy:** The daemon maintains in-memory | |
62 | statistics of cache hits for each cache file. It demotes the cold cache | |
63 | if capacity reaches the configured threshold. | |
64 | ||
65 | #. **File-based caching store:** The daemon maintains a simple file-based cache | |
66 | store. On promotion, the RADOS objects are fetched from RADOS cluster and | |
67 | stored in the local caching directory. | |
68 | ||
69 | When each cloned RBD image is opened, ``librbd`` tries to connect to the cache | |
70 | daemon through its Unix domain socket. After ``librbd`` is successfully | |
71 | connected, it coordinates with the daemon upon every subsequent read. In the | |
72 | case of an uncached read, the daemon promotes the RADOS object to the local | |
73 | caching directory and the next read of the object is serviced from the cache. | |
74 | The daemon maintains simple LRU statistics, which are used to evict cold cache | |
75 | files when required (for example, when the cache is at capacity and under | |
76 | pressure). | |
f67539c2 TL |
77 | |
78 | Here are some important cache configuration settings: | |
79 | ||
80 | ``immutable_object_cache_sock`` | |
81 | ||
82 | :Description: The path to the domain socket used for communication between | |
83 | librbd clients and the ceph-immutable-object-cache daemon. | |
84 | :Type: String | |
85 | :Required: No | |
86 | :Default: ``/var/run/ceph/immutable_object_cache_sock`` | |
87 | ||
88 | ||
89 | ``immutable_object_cache_path`` | |
90 | ||
91 | :Description: The immutable object cache data directory. | |
92 | :Type: String | |
93 | :Required: No | |
94 | :Default: ``/tmp/ceph_immutable_object_cache`` | |
95 | ||
96 | ||
97 | ``immutable_object_cache_max_size`` | |
98 | ||
99 | :Description: The max size for immutable cache. | |
100 | :Type: Size | |
101 | :Required: No | |
102 | :Default: ``1G`` | |
103 | ||
104 | ||
105 | ``immutable_object_cache_watermark`` | |
106 | ||
107 | :Description: The high-water mark for the cache. The value is between (0, 1). | |
108 | If the cache size reaches this threshold the daemon will start | |
109 | to delete cold cache based on LRU statistics. | |
110 | :Type: Float | |
111 | :Required: No | |
112 | :Default: ``0.9`` | |
113 | ||
114 | The ``ceph-immutable-object-cache`` daemon is available within the optional | |
115 | ``ceph-immutable-object-cache`` distribution package. | |
116 | ||
117 | .. important:: ``ceph-immutable-object-cache`` daemon requires the ability to | |
118 | connect RADOS clusters. | |
119 | ||
120 | Running the Immutable Object Cache Daemon | |
121 | ----------------------------------------- | |
122 | ||
123 | ``ceph-immutable-object-cache`` daemon should use a unique Ceph user ID. | |
124 | To `create a Ceph user`_, with ``ceph`` specify the ``auth get-or-create`` | |
125 | command, user name, monitor caps, and OSD caps:: | |
126 | ||
127 | ceph auth get-or-create client.ceph-immutable-object-cache.{unique id} mon 'allow r' osd 'profile rbd-read-only' | |
128 | ||
129 | The ``ceph-immutable-object-cache`` daemon can be managed by ``systemd`` by specifying the user | |
130 | ID as the daemon instance:: | |
131 | ||
132 | systemctl enable ceph-immutable-object-cache@ceph-immutable-object-cache.{unique id} | |
133 | ||
134 | The ``ceph-immutable-object-cache`` can also be run in foreground by ``ceph-immutable-object-cache`` command:: | |
135 | ||
136 | ceph-immutable-object-cache -f --log-file={log_path} | |
137 | ||
138 | QOS Settings | |
139 | ------------ | |
140 | ||
141 | The immutable object cache supports throttling, controlled by the following settings: | |
142 | ||
143 | ``immutable_object_cache_qos_schedule_tick_min`` | |
144 | ||
145 | :Description: Minimum schedule tick for immutable object cache. | |
146 | :Type: Milliseconds | |
147 | :Required: No | |
148 | :Default: ``50`` | |
149 | ||
150 | ||
151 | ``immutable_object_cache_qos_iops_limit`` | |
152 | ||
153 | :Description: The desired immutable object cache IO operations limit per second. | |
154 | :Type: Unsigned Integer | |
155 | :Required: No | |
156 | :Default: ``0`` | |
157 | ||
158 | ||
159 | ``immutable_object_cache_qos_iops_burst`` | |
160 | ||
161 | :Description: The desired burst limit of immutable object cache IO operations. | |
162 | :Type: Unsigned Integer | |
163 | :Required: No | |
164 | :Default: ``0`` | |
165 | ||
166 | ||
167 | ``immutable_object_cache_qos_iops_burst_seconds`` | |
168 | ||
169 | :Description: The desired burst duration in seconds of immutable object cache IO operations. | |
170 | :Type: Seconds | |
171 | :Required: No | |
172 | :Default: ``1`` | |
173 | ||
174 | ||
175 | ``immutable_object_cache_qos_bps_limit`` | |
176 | ||
177 | :Description: The desired immutable object cache IO bytes limit per second. | |
178 | :Type: Unsigned Integer | |
179 | :Required: No | |
180 | :Default: ``0`` | |
181 | ||
182 | ||
183 | ``immutable_object_cache_qos_bps_burst`` | |
184 | ||
185 | :Description: The desired burst limit of immutable object cache IO bytes. | |
186 | :Type: Unsigned Integer | |
187 | :Required: No | |
188 | :Default: ``0`` | |
189 | ||
190 | ||
191 | ``immutable_object_cache_qos_bps_burst_seconds`` | |
192 | ||
193 | :Description: The desired burst duration in seconds of immutable object cache IO bytes. | |
194 | :Type: Seconds | |
195 | :Required: No | |
196 | :Default: ``1`` | |
197 | ||
198 | .. _Cloned RBD Images: ../rbd-snapshot/#layering | |
199 | .. _section: ../../rados/configuration/ceph-conf/#configuration-sections | |
200 | .. _create a Ceph user: ../../rados/operations/user-management#add-a-user | |
201 |