]>
Commit | Line | Data |
---|---|---|
f67539c2 TL |
1 | =============================== |
2 | RBD Persistent Read-only Cache | |
3 | =============================== | |
4 | ||
5 | .. index:: Ceph Block Device; Persistent Read-only Cache | |
6 | ||
7 | Shared, Read-only Parent Image Cache | |
8 | ==================================== | |
9 | ||
10 | `Cloned RBD images`_ usually modify only a small fraction of the parent | |
11 | image. For example, in a VDI use-case, VMs are cloned from the same | |
12 | base image and initially differ only by hostname and IP address. During | |
13 | booting, all of these VMs read portions of the same parent | |
14 | image data. If we have a local cache of the parent | |
15 | image, this speeds up reads on the caching host. We also achieve | |
16 | reduction of client-to-cluster network traffic. | |
17 | RBD cache must be explicitly enabled in | |
18 | ``ceph.conf``. The ``ceph-immutable-object-cache`` daemon is responsible for | |
19 | caching the parent content on the local disk, and future reads on that data | |
20 | will be serviced from the local cache. | |
21 | ||
22 | .. note:: RBD shared read-only parent image cache requires the Ceph Nautilus release or later. | |
23 | ||
24 | .. ditaa:: | |
25 | ||
26 | +--------------------------------------------------------+ | |
27 | | QEMU | | |
28 | +--------------------------------------------------------+ | |
29 | | librbd (cloned images) | | |
30 | +-------------------+-+----------------------------------+ | |
31 | | librados | | ceph--immutable--object--cache | | |
32 | +-------------------+ +----------------------------------+ | |
33 | | OSDs/Mons | | local cached parent image | | |
34 | +-------------------+ +----------------------------------+ | |
35 | ||
36 | ||
37 | Enable RBD Shared Read-only Parent Image Cache | |
38 | ---------------------------------------------- | |
39 | ||
40 | To enable RBD shared read-only parent image cache, the following Ceph settings | |
41 | need to added in the ``[client]`` `section`_ of your ``ceph.conf`` file:: | |
42 | ||
43 | rbd parent cache enabled = true | |
44 | rbd plugins = parent_cache | |
45 | ||
46 | Immutable Object Cache Daemon | |
47 | ============================= | |
48 | ||
49 | Introduction and Generic Settings | |
50 | --------------------------------- | |
51 | ||
52 | The ``ceph-immutable-object-cache`` daemon is responsible for caching parent | |
53 | image content within its local caching directory. For better performance it's | |
54 | recommended to use SSDs as the underlying storage. | |
55 | ||
56 | The key components of the daemon are: | |
57 | ||
58 | #. **Domain socket based IPC:** The daemon will listen on a local domain | |
59 | socket on start up and wait for connections from librbd clients. | |
60 | ||
61 | #. **LRU based promotion/demotion policy:** The daemon will maintain | |
62 | in-memory statistics of cache-hits on each cache file. It will demote the | |
63 | cold cache if capacity reaches to the configured threshold. | |
64 | ||
65 | #. **File-based caching store:** The daemon will maintain a simple file | |
66 | based cache store. On promotion the RADOS objects will be fetched from | |
67 | RADOS cluster and stored in the local caching directory. | |
68 | ||
69 | On opening each cloned rbd image, ``librbd`` will try to connect to the | |
70 | cache daemon through its Unix domain socket. Once successfully connected, | |
71 | ``librbd`` will coordinate with the daemon on the subsequent reads. | |
72 | If there's a read that's not cached, the daemon will promote the RADOS object | |
73 | to local caching directory, so the next read on that object will be serviced | |
74 | from cache. The daemon also maintains simple LRU statistics so that under | |
75 | capacity pressure it will evict cold cache files as needed. | |
76 | ||
77 | Here are some important cache configuration settings: | |
78 | ||
79 | ``immutable_object_cache_sock`` | |
80 | ||
81 | :Description: The path to the domain socket used for communication between | |
82 | librbd clients and the ceph-immutable-object-cache daemon. | |
83 | :Type: String | |
84 | :Required: No | |
85 | :Default: ``/var/run/ceph/immutable_object_cache_sock`` | |
86 | ||
87 | ||
88 | ``immutable_object_cache_path`` | |
89 | ||
90 | :Description: The immutable object cache data directory. | |
91 | :Type: String | |
92 | :Required: No | |
93 | :Default: ``/tmp/ceph_immutable_object_cache`` | |
94 | ||
95 | ||
96 | ``immutable_object_cache_max_size`` | |
97 | ||
98 | :Description: The max size for immutable cache. | |
99 | :Type: Size | |
100 | :Required: No | |
101 | :Default: ``1G`` | |
102 | ||
103 | ||
104 | ``immutable_object_cache_watermark`` | |
105 | ||
106 | :Description: The high-water mark for the cache. The value is between (0, 1). | |
107 | If the cache size reaches this threshold the daemon will start | |
108 | to delete cold cache based on LRU statistics. | |
109 | :Type: Float | |
110 | :Required: No | |
111 | :Default: ``0.9`` | |
112 | ||
113 | The ``ceph-immutable-object-cache`` daemon is available within the optional | |
114 | ``ceph-immutable-object-cache`` distribution package. | |
115 | ||
116 | .. important:: ``ceph-immutable-object-cache`` daemon requires the ability to | |
117 | connect RADOS clusters. | |
118 | ||
119 | Running the Immutable Object Cache Daemon | |
120 | ----------------------------------------- | |
121 | ||
122 | ``ceph-immutable-object-cache`` daemon should use a unique Ceph user ID. | |
123 | To `create a Ceph user`_, with ``ceph`` specify the ``auth get-or-create`` | |
124 | command, user name, monitor caps, and OSD caps:: | |
125 | ||
126 | ceph auth get-or-create client.ceph-immutable-object-cache.{unique id} mon 'allow r' osd 'profile rbd-read-only' | |
127 | ||
128 | The ``ceph-immutable-object-cache`` daemon can be managed by ``systemd`` by specifying the user | |
129 | ID as the daemon instance:: | |
130 | ||
131 | systemctl enable ceph-immutable-object-cache@ceph-immutable-object-cache.{unique id} | |
132 | ||
133 | The ``ceph-immutable-object-cache`` can also be run in foreground by ``ceph-immutable-object-cache`` command:: | |
134 | ||
135 | ceph-immutable-object-cache -f --log-file={log_path} | |
136 | ||
137 | QOS Settings | |
138 | ------------ | |
139 | ||
140 | The immutable object cache supports throttling, controlled by the following settings: | |
141 | ||
142 | ``immutable_object_cache_qos_schedule_tick_min`` | |
143 | ||
144 | :Description: Minimum schedule tick for immutable object cache. | |
145 | :Type: Milliseconds | |
146 | :Required: No | |
147 | :Default: ``50`` | |
148 | ||
149 | ||
150 | ``immutable_object_cache_qos_iops_limit`` | |
151 | ||
152 | :Description: The desired immutable object cache IO operations limit per second. | |
153 | :Type: Unsigned Integer | |
154 | :Required: No | |
155 | :Default: ``0`` | |
156 | ||
157 | ||
158 | ``immutable_object_cache_qos_iops_burst`` | |
159 | ||
160 | :Description: The desired burst limit of immutable object cache IO operations. | |
161 | :Type: Unsigned Integer | |
162 | :Required: No | |
163 | :Default: ``0`` | |
164 | ||
165 | ||
166 | ``immutable_object_cache_qos_iops_burst_seconds`` | |
167 | ||
168 | :Description: The desired burst duration in seconds of immutable object cache IO operations. | |
169 | :Type: Seconds | |
170 | :Required: No | |
171 | :Default: ``1`` | |
172 | ||
173 | ||
174 | ``immutable_object_cache_qos_bps_limit`` | |
175 | ||
176 | :Description: The desired immutable object cache IO bytes limit per second. | |
177 | :Type: Unsigned Integer | |
178 | :Required: No | |
179 | :Default: ``0`` | |
180 | ||
181 | ||
182 | ``immutable_object_cache_qos_bps_burst`` | |
183 | ||
184 | :Description: The desired burst limit of immutable object cache IO bytes. | |
185 | :Type: Unsigned Integer | |
186 | :Required: No | |
187 | :Default: ``0`` | |
188 | ||
189 | ||
190 | ``immutable_object_cache_qos_bps_burst_seconds`` | |
191 | ||
192 | :Description: The desired burst duration in seconds of immutable object cache IO bytes. | |
193 | :Type: Seconds | |
194 | :Required: No | |
195 | :Default: ``1`` | |
196 | ||
197 | .. _Cloned RBD Images: ../rbd-snapshot/#layering | |
198 | .. _section: ../../rados/configuration/ceph-conf/#configuration-sections | |
199 | .. _create a Ceph user: ../../rados/operations/user-management#add-a-user | |
200 |