]>
Commit | Line | Data |
---|---|---|
494da23a TL |
1 | ====================== |
2 | Image Live-Migration | |
3 | ====================== | |
4 | ||
5 | .. index:: Ceph Block Device; live-migration | |
6 | ||
f67539c2 TL |
7 | RBD images can be live-migrated between different pools within the same cluster; |
8 | between different image formats and layouts; or from external data sources. | |
9 | When started, the source will be deep-copied to the destination image, pulling | |
10 | all snapshot history while preserving the sparse allocation of data where | |
11 | possible. | |
12 | ||
13 | By default, when live-migrating RBD images within the same Ceph cluster, the | |
14 | source image will be marked read-only and all clients will instead redirect | |
15 | IOs to the new target image. In addition, this mode can optionally preserve the | |
16 | link to the source image's parent to preserve sparseness, or it can flatten the | |
17 | image during the migration to remove the dependency on the source image's | |
18 | parent. | |
19 | ||
20 | The live-migration process can also be used in an import-only mode where the | |
21 | source image remains unmodified and the target image can be linked to an | |
22 | external data source such as a backing file, HTTP(s) file, or S3 object. | |
23 | ||
24 | The live-migration copy process can safely run in the background while the new | |
25 | target image is in use. There is currently a requirement to temporarily stop | |
26 | using the source image before preparing a migration when not using the | |
27 | import-only mode of operation. This helps to ensure that the client using the | |
28 | image is updated to point to the new target image. | |
494da23a TL |
29 | |
30 | .. note:: | |
f67539c2 TL |
31 | Image live-migration requires the Ceph Nautilus release or later. Support for |
32 | external data sources requires the Ceph Pacific release of later. The | |
33 | ``krbd`` kernel module does not support live-migration at this time. | |
494da23a TL |
34 | |
35 | ||
f91f0fd5 TL |
36 | .. ditaa:: |
37 | ||
38 | +-------------+ +-------------+ | |
494da23a TL |
39 | | {s} c999 | | {s} | |
40 | | Live | Target refers | Live | | |
41 | | migration |<-------------*| migration | | |
42 | | source | to Source | target | | |
43 | | | | | | |
44 | | (read only) | | (writable) | | |
45 | +-------------+ +-------------+ | |
46 | ||
47 | Source Target | |
48 | ||
49 | The live-migration process is comprised of three steps: | |
50 | ||
51 | #. **Prepare Migration:** The initial step creates the new target image and | |
f67539c2 TL |
52 | links the target image to the source. When not configured in the import-only |
53 | mode, the source image will also be linked to the target image and marked | |
54 | read-only. | |
55 | ||
56 | Similar to `layered images`_, attempts to read uninitialized data extents | |
57 | within the target image will internally redirect the read to the source | |
58 | image, and writes to uninitialized extents within the target will internally | |
59 | deep-copy the overlapping source image block to the target image. | |
494da23a TL |
60 | |
61 | ||
62 | #. **Execute Migration:** This is a background operation that deep-copies all | |
63 | initialized blocks from the source image to the target. This step can be | |
64 | run while clients are actively using the new target image. | |
65 | ||
66 | ||
67 | #. **Finish Migration:** Once the background migration process has completed, | |
68 | the migration can be committed or aborted. Committing the migration will | |
69 | remove the cross-links between the source and target images, and will | |
f67539c2 TL |
70 | remove the source image if not configured in the import-only mode. Aborting |
71 | the migration will remove the cross-links, and will remove the target image. | |
494da23a TL |
72 | |
73 | Prepare Migration | |
74 | ================= | |
75 | ||
f67539c2 TL |
76 | The default live-migration process for images within the same Ceph cluster is |
77 | initiated by running the `rbd migration prepare` command, providing the source | |
78 | and target images:: | |
494da23a TL |
79 | |
80 | $ rbd migration prepare migration_source [migration_target] | |
81 | ||
82 | The `rbd migration prepare` command accepts all the same layout optionals as the | |
83 | `rbd create` command, which allows changes to the immutable image on-disk | |
84 | layout. The `migration_target` can be skipped if the goal is only to change the | |
85 | on-disk layout, keeping the original image name. | |
86 | ||
87 | All clients using the source image must be stopped prior to preparing a | |
88 | live-migration. The prepare step will fail if it finds any running clients with | |
89 | the image open in read/write mode. Once the prepare step is complete, the | |
90 | clients can be restarted using the new target image name. Attempting to restart | |
91 | the clients using the source image name will result in failure. | |
92 | ||
93 | The `rbd status` command will show the current state of the live-migration:: | |
94 | ||
95 | $ rbd status migration_target | |
96 | Watchers: none | |
97 | Migration: | |
98 | source: rbd/migration_source (5e2cba2f62e) | |
99 | destination: rbd/migration_target (5e2ed95ed806) | |
100 | state: prepared | |
101 | ||
102 | Note that the source image will be moved to the RBD trash to avoid mistaken | |
103 | usage during the migration process:: | |
104 | ||
105 | $ rbd info migration_source | |
106 | rbd: error opening image migration_source: (2) No such file or directory | |
107 | $ rbd trash ls --all | |
108 | 5e2cba2f62e migration_source | |
109 | ||
110 | ||
f67539c2 TL |
111 | Prepare Import-Only Migration |
112 | ============================= | |
113 | ||
114 | The import-only live-migration process is initiated by running the same | |
115 | `rbd migration prepare` command, but adding the `--import-only` optional | |
116 | and providing a JSON-encoded ``source-spec`` to describe how to access | |
117 | the source image data. This ``source-spec`` can either be passed | |
118 | directly via the `--source-spec` optional, or via a file or STDIN via the | |
b3b6e05e | 119 | `--source-spec-path` optional:: |
f67539c2 TL |
120 | |
121 | $ rbd migration prepare --import-only --source-spec "<JSON>" migration_target | |
122 | ||
123 | The `rbd migration prepare` command accepts all the same layout optionals as the | |
124 | `rbd create` command. | |
125 | ||
126 | The `rbd status` command will show the current state of the live-migration:: | |
127 | ||
128 | $ rbd status migration_target | |
129 | Watchers: none | |
130 | Migration: | |
131 | source: {"stream":{"file_path":"/mnt/image.raw","type":"file"},"type":"raw"} | |
132 | destination: rbd/migration_target (ac69113dc1d7) | |
133 | state: prepared | |
134 | ||
135 | The general format for the ``source-spec`` JSON is as follows:: | |
136 | ||
137 | { | |
138 | "type": "<format-type>", | |
139 | <format unique parameters> | |
140 | "stream": { | |
141 | "type": "<stream-type>", | |
142 | <stream unique parameters> | |
143 | } | |
144 | } | |
145 | ||
146 | The following formats are currently supported: ``native``, ``qcow``, and | |
147 | ``raw``. The following streams are currently supported: ``file``, ``http``, and | |
148 | ``s3``. | |
149 | ||
150 | Formats | |
151 | ~~~~~~~ | |
152 | ||
153 | The ``native`` format can be used to describe a native RBD image within a | |
154 | Ceph cluster as the source image. Its ``source-spec`` JSON is encoded | |
155 | as follows:: | |
156 | ||
157 | { | |
158 | "type": "native", | |
159 | "pool_name": "<pool-name>", | |
160 | ["pool_id": <pool-id>,] (optional alternative to "pool_name") | |
161 | ["pool_namespace": "<pool-namespace",] (optional) | |
162 | "image_name": "<image-name>", | |
163 | ["image_id": "<image-id>",] (optional if image in trash) | |
164 | "snap_name": "<snap-name>", | |
165 | ["snap_id": "<snap-id>",] (optional alternative to "snap_name") | |
166 | } | |
167 | ||
168 | Note that the ``native`` format does not include the ``stream`` object since | |
169 | it utilizes native Ceph operations. For example, to import from the image | |
170 | ``rbd/ns1/image1@snap1``, the ``source-spec`` could be encoded as:: | |
171 | ||
172 | { | |
173 | "type": "native", | |
174 | "pool_name": "rbd", | |
175 | "pool_namespace": "ns1", | |
176 | "image_name": "image1", | |
177 | "snap_name": "snap1" | |
178 | } | |
179 | ||
180 | The ``qcow`` format can be used to describe a QCOW (QEMU copy-on-write) block | |
181 | device. Both the QCOW (v1) and QCOW2 formats are currently supported with the | |
182 | exception of advanced features such as compression, encryption, backing | |
183 | files, and external data files. Support for these missing features may be added | |
184 | in a future release. The ``qcow`` format data can be linked to any supported | |
185 | stream source described below. For example, its base ``source-spec`` JSON is | |
186 | encoded as follows:: | |
187 | ||
188 | { | |
189 | "type": "qcow", | |
190 | "stream": { | |
191 | <stream unique parameters> | |
192 | } | |
193 | } | |
194 | ||
195 | The ``raw`` format can be used to describe a thick-provisioned, raw block device | |
196 | export (i.e. `rbd export --export-format 1 <snap-spec>`). The ``raw`` format | |
197 | data can be linked to any supported stream source described below. For example, | |
198 | its base ``source-spec`` JSON is encoded as follows:: | |
199 | ||
200 | { | |
201 | "type": "raw", | |
202 | "stream": { | |
203 | <stream unique parameters for HEAD, non-snapshot revision> | |
204 | }, | |
205 | "snapshots": [ | |
206 | { | |
207 | "type": "raw", | |
208 | "name": "<snapshot-name>", | |
209 | "stream": { | |
210 | <stream unique parameters for snapshot> | |
211 | } | |
212 | }, | |
213 | ] (optional oldest to newest ordering of snapshots) | |
214 | } | |
215 | ||
216 | The inclusion of the ``snapshots`` array is optional and currently only supports | |
217 | thick-provisioned ``raw`` snapshot exports. | |
218 | ||
219 | Additional formats such as RBD export-format v2 and RBD export-diff | |
220 | snapshots will be added in a future release. | |
221 | ||
222 | Streams | |
223 | ~~~~~~~ | |
224 | ||
225 | The ``file`` stream can be used to import from a locally accessible POSIX file | |
226 | source. Its ``source-spec`` JSON is encoded as follows:: | |
227 | ||
228 | { | |
229 | <format unique parameters> | |
230 | "stream": { | |
231 | "type": "file", | |
232 | "file_path": "<file-path>" | |
233 | } | |
234 | } | |
235 | ||
236 | For example, to import a raw-format image from a file located at | |
237 | "/mnt/image.raw", its ``source-spec`` JSON is encoded as follows:: | |
238 | ||
239 | { | |
240 | "type": "raw", | |
241 | "stream": { | |
242 | "type": "file", | |
243 | "file_path": "/mnt/image.raw" | |
244 | } | |
245 | } | |
246 | ||
247 | The ``http`` stream can be used to import from a remote HTTP or HTTPS web | |
248 | server. Its ``source-spec`` JSON is encoded as follows:: | |
249 | ||
250 | { | |
251 | <format unique parameters> | |
252 | "stream": { | |
253 | "type": "http", | |
254 | "url": "<url-path>" | |
255 | } | |
256 | } | |
257 | ||
258 | For example, to import a raw-format image from a file located at | |
259 | ``http://download.ceph.com/image.raw``, its ``source-spec`` JSON is encoded | |
260 | as follows:: | |
261 | ||
262 | { | |
263 | "type": "raw", | |
264 | "stream": { | |
265 | "type": "http", | |
266 | "url": "http://download.ceph.com/image.raw" | |
267 | } | |
268 | } | |
269 | ||
270 | The ``s3`` stream can be used to import from a remote S3 bucket. Its | |
271 | ``source-spec`` JSON is encoded as follows:: | |
272 | ||
273 | { | |
274 | <format unique parameters> | |
275 | "stream": { | |
276 | "type": "s3", | |
277 | "url": "<url-path>", | |
278 | "access_key": "<access-key>", | |
279 | "secret_key": "<secret-key>" | |
280 | } | |
281 | } | |
282 | ||
283 | For example, to import a raw-format image from a file located at | |
284 | `http://s3.ceph.com/bucket/image.raw`, its ``source-spec`` JSON is encoded | |
285 | as follows:: | |
286 | ||
287 | { | |
288 | "type": "raw", | |
289 | "stream": { | |
290 | "type": "s3", | |
291 | "url": "http://s3.ceph.com/bucket/image.raw", | |
292 | "access_key": "NX5QOQKC6BH2IDN8HC7A", | |
293 | "secret_key": "LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn" | |
294 | } | |
295 | } | |
296 | ||
297 | .. note:: | |
298 | The ``access_key`` and ``secret_key`` parameters support storing the keys in | |
299 | the MON config-key store by prefixing the key values with ``config://`` | |
300 | followed by the path in the MON config-key store to the value. Values can be | |
301 | stored in the config-key store via ``ceph config-key set <key-path> <value>`` | |
302 | (e.g. ``ceph config-key set rbd/s3/access_key NX5QOQKC6BH2IDN8HC7A``). | |
303 | ||
494da23a TL |
304 | Execute Migration |
305 | ================= | |
306 | ||
307 | After preparing the live-migration, the image blocks from the source image | |
308 | must be copied to the target image. This is accomplished by running the | |
309 | `rbd migration execute` command:: | |
310 | ||
311 | $ rbd migration execute migration_target | |
312 | Image migration: 100% complete...done. | |
313 | ||
314 | The `rbd status` command will also provide feedback on the progress of the | |
315 | migration block deep-copy process:: | |
316 | ||
317 | $ rbd status migration_target | |
318 | Watchers: | |
319 | watcher=1.2.3.4:0/3695551461 client.123 cookie=123 | |
320 | Migration: | |
321 | source: rbd/migration_source (5e2cba2f62e) | |
322 | destination: rbd/migration_target (5e2ed95ed806) | |
323 | state: executing (32% complete) | |
324 | ||
325 | ||
326 | Commit Migration | |
327 | ================ | |
328 | ||
329 | Once the live-migration has completed deep-copying all data blocks from the | |
330 | source image to the target, the migration can be committed:: | |
331 | ||
332 | $ rbd status migration_target | |
333 | Watchers: none | |
334 | Migration: | |
335 | source: rbd/migration_source (5e2cba2f62e) | |
336 | destination: rbd/migration_target (5e2ed95ed806) | |
337 | state: executed | |
338 | $ rbd migration commit migration_target | |
339 | Commit image migration: 100% complete...done. | |
340 | ||
341 | If the `migration_source` image is a parent of one or more clones, the `--force` | |
342 | option will need to be specified after ensuring all descendent clone images are | |
343 | not in use. | |
344 | ||
9f95a23c | 345 | Committing the live-migration will remove the cross-links between the source |
494da23a TL |
346 | and target images, and will remove the source image:: |
347 | ||
348 | $ rbd trash list --all | |
349 | ||
350 | ||
351 | Abort Migration | |
352 | =============== | |
353 | ||
354 | If you wish to revert the prepare or execute step, run the `rbd migration abort` | |
355 | command to revert the migration process:: | |
356 | ||
357 | $ rbd migration abort migration_target | |
358 | Abort image migration: 100% complete...done. | |
359 | ||
360 | Aborting the migration will result in the target image being deleted and access | |
361 | to the original source image being restored:: | |
362 | ||
363 | $ rbd ls | |
364 | migration_source | |
365 | ||
366 | ||
367 | .. _layered images: ../rbd-snapshot/#layering |