5 Dumpling is the 4th stable release of Ceph. It is named after the
6 dumpling squid (Euprymna tasmanica).
8 v0.67.12 "Dumpling" (draft)
9 ===========================
11 This stable update for Dumpling fixes a few longstanding issues with
12 backfill in the OSD that can lead to stalled IOs. There is also a fix
13 for memory utilization for reads in librbd when caching is enabled,
14 and then several other small fixes across the rest of the system.
16 Dumpling users who have encountered IO stalls during backfill and who
17 do not expect to upgrade to Firefly soon should upgrade. Everyone
18 else should upgrade to Firefly already. This is likely to be the last stable
19 release for the 0.67.x Dumpling series.
25 * buffer: fix buffer rebuild alignment corner case (#6614 #6003 Loic Dachary, Samuel Just)
26 * ceph-disk: reprobe partitions after zap (#9665 #9721 Loic Dachary)
27 * ceph-disk: use partx instead of partprobe when appropriate (Loic Dachary)
28 * common: add $cctid meta variable (#6228 Adam Crume)
29 * crush: fix get_full_location_ordered (Sage Weil)
30 * crush: pick ruleset id that matches rule_id (#9675 Xiaoxi Chen)
31 * libcephfs: fix tid wrap bug (#9869 Greg Farnum)
32 * libcephfs: get osd location on -1 should return EINVAL (Sage Weil)
33 * librados: fix race condition with C API and op timeouts (#9582 Sage Weil)
34 * librbd: constrain max number of in-flight read requests (#9854 Jason Dillaman)
35 * librbd: enforce cache size on read requests (Jason Dillaman)
36 * librbd: fix invalid close in image open failure path (#10030 Jason Dillaman)
37 * librbd: fix read hang on sparse files (Jason Dillaman)
38 * librbd: gracefully handle deleted/renamed pools (#10270 #10122 Jason Dillaman)
39 * librbd: protect list_children from invalid child pool ioctxs (#10123 Jason Dillaman)
40 * mds: fix ctime updates from clients without dirty caps (#9514 Greg Farnum)
41 * mds: fix rare NULL dereference in cap update path (Greg Farnum)
42 * mds: fix assertion caused by system clock backwards (#11053 Yan, Zheng)
43 * mds: store backtrace on straydir (Yan, Zheng)
44 * osd: fix journal committed_thru update after replay (#6756 Samuel Just)
45 * osd: fix memory leak, busy loop on snap trim (#9113 Samuel Just)
46 * osd: fix misc peering, recovery bugs (#10168 Samuel Just)
47 * osd: fix purged_snap field on backfill start (#9487 Sage Weil, Samuel Just)
48 * osd: handle no-op write with snapshot corner case (#10262 Sage Weil, Loic Dachary)
49 * osd: respect RWORDERED rados flag (Sage Weil)
50 * osd: several backfill fixes and refactors (Samuel Just, David Zafman)
51 * rgw: send http status reason explicitly in fastcgi (Yehuda Sadeh)
56 This stable update for Dumpling fixes several important bugs that
57 affect a small set of users.
59 We recommend that all Dumpling users upgrade at their convenience. If
60 none of these issues are affecting your deployment there is no
67 * common: fix sending dup cluster log items (#9080 Sage Weil)
68 * doc: several doc updates (Alfredo Deza)
69 * libcephfs-java: fix build against older JNI headesr (Greg Farnum)
70 * librados: fix crash in op timeout path (#9362 Matthias Kiefer, Sage Weil)
71 * librbd: fix crash using clone of flattened image (#8845 Josh Durgin)
72 * librbd: fix error path cleanup when failing to open image (#8912 Josh Durgin)
73 * mon: fix crash when adjusting pg_num before any OSDs are added (#9052 Sage Weil)
74 * mon: reduce log noise from paxos (Aanchal Agrawal, Sage Weil)
75 * osd: allow scrub and snap trim thread pool IO priority to be adjusted (Sage Weil)
76 * osd: fix mount/remount sync race (#9144 Sage Weil)
82 This stable update release for Dumpling includes primarily fixes for
83 RGW, including several issues with bucket listings and a potential
84 data corruption problem when multiple multi-part uploads race. There is also
85 some throttling capability added in the OSD for scrub that can mitigate the
86 performance impact on production clusters.
88 We recommend that all Dumpling users upgrade at their convenience.
93 * ceph-disk: partprobe befoere settle, fixing dm-crypt (#6966, Eric Eastman)
94 * librbd: add invalidate cache interface (Josh Durgin)
95 * librbd: close image if remove_child fails (Ilya Dryomov)
96 * librbd: fix potential null pointer dereference (Danny Al-Gaaf)
97 * librbd: improve writeback checks, performance (Haomai Wang)
98 * librbd: skip zeroes when copying image (#6257, Josh Durgin)
99 * mon: fix rule(set) check on 'ceph pool set ... crush_ruleset ...' (#8599, John Spray)
100 * mon: shut down if mon is removed from cluster (#6789, Joao Eduardo Luis)
101 * osd: fix filestore perf reports to mon (Sage Weil)
102 * osd: force any new or updated xattr into leveldb if E2BIG from XFS (#7779, Sage Weil)
103 * osd: lock snapdir object during write to fix race with backfill (Samuel Just)
104 * osd: option sleep during scrub (Sage Weil)
105 * osd: set io priority on scrub and snap trim threads (Sage Weil)
106 * osd: 'status' admin socket command (Sage Weil)
107 * rbd: tolerate missing NULL terminator on block_name_prefix (#7577, Dan Mick)
108 * rgw: calculate user manifest (#8169, Yehuda Sadeh)
109 * rgw: fix abort on chunk read error, avoid using extra memory (#8289, Yehuda Sadeh)
110 * rgw: fix buffer overflow on bucket instance id (#8608, Yehuda Sadeh)
111 * rgw: fix crash in swift CORS preflight request (#8586, Yehuda Sadeh)
112 * rgw: fix implicit removal of old objects on object creation (#8972, Patrycja Szablowska, Yehuda Sadeh)
113 * rgw: fix MaxKeys in bucket listing (Yehuda Sadeh)
114 * rgw: fix race with multiple updates to a single multipart object (#8269, Yehuda Sadeh)
115 * rgw: improve bucket listing with delimiter (Yehuda Sadeh)
116 * rgw: include NextMarker in bucket listing (#8858, Yehuda Sadeh)
117 * rgw: return error early on non-existent bucket (#7064, Yehuda Sadeh)
118 * rgw: set truncation flag correctly in bucket listing (Yehuda Sadeh)
119 * sysvinit: continue starting daemons after pre-mount error (#8554, Sage Weil)
121 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.10.txt>`.
127 This Dumpling point release fixes several minor bugs. The most
128 prevalent in the field is one that occasionally prevents OSDs from
129 starting on recently created clusters.
131 We recommend that all Dumpling users upgrade at their convenience.
136 * ceph-fuse, libcephfs: client admin socket command to kick and inspect MDS sessions (#8021, Zheng Yan)
137 * monclient: fix failure detection during mon handshake (#8278, Sage Weil)
138 * mon: set tid on no-op PGStatsAck messages (#8280, Sage Weil)
139 * msgr: fix a rare bug with connection negotiation between OSDs (Guang Yang)
140 * osd: allow snap trim throttling with simple delay (#6278, Sage Weil)
141 * osd: check for splitting when processing recover/backfill reservations (#6565, Samuel Just)
142 * osd: fix backfill position tracking (#8162, Samuel Just)
143 * osd: fix bug in backfill stats (Samuel Just)
144 * osd: fix bug preventing OSD startup for infant clusters (#8162, Greg Farnum)
145 * osd: fix rare PG resurrection race causing an incomplete PG (#7740, Samuel Just)
146 * osd: only complete replicas count toward min_size (#7805, Samuel Just)
147 * rgw: allow setting ACLs with empty owner (#6892, Yehuda Sadeh)
148 * rgw: send user manifest header field (#8170, Yehuda Sadeh)
150 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.9.txt>`.
156 This Dumpling point release fixes several non-critical issues since
157 v0.67.7. The most notable bug fixes are an auth fix in librbd
158 (observed as an occasional crash from KVM), an improvement in the
159 network failure detection with the monitor, and several hard to hit
160 OSD crashes or hangs.
162 We recommend that all users upgrade at their convenience.
167 * The 'rbd ls' function now returns success and returns an empty when a pool
168 does not store any rbd images. Previously it would return an ENOENT error.
170 * Ceph will now issue a health warning if the 'mon osd down out
171 interval' config option is set to zero. This warning can be
172 disabled by adding 'mon warn on osd down out interval zero = false'
178 * all: improve keepalive detection of failed monitor connections (#7888, Sage Weil)
179 * ceph-fuse, libcephfs: pin inodes during readahead, fixing rare crash (#7867, Sage Weil)
180 * librbd: make cache writeback a bit less aggressive (Sage Weil)
181 * librbd: make symlink for qemu to detect librbd in RPM (#7293, Josh Durgin)
182 * mon: allow 'hashpspool' pool flag to be set and unset (Loic Dachary)
183 * mon: commit paxos state only after entire quorum acks, fixing rare race where prior round state is readable (#7736, Sage Weil)
184 * mon: make elections and timeouts a bit more robust (#7212, Sage Weil)
185 * mon: prevent extreme pool split operations (Greg Farnum)
186 * mon: wait for quorum for get_version requests to close rare pool creation race (#7997, Sage Weil)
187 * mon: warn on 'mon osd down out interval = 0' (#7784, Joao Luis)
188 * msgr: fix byte-order for auth challenge, fixing auth errors on big-endian clients (#7977, Dan Mick)
189 * msgr: fix occasional crash in authentication code (usually triggered by librbd) (#6840, Josh Durgin)
190 * msgr: fix rebind() race (#6992, Xihui He)
191 * osd: avoid timeouts during slow PG deletion (#6528, Samuel Just)
192 * osd: fix bug in pool listing during recovery (#6633, Samuel Just)
193 * osd: fix queue limits, fixing recovery stalls (#7706, Samuel Just)
194 * osd: fix rare peering crashes (#6722, #6910, Samuel Just)
195 * osd: fix rare recovery hang (#6681, Samuel Just)
196 * osd: improve error handling on journal errors (#7738, Sage Weil)
197 * osd: reduce load on the monitor from OSDMap subscriptions (Greg Farnum)
198 * osd: rery GetLog on peer osd startup, fixing some rare peering stalls (#6909, Samuel Just)
199 * osd: reset journal state on remount to fix occasional crash on OSD startup (#8019, Sage Weil)
200 * osd: share maps with peers more aggressively (Greg Farnum)
201 * rbd: make it harder to delete an rbd image that is currently in use (#7076, Ilya Drymov)
202 * rgw: deny writes to secondary zone by non-system users (#6678, Yehuda Sadeh)
203 * rgw: do'nt log system requests in usage log (#6889, Yehuda Sadeh)
204 * rgw: fix bucket recreation (#6951, Yehuda Sadeh)
205 * rgw: fix Swift range response (#7099, Julien Calvet, Yehuda Sadeh)
206 * rgw: fix URL escaping (#8202, Yehuda Sadeh)
207 * rgw: fix whitespace trimming in http headers (#7543, Yehuda Sadeh)
208 * rgw: make multi-object deletion idempotent (#7346, Yehuda Sadeh)
210 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.8.txt>`.
215 This Dumpling point release fixes a few critical issues in v0.67.6.
217 All v0.67.6 users are urgently encouraged to upgrade. We also
218 recommend that all v0.67.5 (or older) users upgrade.
223 * Once you have upgraded a radosgw instance or OSD to v0.67.7, you should not
224 downgrade to a previous version.
229 * ceph-disk: additional unit tests
230 * librbd: revert caching behavior change in v0.67.6
231 * osd: fix problem reading xattrs due to incomplete backport in v0.67.6
232 * radosgw-admin: fix reading object policy
234 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.7.txt>`.
240 .. note: This release contains a librbd bug that is fixed in v0.67.7. Please upgrade to v0.67.7 and do not use v0.67.6.
242 This Dumpling point release contains a number of important fixed for
243 the OSD, monitor, and radosgw. Most significantly, a change that
244 forces large object attributes to spill over into leveldb has been
245 backported that can prevent objects and the cluster from being damaged
246 by large attributes (which can be induced via the radosgw). There is
247 also a set of fixes that improves data safety and RADOS semantics when
248 the cluster becomes full and then non-full.
250 We recommend that all 0.67.x Dumpling users skip this release and upgrade to v0.67.7.
255 * The OSD has long contained a feature that allows large xattrs to
256 spill over into the leveldb backing store in situations where not
257 all local file systems are able to store them reliably. This option
258 is now enabled unconditionally in order to avoid rare cases where
259 storing large xattrs renders the object unreadable. This is known to
260 be triggered by very large multipart objects, but could be caused by
261 other workloads as well. Although there is some small risk that
262 performance for certain workloads will degrade, it is more important
263 that data be retrievable. Note that newer versions of Ceph (e.g.,
264 firefly) do some additional work to avoid the potential performance
265 regression in this case, but that is current considered too complex
266 for backport to the Dumpling stable series.
268 * It is very dangerous to downgrade from v0.67.6 to a prior version of
269 Dumpling. If the old version does not have 'filestore xattr use
270 omap = true' it may not be able to read all xattrs for an object and
271 can cause undefined behavior.
276 * ceph-disk: misc bug fixes, particularly on RHEL (Loic Dachary, Alfredo Deza, various)
277 * ceph-fuse, libcephfs: fix crash from read over certain sparseness patterns (Sage Weil)
278 * ceph-fuse, libcephfs: fix integer overflow for sync reads racing with appends (Sage Weil)
279 * ceph.spec: fix udev rule when building RPM under RHEL (Derek Yarnell)
280 * common: fix crash from bad format from admin socket (Loic Dachary)
281 * librados: add optional timeouts (Josh Durgin)
282 * librados: do not leak budget when resending localized or redirected ops (Josh Durgin)
283 * librados, osd: fix and improve full cluster handling (Josh Durgin)
284 * librbd: fix use-after-free when updating perfcounters during image close (Josh Durgin)
285 * librbd: remove limit on objects in cache (Josh Durgin)
286 * mon: avoid on-disk full OSDMap corruption from pg_temp removal (Sage Weil)
287 * mon: avoid stray pg_temp entries from pool deletion race (Joao Eduardo Luis)
288 * mon: do not generate spurious MDSMaps from laggy daemons (Joao Eduardo Luis)
289 * mon: fix error code from 'osd rm|down|out|in ...' commands (Loic Dachary)
290 * mon: include all health items in summary output (John Spray)
291 * osd: fix occasional race/crash during startup (Sage Weil)
292 * osd: ignore stray OSDMap messages during init (Sage Weil)
293 * osd: unconditionally let xattrs overflow into leveldb (David Zafman)
294 * rados: fix a few error checks for the CLI (Josh Durgin)
295 * rgw: convert legacy bucket info objects on demand (Yehuda Sadeh)
296 * rgw: fix bug causing system users to lose privileges (Yehuda Sadeh)
297 * rgw: fix CORS bugs related to headers and case sensitivity (Robin H. Johnson)
298 * rgw: fix multipart object listing (Yehuda Sadeh)
299 * rgw: fix racing object creations (Yehuda Sadeh)
300 * rgw: fix racing object put and delete (Yehuda Sadeh)
301 * rgw: fix S3 auth when using response-* query string params (Sylvain Munaut)
302 * rgw: use correct secret key for POST authentication (Robin H. Johnson)
304 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.6.txt>`.
310 This release includes a few critical bug fixes for the radosgw,
311 including a fix for hanging operations on large objects. There are also
312 several bug fixes for radosgw multi-site replications, and a few
313 backported features. Also, notably, the 'osd perf' command (which dumps
314 recent performance information about active OSDs) has been backported.
316 We recommend that all 0.67.x Dumpling users upgrade.
321 * ceph-fuse: fix crash in caching code
322 * mds: fix looping in populate_mydir()
323 * mds: fix standby-replay race
324 * mon: accept 'osd pool set ...' as string
325 * mon: backport: 'osd perf' command to dump recent OSD performance stats
326 * osd: add feature compat check for upcoming object sharding
327 * osd: fix osd bench block size argument
328 * rbd.py: increase parent name size limit
329 * rgw: backport: allow wildcard in supported keystone roles
330 * rgw: backport: improve swift COPY behavior
331 * rgw: backport: log and open admin socket by default
332 * rgw: backport: validate S3 tokens against keystone
333 * rgw: fix bucket removal
334 * rgw: fix client error code for chunked PUT failure
335 * rgw: fix hang on large object GET
336 * rgw: fix rare use-after-free
337 * rgw: various DR bug fixes
338 * sysvinit, upstart: prevent starting daemons using both init systems
340 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.5.txt>`.
346 This point release fixes an important performance issue with radosgw,
347 keystone authentication token caching, and CORS. All users
348 (especially those of rgw) are encouraged to upgrade.
353 * crush: fix invalidation of cached names
354 * crushtool: do not crash on non-unique bucket ids
355 * mds: be more careful when decoding LogEvents
356 * mds: fix heap check debugging commands
357 * mon: avoid rebuilding old full osdmaps
358 * mon: fix 'ceph crush move ...'
359 * mon: fix 'ceph osd crush reweight ...'
360 * mon: fix writeout of full osdmaps during trim
361 * mon: limit size of transactions
362 * mon: prevent both unmanaged and pool snaps
363 * osd: disable xattr size limit (prevents upload of large rgw objects)
364 * osd: fix recovery op throttling
365 * osd: fix throttling of log messages for very slow requests
366 * rgw: drain pending requests before completing write
368 * rgw: fix inefficient list::size() usage
369 * rgw: fix keystone token expiration
370 * rgw: fix minor memory leaks
371 * rgw: fix null termination of buffer
373 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.4.txt>`.
379 This point release fixes a few important performance regressions with
380 the OSD (both with CPU and disk utilization), as well as several other
381 important but less common problems. We recommend that all production users
387 * ceph-disk: partprobe after creation journal partition
388 * ceph-disk: specify fs type when mounting
389 * ceph-post-file: new utility to help share logs and other files with ceph developers
390 * libcephfs: fix truncate vs readahead race (crash)
391 * mds: fix flock/fcntl lock deadlock
392 * mds: fix rejoin loop when encountering pre-dumpling backpointers
393 * mon: allow name and addr discovery during election stage
394 * mon: always refresh after Paxos store_state (fixes recovery corner case)
395 * mon: fix off-by-4x bug with osd byte counts
396 * osd: add and disable 'pg log keys debug' by default
397 * osd: add option to disable throttling
398 * osd: avoid leveldb iterators for pg log append and trim
399 * osd: fix readdir_r invocations
400 * osd: use fdatasync instead of sync
401 * radosgw: fix sysvinit script return status
402 * rbd: relicense as LGPL2
403 * rgw: flush pending data on multipart upload
404 * rgw: recheck object name during S3 POST
405 * rgw: reorder init/startup
406 * rpm: fix debuginfo package build
408 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.3.txt>`.
414 This is an important point release for Dumpling. Most notably, it
415 fixes a problem when upgrading directly from v0.56.x Bobtail to
416 v0.67.x Dumpling (without stopping at v0.61.x Cuttlefish along the
417 way). It also fixes a problem with the CLI parsing of the CEPH_ARGS
418 environment variable, high CPU utilization by the ceph-osd daemons,
419 and cleans up the radosgw shutdown sequence.
424 * objecter: resend linger requests when cluster goes from full to non-full
425 * ceph: parse CEPH_ARGS environment variable
426 * librados: fix small memory leak
427 * osd: remove old log objects on upgrade (fixes bobtail -> dumpling jump)
428 * osd: disable PGLog::check() via config option (fixes CPU burn)
429 * rgw: drain requests on shutdown
430 * rgw: misc memory leaks on shutdown
432 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.2.txt>`.
438 This is a minor point release for Dumpling that fixes problems with
439 OpenStack and librbd hangs when caching is disabled.
444 * librados, librbd: fix constructor for python bindings with certain
445 usages (in particular, that used by OpenStack)
446 * librados, librbd: fix aio_flush wakeup when cache is disabled
447 * librados: fix locking for aio completion refcounting
448 * fixes 'ceph --admin-daemon ...' command error code on error
449 * fixes 'ceph daemon ... config set ...' command for boolean config
452 For more detailed information, see :download:`the complete changelog <../changelog/v0.67.1.txt>`.
457 This is the fourth major release of Ceph, code-named "Dumpling." The
458 headline features for this release include:
460 * Multi-site support for radosgw. This includes the ability to set up
461 separate "regions" in the same or different Ceph clusters that share
462 a single S3/Swift bucket/container namespace.
464 * RESTful API endpoint for Ceph cluster administration.
465 ceph-rest-api, a wrapper around ceph_rest_api.py, can be used to
466 start up a test single-threaded HTTP server that provides access to
467 cluster information and administration in very similar ways to the
468 ceph commandline tool. ceph_rest_api.py can be used as a WSGI
469 application for deployment in a more-capable web server. See
470 ceph-rest-api.8 for more.
472 * Object namespaces in librados.
475 .. _dumpling-upgrade:
480 It is possible to do a rolling upgrade from Cuttlefish to Dumpling.
482 #. Upgrade ceph-common on all nodes that will use the command line
484 #. Upgrade all monitors (upgrade ceph package, restart ceph-mon
485 daemons). This can happen one daemon or host at a time. Note that
486 because cuttlefish and dumpling monitors can't talk to each other,
487 all monitors should be upgraded in relatively short succession to
488 minimize the risk that an a untimely failure will reduce
490 #. Upgrade all osds (upgrade ceph package, restart ceph-osd daemons).
491 This can happen one daemon or host at a time.
492 #. Upgrade radosgw (upgrade radosgw package, restart radosgw daemons).
498 * There is monitor internal protocol change, which means that v0.67
499 ceph-mon daemons cannot talk to v0.66 or older daemons. We
500 recommend upgrading all monitors at once (or in relatively quick
501 succession) to minimize the possibility of downtime.
503 * The output of 'ceph status --format=json' or 'ceph -s --format=json'
504 has changed to return status information in a more structured and
507 * The 'ceph pg dump_stuck [threshold]' command used to require a
508 --threshold or -t prefix to the threshold argument, but now does
511 * Many more ceph commands now output formatted information; select
512 with '--format=<format>', where <format> can be 'json', 'json-pretty',
513 'xml', or 'xml-pretty'.
515 * The 'ceph pg <pgid> ...' commands (like 'ceph pg <pgid> query') are
516 deprecated in favor of 'ceph tell <pgid> ...'. This makes the
517 distinction between 'ceph pg <command> <pgid>' and 'ceph pg <pgid>
518 <command>' less awkward by making it clearer that the 'tell'
519 commands are talking to the OSD serving the placement group, not the
522 * The 'ceph --admin-daemon <path> <command ...>' used to accept the
523 command and arguments as either a single string or as separate
524 arguments. It will now only accept the command spread across
525 multiple arguments. This means that any script which does something
528 ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok 'config set debug_ms 1'
530 needs to remove the quotes. Also, note that the above can now be
533 ceph daemon osd.0 config set debug_ms 1
535 * The radosgw caps were inconsistently documented to be either 'mon =
536 allow r' or 'mon = allow rw'. The 'mon = allow rw' is required for
537 radosgw to create its own pools. All documentation has been updated
540 * The radosgw copy object operation may return extra progress info
541 during the operation. At this point it will only happen when doing
542 cross zone copy operations. The S3 response will now return extra
543 <Progress> field under the <CopyResult> container. The Swift
544 response will now send the progress as a json array.
546 * In v0.66 and v0.65 the HASHPSPOOL pool flag was enabled by default
547 for new pools, but has been disabled again until Linux kernel client
548 support reaches more distributions and users.
550 * ceph-osd now requires a max file descriptor limit (e.g., ``ulimit -n
552 filestore_wbthrottle_(xfs|btrfs)_inodes_hard_limit (5000 by default)
553 in order to accommodate the new write back throttle system. On
554 Ubuntu, upstart now sets the fd limit to 32k. On other platforms,
555 the sysvinit script will set it to 32k by default (still
556 overrideable via max_open_files). If this field has been customized
557 in ceph.conf it should likely be adjusted upwards.
559 Upgrading from v0.61 "Cuttlefish"
560 ---------------------------------
562 In addition to the above notes about upgrading from v0.66:
564 * There has been a huge revamp of the 'ceph' command-line interface
565 implementation. The ``ceph-common`` client library needs to be
566 upgrade before ``ceph-mon`` is restarted in order to avoid problems
567 using the CLI (the old ``ceph`` client utility cannot talk to the
570 * The CLI is now very careful about sending the 'status' one-liner
571 output to stderr and command output to stdout. Scripts relying on
572 output should take care.
574 * The 'ceph osd tell ...' and 'ceph mon tell ...' commands are no
575 longer supported. Any callers should use::
577 ceph tell osd.<id or *> ...
578 ceph tell mon.<id or name or *> ...
580 The 'ceph mds tell ...' command is still there, but will soon also
581 transition to 'ceph tell mds.<id or name or \*> ...'
583 * The 'ceph osd crush add ...' command used to take one of two forms::
585 ceph osd crush add 123 osd.123 <weight> <location ...>
586 ceph osd crush add osd.123 <weight> <location ...>
588 This is because the id and crush name are redundant. Now only the
589 simple form is supported, where the osd name/id can either be a bare
590 id (integer) or name (osd.<id>)::
592 ceph osd crush add osd.123 <weight> <location ...>
593 ceph osd crush add 123 <weight> <location ...>
595 * There is now a maximum RADOS object size, configurable via 'osd max
596 object size', defaulting to 100 GB. Note that this has no effect on
597 RBD, CephFS, or radosgw, which all stripe over objects. If you are
598 using librados and storing objects larger than that, you will need
599 to adjust 'osd max object size', and should consider using smaller
602 * The 'osd min down {reporters|reports}' config options have been
603 renamed to 'mon osd min down {reporters|reports}', and the
604 documentation has been updated to reflect that these options apply
605 to the monitors (who process failure reports) and not OSDs. If you
606 have adjusted these settings, please update your ``ceph.conf``
610 Notable changes since v0.66
611 ---------------------------
613 * mon: sync improvements (performance and robustness)
614 * mon: many bug fixes (paxos and services)
615 * mon: fixed bugs in recovery and io rate reporting (negative/large values)
616 * mon: collect metadata on osd performance
617 * mon: generate health warnings from slow or stuck requests
618 * mon: expanded --format=<json|xml|...> support for monitor commands
619 * mon: scrub function for verifying data integrity
620 * mon, osd: fix old osdmap trimming logic
621 * mon: enable leveldb caching by default
622 * mon: more efficient storage of PG metadata
623 * ceph-rest-api: RESTful endpoint for administer cluster (mirrors CLI)
624 * rgw: multi-region support
625 * rgw: infrastructure to support georeplication of bucket and user metadata
626 * rgw: infrastructure to support georeplication of bucket data
627 * rgw: COPY object support between regions
628 * rbd: /etc/ceph/rbdmap file for mapping rbd images on startup
629 * osd: many bug fixes
630 * osd: limit number of incremental osdmaps sent to peers (could cause osds to be wrongly marked down)
631 * osd: more efficient small object recovery
632 * osd, librados: support for object namespaces
633 * osd: automatically enable xattrs on leveldb as necessary
634 * mds: fix bug in LOOKUPINO (used by nfs reexport)
635 * mds: fix O_TRUNC locking
636 * msgr: fixed race condition in inter-osd network communication
637 * msgr: fixed various memory leaks related to network sessions
638 * ceph-disk: fixes for unusual device names, partition detection
639 * hypertable: fixes for hypertable CephBroker bindings
640 * use SSE4.2 crc32c instruction if present
643 Notable changes since v0.61 "Cuttlefish"
644 ----------------------------------------
646 * add 'config get' admin socket command
647 * ceph-conf: --show-config-value now reflects daemon defaults
648 * ceph-disk: add '[un]suppress-active DEV' command
649 * ceph-disk: avoid mounting over an existing osd in /var/lib/ceph/osd/*
650 * ceph-disk: fixes for unusual device names, partition detection
651 * ceph-disk: improved handling of odd device names
652 * ceph-disk: many fixes for RHEL/CentOS, Fedora, wheezy
653 * ceph-disk: simpler, more robust locking
654 * ceph-fuse, libcephfs: fix a few caps revocation bugs
655 * ceph-fuse, libcephfs: fix read zeroing at EOF
656 * ceph-fuse, libcephfs: fix request refcounting bug (hang on shutdown)
657 * ceph-fuse, libcephfs: fix truncatation bug on >4MB files (Yan, Zheng)
658 * ceph-fuse, libcephfs: fix for cap release/hang
659 * ceph-fuse: add ioctl support
660 * ceph-fuse: fixed long-standing O_NOATIME vs O_LAZY bug
661 * ceph-rest-api: RESTful endpoint for administer cluster (mirrors CLI)
662 * ceph, librados: fix resending of commands on mon reconnect
663 * daemons: create /var/run/ceph as needed
664 * debian wheezy: fix udev rules
665 * debian, specfile: packaging cleanups
666 * debian: fix upstart behavior with upgrades
667 * debian: rgw: stop daemon on uninstall
668 * debian: stop daemons on uninstall; fix dependencies
669 * hypertable: fixes for hypertable CephBroker bindings
670 * librados python binding cleanups
671 * librados python: fix xattrs > 4KB (Josh Durgin)
672 * librados: configurable max object size (default 100 GB)
673 * librados: new calls to administer the cluster
674 * librbd: ability to read from local replicas
675 * librbd: locking tests (Josh Durgin)
676 * librbd: make default options/features for newly created images (e.g., via qemu-img) configurable
677 * librbd: parallelize delete, rollback, flatten, copy, resize
678 * many many fixes from static code analysis (Danny Al-Gaaf)
679 * mds: fix O_TRUNC locking
680 * mds: fix bug in LOOKUPINO (used by nfs reexport)
681 * mds: fix rare hang after client restart
682 * mds: fix several bugs (Yan, Zheng)
683 * mds: many backpointer improvements (Yan, Zheng)
684 * mds: many fixes for mds clustering
685 * mds: misc stability fixes (Yan, Zheng, Greg Farnum)
686 * mds: new robust open-by-ino support (Yan, Zheng)
687 * mds: support robust lookup by ino number (good for NFS) (Yan, Zheng)
688 * mon, ceph: huge revamp of CLI and internal admin API. (Dan Mick)
689 * mon, osd: fix old osdmap trimming logic
690 * mon, osd: many memory leaks fixed
691 * mon: better trim/compaction behavior
692 * mon: collect metadata on osd performance
693 * mon: enable leveldb caching by default
694 * mon: expanded --format=<json|xml|...> support for monitor commands
695 * mon: fix election timeout
696 * mon: fix leveldb compression, trimming
697 * mon: fix start fork behavior
698 * mon: fix units in 'ceph df' output
699 * mon: fix validation of mds ids from CLI commands
700 * mon: fixed bugs in recovery and io rate reporting (negative/large values)
701 * mon: generate health warnings from slow or stuck requests
702 * mon: many bug fixes (paxos and services, sync)
703 * mon: many stability fixes (Joao Luis)
704 * mon: more efficient storage of PG metadata
705 * mon: new --extract-monmap to aid disaster recovery
706 * mon: new capability syntax
707 * mon: scrub function for verifying data integrity
708 * mon: simplify PaxosService vs Paxos interaction, fix readable/writeable checks
709 * mon: sync improvements (performance and robustness)
710 * mon: tuning, performance improvements
711 * msgr: fix various memory leaks
712 * msgr: fixed race condition in inter-osd network communication
713 * msgr: fixed various memory leaks related to network sessions
714 * osd, librados: support for object namespaces
715 * osd, mon: optionally dump leveldb transactions to a log
716 * osd: automatically enable xattrs on leveldb as necessary
717 * osd: avoid osd flapping from asymmetric network failure
718 * osd: break blacklisted client watches (David Zafman)
719 * osd: close narrow journal race
720 * osd: do not use fadvise(DONTNEED) on XFS (data corruption on power cycle)
721 * osd: fix for an op ordering bug
722 * osd: fix handling for split after upgrade from bobtail
723 * osd: fix incorrect mark-down of osds
724 * osd: fix internal heartbeart timeouts when scrubbing very large objects
725 * osd: fix memory/network inefficiency during deep scrub
726 * osd: fixed problem with front-side heartbeats and mixed clusters (David Zafman)
727 * osd: limit number of incremental osdmaps sent to peers (could cause osds to be wrongly marked down)
728 * osd: many bug fixes
729 * osd: monitor both front and back interfaces
730 * osd: more efficient small object recovery
731 * osd: new writeback throttling (for less bursty write performance) (Sam Just)
732 * osd: pg log (re)writes are now vastly more efficient (faster peering) (Sam Just)
733 * osd: ping/heartbeat on public and private interfaces
734 * osd: prioritize recovery for degraded PGs
735 * osd: re-use partially deleted PG contents when present (Sam Just)
736 * osd: recovery and peering performance improvements
737 * osd: resurrect partially deleted PGs
738 * osd: verify both front and back network are working before rejoining cluster
739 * rados: clonedata command for cli
740 * radosgw-admin: create keys for new users by default
741 * rbd: /etc/ceph/rbdmap file for mapping rbd images on startup
742 * rgw: COPY object support between regions
744 * rgw: fix locking issue, user operation mask,
745 * rgw: fix radosgw-admin buckets list (Yehuda Sadeh)
746 * rgw: fix usage log scanning for large, untrimmed logs
747 * rgw: handle deep uri resources
748 * rgw: infrastructure to support georeplication of bucket and user metadata
749 * rgw: infrastructure to support georeplication of bucket data
750 * rgw: multi-region support
751 * sysvinit: fix enumeration of local daemons
752 * sysvinit: fix osd crush weight calculation when using -a
753 * sysvinit: handle symlinks in /var/lib/ceph/osd/*
754 * use SSE4.2 crc32c instruction if present
763 * There is now a configurable maximum rados object size, defaulting to 100 GB. If you
764 are using librados and storing objects larger than that, you will need to adjust
765 'osd max object size', and should consider using smaller objects instead.
770 * osd: pg log (re)writes are now vastly more efficient (faster peering) (Sam Just)
771 * osd: fixed problem with front-side heartbeats and mixed clusters (David Zafman)
772 * mon: tuning, performance improvements
773 * mon: simplify PaxosService vs Paxos interaction, fix readable/writeable checks
774 * rgw: fix radosgw-admin buckets list (Yehuda Sadeh)
775 * mds: support robust lookup by ino number (good for NFS) (Yan, Zheng)
776 * mds: fix several bugs (Yan, Zheng)
777 * ceph-fuse, libcephfs: fix truncatation bug on >4MB files (Yan, Zheng)
778 * ceph/librados: fix resending of commands on mon reconnect
779 * librados python: fix xattrs > 4KB (Josh Durgin)
780 * librados: configurable max object size (default 100 GB)
781 * msgr: fix various memory leaks
782 * ceph-fuse: fixed long-standing O_NOATIME vs O_LAZY bug
783 * ceph-fuse, libcephfs: fix request refcounting bug (hang on shutdown)
784 * ceph-fuse, libcephfs: fix read zeroing at EOF
785 * ceph-conf: --show-config-value now reflects daemon defaults
786 * ceph-disk: simpler, more robust locking
787 * ceph-disk: avoid mounting over an existing osd in /var/lib/ceph/osd/*
788 * sysvinit: handle symlinks in /var/lib/ceph/osd/*
797 * Huge revamp of the 'ceph' command-line interface implementation.
798 The ``ceph-common`` client library needs to be upgrade before
799 ``ceph-mon`` is restarted in order to avoid problems using the CLI
800 (the old ``ceph`` client utility cannot talk to the new
803 * The CLI is now very careful about sending the 'status' one-liner
804 output to stderr and command output to stdout. Scripts relying on
805 output should take care.
807 * The 'ceph osd tell ...' and 'ceph mon tell ...' commands are no
808 longer supported. Any callers should use::
810 ceph tell osd.<id or *> ...
811 ceph tell mon.<id or name or *> ...
813 The 'ceph mds tell ...' command is still there, but will soon also
814 transition to 'ceph tell mds.<id or name or \*> ...'
816 * The 'ceph osd crush add ...' command used to take one of two forms::
818 ceph osd crush add 123 osd.123 <weight> <location ...>
819 ceph osd crush add osd.123 <weight> <location ...>
821 This is because the id and crush name are redundant. Now only the
822 simple form is supported, where the osd name/id can either be a bare
823 id (integer) or name (osd.<id>)::
825 ceph osd crush add osd.123 <weight> <location ...>
826 ceph osd crush add 123 <weight> <location ...>
828 * There is now a maximum RADOS object size, configurable via 'osd max
829 object size', defaulting to 100 GB. Note that this has no effect on
830 RBD, CephFS, or radosgw, which all stripe over objects.
836 * mon, ceph: huge revamp of CLI and internal admin API. (Dan Mick)
837 * mon: new capability syntax
838 * osd: do not use fadvise(DONTNEED) on XFS (data corruption on power cycle)
839 * osd: recovery and peering performance improvements
840 * osd: new writeback throttling (for less bursty write performance) (Sam Just)
841 * osd: ping/heartbeat on public and private interfaces
842 * osd: avoid osd flapping from asymmetric network failure
843 * osd: re-use partially deleted PG contents when present (Sam Just)
844 * osd: break blacklisted client watches (David Zafman)
845 * mon: many stability fixes (Joao Luis)
846 * mon, osd: many memory leaks fixed
847 * mds: misc stability fixes (Yan, Zheng, Greg Farnum)
848 * mds: many backpointer improvements (Yan, Zheng)
849 * mds: new robust open-by-ino support (Yan, Zheng)
850 * ceph-fuse, libcephfs: fix a few caps revocation bugs
851 * librados: new calls to administer the cluster
852 * librbd: locking tests (Josh Durgin)
853 * ceph-disk: improved handling of odd device names
854 * ceph-disk: many fixes for RHEL/CentOS, Fedora, wheezy
855 * many many fixes from static code analysis (Danny Al-Gaaf)
856 * daemons: create /var/run/ceph as needed
865 * New pools now have the HASHPSPOOL flag set by default to provide
866 better distribution over OSDs. Support for this feature was
867 introduced in v0.59 and Linux kernel version v3.9. If you wish to
868 access the cluster from an older kernel, set the 'osd pool default
869 flag hashpspool = false' option in your ceph.conf prior to creating
870 the cluster or creating new pools. Note that the presence of any
871 pool in the cluster with the flag enabled will make the OSD require
872 support from all clients.
877 * osd: monitor both front and back interfaces
878 * osd: verify both front and back network are working before rejoining cluster
879 * osd: fix memory/network inefficiency during deep scrub
880 * osd: fix incorrect mark-down of osds
881 * mon: fix start fork behavior
882 * mon: fix election timeout
883 * mon: better trim/compaction behavior
884 * mon: fix units in 'ceph df' output
885 * mon, osd: misc memory leaks
886 * librbd: make default options/features for newly created images (e.g., via qemu-img) configurable
887 * mds: many fixes for mds clustering
888 * mds: fix rare hang after client restart
889 * ceph-fuse: add ioctl support
890 * ceph-fuse/libcephfs: fix for cap release/hang
891 * rgw: handle deep uri resources
893 * ceph-disk: add '[un]suppress-active DEV' command
894 * debian: rgw: stop daemon on uninstall
895 * debian: fix upstart behavior with upgrades
904 * The 'osd min down {reporters|reports}' config options have been
905 renamed to 'mon osd min down {reporters|reports}', and the
906 documentation has been updated to reflect that these options apply
907 to the monitors (who process failure reports) and not OSDs. If you
908 have adjusted these settings, please update your ``ceph.conf``
914 * librbd: parallelize delete, rollback, flatten, copy, resize
915 * librbd: ability to read from local replicas
916 * osd: resurrect partially deleted PGs
917 * osd: prioritize recovery for degraded PGs
918 * osd: fix internal heartbeart timeouts when scrubbing very large objects
919 * osd: close narrow journal race
920 * rgw: fix usage log scanning for large, untrimmed logs
921 * rgw: fix locking issue, user operation mask,
922 * initscript: fix osd crush weight calculation when using -a
923 * initscript: fix enumeration of local daemons
924 * mon: several fixes to paxos, sync
925 * mon: new --extract-monmap to aid disaster recovery
926 * mon: fix leveldb compression, trimming
927 * add 'config get' admin socket command
928 * rados: clonedata command for cli
929 * debian: stop daemons on uninstall; fix dependencies
930 * debian wheezy: fix udev rules
931 * many many small fixes from coverity scan
940 * mon: fix validation of mds ids from CLI commands
941 * osd: fix for an op ordering bug
942 * osd, mon: optionally dump leveldb transactions to a log
943 * osd: fix handling for split after upgrade from bobtail
944 * debian, specfile: packaging cleanups
945 * radosgw-admin: create keys for new users by default
946 * librados python binding cleanups
projects / ceph.git / blob