.. versionadded:: Luminous
-A large bucket index can lead to performance problems. In order
-to address this problem we introduced bucket index sharding.
+A large bucket index can lead to performance problems, which can
+be addressed by sharding bucket indexes.
Until Luminous, changing the number of bucket shards (resharding)
-needed to be done offline. Starting with Luminous we support
-online bucket resharding.
+needed to be done offline, with RGW services disabled.
+Since the Luminous release Ceph has supported online bucket resharding.
Each bucket index shard can handle its entries efficiently up until
-reaching a certain threshold number of entries. If this threshold is
+reaching a certain threshold. If this threshold is
exceeded the system can suffer from performance issues. The dynamic
resharding feature detects this situation and automatically increases
-the number of shards used by the bucket index, resulting in a
-reduction of the number of entries in each bucket index shard. This
-process is transparent to the user. Write I/Os to the target bucket
-are blocked and read I/Os are not during resharding process.
+the number of shards used by a bucket's index, resulting in a
+reduction of the number of entries in each shard. This
+process is transparent to the user. Writes to the target bucket
+are blocked (but reads are not) briefly during resharding process.
By default dynamic bucket index resharding can only increase the
number of bucket index shards to 1999, although this upper-bound is a
configuration parameter (see Configuration below). When
-possible, the process chooses a prime number of bucket index shards to
-spread the number of bucket index entries across the bucket index
+possible, the process chooses a prime number of shards in order to
+spread the number of entries across the bucket index
shards more evenly.
-The detection process runs in a background process that periodically
-scans all the buckets. A bucket that requires resharding is added to
-the resharding queue and will be scheduled to be resharded later. The
-reshard thread runs in the background and execute the scheduled
-resharding tasks, one at a time.
+Detection of resharding opportunities runs as a background process
+that periodically
+scans all buckets. A bucket that requires resharding is added to
+a queue. A thread runs in the background and processes the queueued
+resharding tasks, one at a time and in order.
Multisite
=========
-Prior to the Reef release, RGW does not support dynamic resharding in a
+With Ceph releases Prior to Reef, the Ceph Object Gateway (RGW) does not support
+dynamic resharding in a
multisite environment. For information on dynamic resharding, see
:ref:`Resharding <feature_resharding>` in the RGW multisite documentation.
Configuration options that control the resharding process:
-- ``rgw_max_objs_per_shard``: maximum number of objects per bucket index shard before resharding is triggered, default: 100000 objects
+- ``rgw_max_objs_per_shard``: maximum number of objects per bucket index shard before resharding is triggered, default: 100000
-- ``rgw_max_dynamic_shards``: maximum number of shards that dynamic bucket index resharding can increase to, default: 1999
+- ``rgw_max_dynamic_shards``: maximum number of bucket index shards that dynamic resharding can increase to, default: 1999
-- ``rgw_reshard_bucket_lock_duration``: duration, in seconds, of lock on bucket obj during resharding, default: 360 seconds (i.e., 6 minutes)
+- ``rgw_reshard_bucket_lock_duration``: duration, in seconds, that writes to the bucket are locked during resharding, default: 360 (i.e., 6 minutes)
- ``rgw_reshard_thread_interval``: maximum time, in seconds, between rounds of resharding queue processing, default: 600 seconds (i.e., 10 minutes)
# radosgw-admin reshard status --bucket <bucket_name>
-The output is a json array of 3 objects (reshard_status, new_bucket_instance_id, num_shards) per shard.
+The output is a JSON array of 3 objects (reshard_status, new_bucket_instance_id, num_shards) per shard.
-For example, the output at different Dynamic Resharding stages is shown below:
+For example, the output at each dynamic resharding stage is shown below:
``1. Before resharding occurred:``
::
}
]
-``3, After resharding completed:``
+``3. After resharding completed:``
::
[
Cancel pending bucket resharding
--------------------------------
-Note: Ongoing bucket resharding operations cannot be cancelled. ::
+Note: Bucket resharding operations cannot be cancelled while executing. ::
# radosgw-admin reshard cancel --bucket <bucket_name>
# radosgw-admin bucket reshard --bucket <bucket_name> --num-shards <new number of shards>
-When choosing a number of shards, the administrator should keep a
-number of items in mind. Ideally the administrator is aiming for no
-more than 100000 entries per shard, now and through some future point
-in time.
+When choosing a number of shards, the administrator must anticipate each
+bucket's peak number of objects. Ideally one should aim for no
+more than 100000 entries per shard at any given time.
-Additionally, bucket index shards that are prime numbers tend to work
-better in evenly distributing bucket index entries across the
-shards. For example, 7001 bucket index shards is better than 7000
+Additionally, bucket index shards that are prime numbers are more effective
+in evenly distributing bucket index entries.
+For example, 7001 bucket index shards is better than 7000
since the former is prime. A variety of web sites have lists of prime
-numbers; search for "list of prime numbers" withy your favorite web
+numbers; search for "list of prime numbers" with your favorite
search engine to locate some web sites.
Troubleshooting
===============
Clusters prior to Luminous 12.2.11 and Mimic 13.2.5 left behind stale bucket
-instance entries, which were not automatically cleaned up. The issue also affected
-LifeCycle policies, which were not applied to resharded buckets anymore. Both of
-these issues can be worked around using a couple of radosgw-admin commands.
+instance entries, which were not automatically cleaned up. This issue also affected
+LifeCycle policies, which were no longer applied to resharded buckets. Both of
+these issues could be worked around by running ``radosgw-admin`` commands.
Stale instance management
-------------------------
# radosgw-admin reshard stale-instances list
Clean up the stale instances in a cluster. Note: cleanup of these
-instances should only be done on a single site cluster.
+instances should only be done on a single-site cluster.
::
Lifecycle fixes
---------------
-For clusters that had resharded instances, it is highly likely that the old
+For clusters with resharded instances, it is highly likely that the old
lifecycle processes would have flagged and deleted lifecycle processing as the
-bucket instance changed during a reshard. While this is fixed for newer clusters
-(from Mimic 13.2.6 and Luminous 12.2.12), older buckets that had lifecycle policies and
-that have undergone resharding will have to be manually fixed.
+bucket instance changed during a reshard. While this is fixed for buckets
+deployed on newer Ceph releases (from Mimic 13.2.6 and Luminous 12.2.12),
+older buckets that had lifecycle policies and that have undergone
+resharding must be fixed manually.
The command to do so is:
# radosgw-admin lc reshard fix --bucket {bucketname}
-As a convenience wrapper, if the ``--bucket`` argument is dropped then this
-command will try and fix lifecycle policies for all the buckets in the cluster.
+If the ``--bucket`` argument is not provided, this
+command will try to fix lifecycle policies for all the buckets in the cluster.
Object Expirer fixes
--------------------
resharded. This would happen if their expiration time was before the
cluster was upgraded, but if their expiration was after the upgrade
the objects would be correctly handled. To manage these expire-stale
-objects, radosgw-admin provides two subcommands.
+objects, ``radosgw-admin`` provides two subcommands.
Listing:
projects / ceph.git / blobdiff