ceph/doc/radosgw/dynamicresharding.rst

   1 .. _rgw_dynamic_bucket_index_resharding:
   2
   3 ===================================
   4 RGW Dynamic Bucket Index Resharding
   5 ===================================
   6
   7 .. versionadded:: Luminous
   8
   9 A large bucket index can lead to performance problems. In order
  10 to address this problem we introduced bucket index sharding.
  11 Until Luminous, changing the number of bucket shards (resharding)
  12 needed to be done offline. Starting with Luminous we support
  13 online bucket resharding.
  14
  15 Each bucket index shard can handle its entries efficiently up until
  16 reaching a certain threshold number of entries. If this threshold is exceeded the system
  17 can encounter performance issues.
  18 The dynamic resharding feature detects this situation and automatically
  19 increases the number of shards used by the bucket index,
  20 resulting in the reduction of the number of entries in each bucket index shard.
  21 This process is transparent to the user.
  22
  23 The detection process runs:
  24
  25 1. when new objects are added to the bucket and
  26 2. in a background process that periodically scans all the buckets.
  27
  28 The background process is needed in order to deal with existing buckets in the system that are not being updated.
  29 A bucket that requires resharding is added to the resharding queue and will be
  30 scheduled to be resharded later.
  31 The reshard threads run in the background and execute the scheduled resharding tasks, one at a time.
  32
  33 Multisite
  34 =========
  35
  36 Dynamic resharding is not supported in a multisite environment.
  37
  38
  39 Configuration
  40 =============
  41
  42 Enable/Disable dynamic bucket index resharding:
  43
  44 - ``rgw_dynamic_resharding``:  true/false, default: true
  45
  46 Configuration options that control the resharding process:
  47
  48 - ``rgw_reshard_num_logs``: number of shards for the resharding queue, default: 16
  49
  50 - ``rgw_reshard_bucket_lock_duration``: duration, in seconds, of lock on bucket obj during resharding, default: 120 seconds
  51
  52 - ``rgw_max_objs_per_shard``: maximum number of objects per bucket index shard before resharding is triggered, default: 100000 objects
  53
  54 - ``rgw_reshard_thread_interval``: maximum time, in seconds, between rounds of resharding queue processing, default: 600 seconds
  55
  56
  57 Admin commands
  58 ==============
  59
  60 Add a bucket to the resharding queue
  61 ------------------------------------
  62
  63 ::
  64
  65    # radosgw-admin reshard add --bucket <bucket_name> --num-shards <new number of shards>
  66
  67 List resharding queue
  68 ---------------------
  69
  70 ::
  71
  72    # radosgw-admin reshard list
  73
  74 Process tasks on the resharding queue
  75 -------------------------------------
  76
  77 ::
  78
  79    # radosgw-admin reshard process
  80
  81 Bucket resharding status
  82 ------------------------
  83
  84 ::
  85
  86    # radosgw-admin reshard status --bucket <bucket_name>
  87
  88 The output is a json array of 3 objects (reshard_status, new_bucket_instance_id, num_shards) per shard.
  89
  90 For example, the output at different Dynamic Resharding stages is shown below:
  91
  92 ``1. Before resharding occurred:``
  93 ::
  94
  95   [
  96     {
  97         "reshard_status": "not-resharding",
  98         "new_bucket_instance_id": "",
  99         "num_shards": -1
 100     }
 101   ]
 102
 103 ``2. During resharding:``
 104 ::
 105
 106   [
 107     {
 108         "reshard_status": "in-progress",
 109         "new_bucket_instance_id": "1179f470-2ebf-4630-8ec3-c9922da887fd.8652.1",
 110         "num_shards": 2
 111     },
 112     {
 113         "reshard_status": "in-progress",
 114         "new_bucket_instance_id": "1179f470-2ebf-4630-8ec3-c9922da887fd.8652.1",
 115         "num_shards": 2
 116     }
 117   ]
 118
 119 ``3, After resharding completed:``
 120 ::
 121
 122   [
 123     {
 124         "reshard_status": "not-resharding",
 125         "new_bucket_instance_id": "",
 126         "num_shards": -1
 127     },
 128     {
 129         "reshard_status": "not-resharding",
 130         "new_bucket_instance_id": "",
 131         "num_shards": -1
 132     }
 133   ]
 134
 135
 136 Cancel pending bucket resharding
 137 --------------------------------
 138
 139 Note: Ongoing bucket resharding operations cannot be cancelled. ::
 140
 141    # radosgw-admin reshard cancel --bucket <bucket_name>
 142
 143 Manual immediate bucket resharding
 144 ----------------------------------
 145
 146 ::
 147
 148    # radosgw-admin bucket reshard --bucket <bucket_name> --num-shards <new number of shards>
 149
 150
 151 Troubleshooting
 152 ===============
 153
 154 Clusters prior to Luminous 12.2.11 and Mimic 13.2.5 left behind stale bucket
 155 instance entries, which were not automatically cleaned up. The issue also affected
 156 LifeCycle policies, which were not applied to resharded buckets anymore. Both of
 157 these issues can be worked around using a couple of radosgw-admin commands.
 158
 159 Stale instance management
 160 -------------------------
 161
 162 List the stale instances in a cluster that are ready to be cleaned up.
 163
 164 ::
 165
 166    # radosgw-admin reshard stale-instances list
 167
 168 Clean up the stale instances in a cluster. Note: cleanup of these
 169 instances should only be done on a single site cluster.
 170
 171 ::
 172
 173    # radosgw-admin reshard stale-instances rm
 174
 175
 176 Lifecycle fixes
 177 ---------------
 178
 179 For clusters that had resharded instances, it is highly likely that the old
 180 lifecycle processes would have flagged and deleted lifecycle processing as the
 181 bucket instance changed during a reshard. While this is fixed for newer clusters
 182 (from Mimic 13.2.6 and Luminous 12.2.12), older buckets that had lifecycle policies and
 183 that have undergone resharding will have to be manually fixed.
 184
 185 The command to do so is:
 186
 187 ::
 188
 189    # radosgw-admin lc reshard fix --bucket {bucketname}
 190
 191
 192 As a convenience wrapper, if the ``--bucket`` argument is dropped then this
 193 command will try and fix lifecycle policies for all the buckets in the cluster.
 194
 195 Object Expirer fixes
 196 --------------------
 197
 198 Objects subject to Swift object expiration on older clusters may have
 199 been dropped from the log pool and never deleted after the bucket was
 200 resharded. This would happen if their expiration time was before the
 201 cluster was upgraded, but if their expiration was after the upgrade
 202 the objects would be correctly handled. To manage these expire-stale
 203 objects, radosgw-admin provides two subcommands.
 204
 205 Listing:
 206
 207 ::
 208
 209    # radosgw-admin objects expire-stale list --bucket {bucketname}
 210
 211 Displays a list of object names and expiration times in JSON format.
 212
 213 Deleting:
 214
 215 ::
 216
 217    # radosgw-admin objects expire-stale rm --bucket {bucketname}
 218
 219
 220 Initiates deletion of such objects, displaying a list of object names, expiration times, and deletion status in JSON format.