Ceph » rgw

I've also got two clusters here with this problem, one is running 14.2.1 (50M objects in a bucket) and one 13.2.5 (450M objects in a bucket).

Looks like radosgw-admin uses libc malloc, so it's hard to say what the memory is being used for

Investigating this issue,
it is possible to alleviate the "wait time" increasing incrementally after each iteration of

radosgw-admin bucket rm --bucket=$BIG_BUCKET --bypass-gc --purge-objects

radosgw-admin bucket check --bucket=$BIG_BUCKET --fix

That's interesting, Mark!

So the bucket index is left in an unsynchronized state (i.e., original state) when bucket removal is terminated part-way through. And then when bucket removal is restarted, it begins by trying to re-remove those same objects at the head of the bucket index all over again, causing a delay before forward progress is made.

Since the bucket removal is generally expected to complete, there "should" be no need to update the bucket index at "check-points" during the bucket removal process.

If terminating bucket removal is semi-expected (possibly through manual admin intervention), it seems that updating the index after every 100,000 to 1,000,000 objects is removed would mitigate this, without creating a lot of overhead.

And would there be any benefit to removing the objects from back to front in the bucket index? In other words, is there an easy way to truncate the index of its tail members, making the update of the bucket index quick?

Update -
found the source of the memory growth:

src/rgw/rgw_rados.cc

RGWObjState *RGWObjectCtx::get_state(const rgw_obj& obj) {
  RGWObjState *result;
  typename std::map<rgw_obj, RGWObjState>::iterator iter;
  lock.lock_shared();
  assert (!obj.empty());
  iter = objs_state.find(obj);
  if (iter != objs_state.end()) {
    result = &iter->second;
    lock.unlock_shared();
  } else {
    lock.unlock_shared();
    lock.lock();
    result = &objs_state[obj];    <--------------
    lock.unlock();
  }
  return result;
}

Submitted proposed fix PR.