Ceph » rgw

What I know is that ceph intentionally creates a new random oid for the reuploaded parts. My understanding that, among other things, this is to prevent race conditions and/or issues with simultaneous uploads of the same part (two instances of the client uploading the same part at the same time).

When the multipart complete is issued, ceph knows how to look up the valid pieces via some stored metadata (manifest maybe?), however it appears the original (no longer needed) parts are NOT kept track of by any mechanism, and thus are just abandoned in the data pool.

I am not not yet clear why the index entries for the replacement parts remain in the index. I could be way off here, but it looks like it is only referencing "src_obj" at the end of the following function for removal, and not the new replacement part index entries:

rgw_op.cc function void RGWCompleteMultipart::execute()
<--- code ommitted --->
rgw_obj_index_key remove_key;
src_obj.key.get_index_key(&remove_key);
remove_objs.push_back(remove_key);

Just to add onto this with our own findings:

The orphan issue seems to be related to bucket sharding. Running

# radosgw-admin bucket check --check-objects --fix --bucket <bucket name>

Removed the orphaned data from multipart uploads, but only when bucket shards are set to 0. It fails to clean data on buckets with shards > 0.

The ticket points out that remaining orphans have a different id than the pieces which are recombined. The fact that these pieces aren’t visible from a client service accessing ceph metadata sounds like its an issue with the way ceph is tracking the pieces internally. The method for cataloging multipart objects seems to not have taken into account shards or forgotten the system “shadow tags” added onto the upload ID to differentiate them.

Setting bucket shards to a lower amount doesn’t change anything. After running this command than a bucket check, the orphaned data remains.

[root@os1-sin1 ~]# radosgw-admin reshard add --bucket test --num-shards 7 --yes-i-really-mean-it
[root@os1-sin1 ~]# radosgw-admin reshard list
[
 {
        "time": "2020-09-24T17:14:42.189517Z",
        "tenant": "",
        "bucket_name": "test",
        "bucket_id": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.12045805.1",
        "new_instance_id": "",
        "old_num_shards": 11,
        "new_num_shards": 7
    }
]

However setting bucket shards to 0 then running the bucket check command removed the orphan data.

[root@os1-sin1 ~]# radosgw-admin reshard add --bucket test --num-shards 0 --yes-i-really-mean-it
[root@os1-sin1 ~]# radosgw-admin reshard list
   [
 {
        "time": "2020-09-24T17:23:34.843021Z",
        "tenant": "",
        "bucket_name": "test",
        "bucket_id": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.14335315.1",
        "new_instance_id": "",
        "old_num_shards": 7,
        "new_num_shards": 0
    }
]

Gavin Chen wrote:

Removed the orphaned data from multipart uploads, but only when bucket shards are set to 0. It fails to clean data on buckets with shards > 0.

This doesn't work anymore for us since version 17.2.1. The command runs without an error on an affected bucket, shows a lot of removing manifest part from index messages, but doesn't actually do anything. The number of objects inside the bucket doesn't change.

I'm certain that this workaround did work in version 17.2.0 and before. Is this a new bug? And is there anything else we can do to get rid of the orphaned data? So far the only working method we have found is to sync the bucket with aws cli and just delete the bucket with orphan data, which is not always feasible.

Note: The orphaned data was most likely created before the upgrade to 17.2.1 in our case. Maybe that is causing an issue? I haven't seen a case with new orphaned data after the upgrade yet.

Writing on behalf of Ulrich Klein <Ulrich.Klein@ulrichklein.de>, he wanted to add some info to this tracker, below is the data he provided me:

TLDR:
Repeated multipart uploads via S3/RGW create orphaned multipart upload objects which can't be deleted.
The fragments become part of the "used space" for a user and thus screw up space accounting.
In addition these fragments eat up space on the system, and there is no way to tell how much is lost over time.
Not nice.

Searching for orphans via rgw-orphan-list and deleting the found orphaned objects doesn't help much for accounting and is dangerous (as it says) for the health of the system.

This has been the case (for us) since early Pacific releases and has not changed one bit up to my current 17.2.3
Yes. I'll update to the latest version eventually. Had that suggestion a few times before, without any change in behavior.

How I can easily reproduce the problem:
---------------------------------------
My small test system. It's the primary of a multi-site setup, I.e. no resharding. But the system and version at least since early Pacific don't matter.

#ceph -s
cluster:
id: c05843a6-f154-11ec-a618-0de3c7490b0c
health: HEALTH_OK

services:
mon: 5 daemons, quorum maxvm1,maxvm2,maxvm3,maxvm5,maxvm6 (age 5h)
mgr: maxvm1.gykqvn(active, since 5h), standbys: maxvm2.uieuyz, maxvm4.xqlsdy
osd: 12 osds: 12 up (since 5h), 12 in (since 8w)
rgw: 3 daemons active (3 hosts, 1 zones)

data:
pools: 10 pools, 289 pgs
objects: 38.82k objects, 124 GiB
usage: 252 GiB used, 3.3 TiB / 3.5 TiB avail
pgs: 289 active+clean

#ceph orch ps
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
alertmanager.maxvm3 maxvm3 *:9093,9094 running (5h) 7m ago 3M 35.7M - 44a71f29f42b 6bc13919d589
crash.grafavm grafavm running (7d) 7m ago 2w 21.4M - 17.2.3 7080ad88bed7 ac97a757922b
crash.maxvm1 maxvm1 running (5h) 7m ago 3M 7071k - 17.2.3 7080ad88bed7 f075f517a8e1
crash.maxvm2 maxvm2 running (5h) 7m ago 3M 19.4M - 17.2.3 7080ad88bed7 370095c77547
crash.maxvm3 maxvm3 running (5h) 7m ago 3M 7629k - 17.2.3 7080ad88bed7 6aac212190a3
crash.maxvm4 maxvm4 running (5h) 7m ago 3M 7982k - 17.2.3 7080ad88bed7 57d017d9bbc2
crash.maxvm5 maxvm5 running (5h) 7m ago 3M 7944k - 17.2.3 7080ad88bed7 b7faea6ea28c
crash.maxvm6 maxvm6 running (5h) 7m ago 3M 9235k - 17.2.3 7080ad88bed7 0d156c89fad7
grafana.maxvm1 maxvm1 *:3000 running (5h) 7m ago 3M 115M - 8.3.5 046209d1c628 2d478303af2b
mgr.maxvm1.gykqvn maxvm1 *:8443,9283 running (5h) 7m ago 3M 630M - 17.2.3 7080ad88bed7 82c614421f4b
mgr.maxvm2.uieuyz maxvm2 *:8443,9283 running (5h) 7m ago 3M 509M - 17.2.3 7080ad88bed7 50649fb04ee6
mgr.maxvm4.xqlsdy maxvm4 *:8443,9283 running (5h) 7m ago 3M 522M - 17.2.3 7080ad88bed7 9f89cb894c23
mon.maxvm1 maxvm1 running (5h) 7m ago 3M 478M 2048M 17.2.3 7080ad88bed7 676a2bff6dfc
mon.maxvm2 maxvm2 running (5h) 7m ago 3M 466M 2048M 17.2.3 7080ad88bed7 0bc966a58462
mon.maxvm3 maxvm3 running (5h) 7m ago 3M 472M 2048M 17.2.3 7080ad88bed7 9415094eb85a
mon.maxvm5 maxvm5 running (5h) 7m ago 3M 469M 2048M 17.2.3 7080ad88bed7 43174ecc265f
mon.maxvm6 maxvm6 running (5h) 7m ago 3M 465M 2048M 17.2.3 7080ad88bed7 f407c9f1347f
node-exporter.grafavm grafavm *:9100 running (7d) 7m ago 2w 17.8M - bb203ba967a8 64a51468e763
node-exporter.maxvm1 maxvm1 *:9100 running (5h) 7m ago 3M 22.4M - bb203ba967a8 31d90265f164
node-exporter.maxvm2 maxvm2 *:9100 running (5h) 7m ago 3M 25.0M - bb203ba967a8 2bd9fa6feb24
node-exporter.maxvm3 maxvm3 *:9100 running (5h) 7m ago 3M 24.8M - bb203ba967a8 189c93f71251
node-exporter.maxvm4 maxvm4 *:9100 running (5h) 7m ago 3M 22.6M - bb203ba967a8 69166f53968a
node-exporter.maxvm5 maxvm5 *:9100 running (5h) 7m ago 3M 22.7M - bb203ba967a8 10c9eaf15107
node-exporter.maxvm6 maxvm6 *:9100 running (5h) 7m ago 3M 22.4M - bb203ba967a8 d7381d02b817
osd.0 maxvm1 running (5h) 7m ago 8w 836M 4096M 17.2.3 7080ad88bed7 9217317e6d50
osd.1 maxvm2 running (5h) 7m ago 8w 903M 4096M 17.2.3 7080ad88bed7 84f3accce498
osd.2 maxvm3 running (5h) 7m ago 8w 916M 4096M 17.2.3 7080ad88bed7 715d2c0751c0
osd.3 maxvm4 running (5h) 7m ago 8w 1085M 4096M 17.2.3 7080ad88bed7 79a0d68f2cf0
osd.4 maxvm5 running (5h) 7m ago 8w 843M 4096M 17.2.3 7080ad88bed7 1db0ad36a5fd
osd.5 maxvm6 running (5h) 7m ago 8w 1035M 4096M 17.2.3 7080ad88bed7 4d908daa7057
osd.6 maxvm1 running (5h) 7m ago 8w 897M 4096M 17.2.3 7080ad88bed7 262c0b437132
osd.7 maxvm2 running (5h) 7m ago 8w 978M 4096M 17.2.3 7080ad88bed7 a6f8994da77c
osd.8 maxvm3 running (5h) 7m ago 8w 900M 4096M 17.2.3 7080ad88bed7 7c93e034f1cb
osd.9 maxvm4 running (5h) 7m ago 8w 788M 4096M 17.2.3 7080ad88bed7 ce9554342946
osd.10 maxvm5 running (5h) 7m ago 8w 961M 4096M 17.2.3 7080ad88bed7 07fe69fb59b7
osd.11 maxvm6 running (5h) 7m ago 8w 850M 4096M 17.2.3 7080ad88bed7 8cb1fd47f5e1
prometheus.maxvm2 maxvm2 *:9095 running (5h) 7m ago 3M 367M - 49058af74c32 ebb8034d22d0
rgw.max.maxvm4.kokduw maxvm4 *:8080 running (5h) 7m ago 10w 273M - 17.2.3 7080ad88bed7 1882fe5f9ccb
rgw.max.maxvm5.aunemj maxvm5 *:8080 running (5h) 7m ago 10w 268M - 17.2.3 7080ad88bed7 f4feebceda1a
rgw.max.maxvm6.lrgyyj maxvm6 *:8080 running (5h) 7m ago 10w 270M - 17.2.3 7080ad88bed7 a1246877cef8

All arm64 VMs running Ubuntu 22.04, set up via cephadm, working just fine otherwise, except for the missing ingress for which
ceph always pulls an x86_64 container image for keepalived, so I had to set up haproxy/keepalived manually ================================================================================================================================

My test execution which works every single time:
------------------------------------------------
#create ec21, not strictly necessary, but as we use EC-Data pools it makes the pool "similar"
ceph osd erasure-code-profile set ec21 k=2 m=1

#Create pools for new placement target, so only these pools get screwed up
ceph osd pool create max.rgw.mptest.data erasure ec21 --autoscale-mode=on
ceph osd pool application enable max.rgw.mptest.data rgw
ceph osd pool create max.rgw.mptest.index replicated --autoscale-mode=on
ceph osd pool application enable max.rgw.mptest.index rgw
ceph osd pool create max.rgw.mptest.non-ec replicated --autoscale-mode=on
ceph osd pool application enable max.rgw.mptest.non-ec rgw

#create zonegroup placement target
radosgw-admin zonegroup placement add --rgw-zonegroup maxzg --placement-id mptest

#create zone placement target
radosgw-admin zone placement add --rgw-zone max --placement-id mptest --data-pool max.rgw.mptest.data --index-pool max.rgw.mptest.index --data-extra-pool max.rgw.mptest.non-ec

radosgw-admin period update --commit

#Create user mptester
radosgw-admin user create --uid=mptester --display-name="MP Tester" --email=mptester@ulrichklein.de

#make that user use the new placement target, the only one using this placement target
radosgw-admin user modify --uid=mptester --placement-id mptest

rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.mgr 4.7 MiB 2 0 6 0 0 0 15207 37 MiB 10455 176 MiB 0 B 0 B
.rgw.root 672 KiB 57 0 171 0 0 0 13466 16 MiB 616 422 KiB 0 B 0 B
max.rgw.control 0 B 8 0 24 0 0 0 0 0 B 0 0 B 0 B 0 B
...
max.rgw.log 5.9 MiB 826 0 2478 0 0 0 6611719 4.9 GiB 1418196 191 MiB 0 B 0 B
max.rgw.meta 793 KiB 75 0 225 0 0 0 30367 25 MiB 3068 1.2 MiB 0 B 0 B
max.rgw.mptest.data 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
max.rgw.mptest.index 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
max.rgw.mptest.non-ec 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
max.rgw.otp 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

total_objects 38814
total_used 251 GiB
total_avail 3.3 TiB
total_space 3.5 TiB

#Go to client machine #====================
#use rclone ...
#Create a bucket for mptester
rclone mkdir mptester:/mptest

#On Ceph #=======
adosgw-admin bucket stats --bucket=mptest {
"bucket": "mptest",
"num_shards": 11,
"tenant": "",
"zonegroup": "d8ea45b1-d527-427d-ba1e-fd9cdfe526f8",
"placement_rule": "mptest",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1",
"marker": "47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1",
"index_type": "Normal",
"owner": "mptester",
"ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2022-10-11T11:27:45.247514Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

#on Client machine ... #=====================
#have a dir mptestfiles with just one 8G file:
ls l mptestfiles/
total 16797696
-rw-r---- 1 mptester staff 8589934592 Oct 11 12:23 file1

#rclone sync the dir to the bucket, interrupting the transfer 5 times at about 50%
rclone -P -v sync mptestfiles mptester:/mptest
^C

#then let one finish
rclone -P -v sync mptestfiles mptester:/mptest

#Let's check for unfinished MP uploads:
rclone backend list-multipart-uploads mptester:/mptest
"mptest": [ {
"Initiated": "2022-10-11T10:32:54.116Z",
"Initiator": {
"DisplayName": "MP Tester",
"ID": "mptester"
},
"Key": "file1",
"Owner": {
"DisplayName": "MP Tester",
"ID": "mptester"
},
"StorageClass": "STANDARD",
"UploadId": "2~zHESigB_NAcofmgz7VZ592IYykGChha"
}
]
}

#.. and clean up the visible one(s) using s3cmd
s3cmd abortmp s3://mptest/file1 "2~zHESigB_NAcofmgz7VZ592IYykGChha"
s3://mptest/file1

#check again ...
rclone backend list-multipart-uploads mptester:/mptest {
"mptest": []
}

#.... So, LOOKS clean

#Back to ceph system: #====================
radosgw-admin bucket stats --bucket=mptest {
"bucket": "mptest",
"num_shards": 11,
"tenant": "",
"zonegroup": "d8ea45b1-d527-427d-ba1e-fd9cdfe526f8",
"placement_rule": "mptest",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1",
"marker": "47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1",
"index_type": "Normal",
"owner": "mptester",
"ver": "0#1,1#1,2#12007,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2022-10-11T11:27:45.247514Z",
"max_marker": "0#,1#,2#00000012006.12038.5,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {
"rgw.main": {
"size": 8689549312,
"size_actual": 8689549312,
"size_utilized": 8689549312,
"size_kb": 8485888,
"size_kb_actual": 8485888,
"size_kb_utilized": 8485888,
"num_objects": 20
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 0
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

#Hmmm, 19 objects for a single file, and size 8689549312 instead of 8589934592
#(Actually pretty "good" numbers. In other test runs it was much worse)

#Back to the client ... #======================
#Delete the object
rclone delete mptester:/mptest/file1
rclone lsl mptester:/mptest
<nothing there>

#Back on Ceph #============
radosgw-admin bucket stats --bucket=mptest {
"bucket": "mptest",
"num_shards": 11,
"tenant": "",
"zonegroup": "d8ea45b1-d527-427d-ba1e-fd9cdfe526f8",
"placement_rule": "mptest",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1",
"marker": "47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1",
"index_type": "Normal",
"owner": "mptester",
"ver": "0#1,1#1,2#12008,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2022-10-11T11:27:45.247514Z",
"max_marker": "0#,1#,2#00000012007.12040.5,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {
"rgw.main": {
"size": 99614720,
"size_actual": 99614720,
"size_utilized": 99614720,
"size_kb": 97280,
"size_kb_actual": 97280,
"size_kb_utilized": 97280,
"num_objects": 19
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 0
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

#Now the "empty" bucket contains 19 objects and 99614720 bytes (95MB) of data ?
#That's already a really bad problem, as the user would be billed for 95MB instead of 0

radosgw-admin gc process --include-all

rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.mgr 4.7 MiB 2 0 6 0 0 0 15207 37 MiB 10455 176 MiB 0 B 0 B
.rgw.root 672 KiB 57 0 171 0 0 0 13539 16 MiB 616 422 KiB 0 B 0 B
max.rgw.control 0 B 8 0 24 0 0 0 0 0 B 0 0 B 0 B 0 B
...
max.rgw.log 6.8 MiB 826 0 2478 0 0 0 6618895 5.0 GiB 1419457 194 MiB 0 B 0 B
max.rgw.meta 817 KiB 78 0 234 0 0 0 30516 25 MiB 3093 1.2 MiB 0 B 0 B
max.rgw.mptest.data 1.1 GiB 274 0 822 0 0 0 11727 9 KiB 59798 29 GiB 0 B 0 B
max.rgw.mptest.index 0 B 11 0 33 0 0 0 36468 36 MiB 36065 23 MiB 0 B 0 B
max.rgw.mptest.non-ec 0 B 0 0 0 0 0 0 24057 14 MiB 6070 5.9 MiB 0 B 0 B
max.rgw.otp 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

total_objects 39102
total_used 253 GiB
total_avail 3.3 TiB
total_space 3.5 TiB
#... and the data pool 1.1 GB in 274 object, index 11 objects

#On the client delete the bucket #===============================
rclone rmdir mptester:/mptest

#On Ceph test: #=============
radosgw-admin bucket stats --bucket=mptest
failure: (2002) Unknown error 2002:
#Okaay, I guess that means "bucket not found" #
radosgw-admin gc process --include-all
rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.mgr 4.7 MiB 2 0 6 0 0 0 15207 37 MiB 10455 176 MiB 0 B 0 B
.rgw.root 672 KiB 57 0 171 0 0 0 13565 16 MiB 616 422 KiB 0 B 0 B
max.rgw.control 0 B 8 0 24 0 0 0 0 0 B 0 0 B 0 B 0 B
...
max.rgw.log 5.9 MiB 826 0 2478 0 0 0 6619613 5.0 GiB 1419613 194 MiB 0 B 0 B
max.rgw.meta 805 KiB 77 0 231 0 0 0 30666 25 MiB 3107 1.2 MiB 0 B 0 B
max.rgw.mptest.data 75 MiB 20 0 60 0 0 0 11981 9 KiB 60052 29 GiB 0 B 0 B
max.rgw.mptest.index 0 B 11 0 33 0 0 0 36592 37 MiB 36065 23 MiB 0 B 0 B
max.rgw.mptest.non-ec 0 B 0 0 0 0 0 0 24057 14 MiB 6070 5.9 MiB 0 B 0 B
max.rgw.otp 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

total_objects 38847
total_used 251 GiB
total_avail 3.3 TiB
total_space 3.5 TiB

#So, now we have a pool with nothing in it which uses 75MB in 20 objects
#What's in it:
rados ls --pool max.rgw.mptest.data | sort
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~4fhdFoPsGu4sUwa0HiOj6z4TQteBJB9.863
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.867
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.869
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.870
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~U_1f1XG0Y-Qcka8Rg9BkZqSPXSbxGpm.875
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~U_1f1XG0Y-Qcka8Rg9BkZqSPXSbxGpm.877
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~aKcRL17Bs3m3R3LB5EEfX3HTiw1SArT.874
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.874
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.875
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.876
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~4fhdFoPsGu4sUwa0HiOj6z4TQteBJB9.863_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.867_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.869_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.870_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~U_1f1XG0Y-Qcka8Rg9BkZqSPXSbxGpm.875_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~U_1f1XG0Y-Qcka8Rg9BkZqSPXSbxGpm.877_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~aKcRL17Bs3m3R3LB5EEfX3HTiw1SArT.874_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.874_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.875_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.876_1

#So far:
#RGW somehow completely forgets about MP upload "fragments" if a "failed" MP upload is repeated.
#Not only do the fragments remain in the corresponding buckets and are used in accounting,
#but even deleting the bucket does not free the space. It is lost forever, and accumulating

• WARNING: This is EXPERIMENTAL code and the results should be used
• only with CAUTION! ***
  Done at Tue Oct 11 13:59:53 CEST 2022.

cat orphan-list-20221011115953.out
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~4fhdFoPsGu4sUwa0HiOj6z4TQteBJB9.863
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.867
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.869
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.870
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~U_1f1XG0Y-Qcka8Rg9BkZqSPXSbxGpm.875
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~U_1f1XG0Y-Qcka8Rg9BkZqSPXSbxGpm.877
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~aKcRL17Bs3m3R3LB5EEfX3HTiw1SArT.874
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.874
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.875
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.876
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~4fhdFoPsGu4sUwa0HiOj6z4TQteBJB9.863_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.867_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.869_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~FYaScn-NiPEdPk6sEY5tJKXx569tO-R.870_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~U_1f1XG0Y-Qcka8Rg9BkZqSPXSbxGpm.875_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~U_1f1XG0Y-Qcka8Rg9BkZqSPXSbxGpm.877_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~aKcRL17Bs3m3R3LB5EEfX3HTiw1SArT.874_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.874_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.875_1
47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__shadow_file1.2~tyFuo0ItA0IuZh2Q335M6-Y7FNDLsrh.876_1

#Yo, as expected
#So delete them ...
rados -p max.rgw.mptest.data rm 47bae180-d58b-4438-a057-a89cc2d403f2.1934912.1__multipart_file1.2~4fhdFoPsGu4sUwa0HiOj6z4TQteBJB9.863
... for all

#So, that does seem to work for data that is not in any bucket

#Now back to the client again #============================
#Again create the bucket, do 5 interrupted and one full upload, find and delete visible mp upload fragments

#Afterwards we have on Ceph/RGW: #===============================

radosgw-admin bucket stats --bucket=mptest {
"bucket": "mptest",
"num_shards": 11,
"tenant": "",
"zonegroup": "d8ea45b1-d527-427d-ba1e-fd9cdfe526f8",
"placement_rule": "mptest",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "47bae180-d58b-4438-a057-a89cc2d403f2.1935023.1",
"marker": "47bae180-d58b-4438-a057-a89cc2d403f2.1935023.1",
"index_type": "Normal",
"owner": "mptester",
"ver": "0#1,1#1,2#12361,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2022-10-11T12:09:42.119166Z",
"max_marker": "0#,1#,2#00000012360.12395.5,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {
"rgw.main": {
"size": 94371840,
"size_actual": 94371840,
"size_utilized": 94371840,
"size_kb": 92160,
"size_kb_actual": 92160,
"size_kb_utilized": 92160,
"num_objects": 18
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 0
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.mgr 4.7 MiB 2 0 6 0 0 0 15207 37 MiB 10455 176 MiB 0 B 0 B
.rgw.root 672 KiB 57 0 171 0 0 0 13609 16 MiB 616 422 KiB 0 B 0 B
max.rgw.control 0 B 8 0 24 0 0 0 0 0 B 0 0 B 0 B 0 B
...
max.rgw.log 15 MiB 826 0 2478 0 0 0 6623585 5.0 GiB 1422665 198 MiB 0 B 0 B
max.rgw.meta 829 KiB 79 0 237 0 0 0 31224 26 MiB 3134 1.2 MiB 0 B 0 B
max.rgw.mptest.data 45 GiB 12333 0 36999 0 0 0 11989 18 KiB 109571 60 GiB 0 B 0 B
max.rgw.mptest.index 0 B 22 0 66 0 0 0 74073 74 MiB 73188 48 MiB 0 B 0 B
max.rgw.mptest.non-ec 0 B 0 0 0 0 0 0 48814 29 MiB 12310 12 MiB 0 B 0 B
max.rgw.otp 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

• WARNING: This is EXPERIMENTAL code and the results should be used
• only with CAUTION! ***
  Done at Tue Oct 11 14:20:35 CEST 2022.
  #Wow, that's a lot ...
  #Delete them all ... then
  rados df
  POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
  .mgr 4.7 MiB 2 0 6 0 0 0 15207 37 MiB 10455 176 MiB 0 B 0 B
  .rgw.root 672 KiB 57 0 171 0 0 0 13620 16 MiB 616 422 KiB 0 B 0 B
  max.rgw.control 0 B 8 0 24 0 0 0 0 0 B 0 0 B 0 B 0 B
  ...
  max.rgw.log 15 MiB 826 0 2478 0 0 0 6633903 5.0 GiB 1423537 198 MiB 0 B 0 B
  max.rgw.meta 829 KiB 79 0 237 0 0 0 31594 26 MiB 3134 1.2 MiB 0 B 0 B
  max.rgw.mptest.data 18 MiB 3 0 9 0 0 0 11995 21 KiB 121901 60 GiB 0 B 0 B
  max.rgw.mptest.index 0 B 22 0 66 0 0 0 74140 74 MiB 73188 48 MiB 0 B 0 B
  max.rgw.mptest.non-ec 0 B 0 0 0 0 0 0 48814 29 MiB 12310 12 MiB 0 B 0 B
  max.rgw.otp 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

total_objects 38843
total_used 252 GiB
total_avail 3.3 TiB
total_space 3.5 TiB

radosgw-admin bucket stats --bucket=mptest {
"bucket": "mptest",
"num_shards": 11,
"tenant": "",
"zonegroup": "d8ea45b1-d527-427d-ba1e-fd9cdfe526f8",
"placement_rule": "mptest",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "47bae180-d58b-4438-a057-a89cc2d403f2.1935023.1",
"marker": "47bae180-d58b-4438-a057-a89cc2d403f2.1935023.1",
"index_type": "Normal",
"owner": "mptester",
"ver": "0#1,1#1,2#12361,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2022-10-11T12:09:42.119166Z",
"max_marker": "0#,1#,2#00000012360.12395.5,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {
"rgw.main": {
"size": 94371840,
"size_actual": 94371840,
"size_utilized": 94371840,
"size_kb": 92160,
"size_kb_actual": 92160,
"size_kb_utilized": 92160,
"num_objects": 18
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 0
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

• WARNING: This is EXPERIMENTAL code and the results should be used
• only with CAUTION! ***
  Done at Tue Oct 11 14:30:02 CEST 2022.

#... delete them all and the bucket is screwed:
radosgw-admin bucket stats --bucket=mptest
error getting bucket stats bucket=mptest ret=-2
failure: (2) No such file or directory:

radosgw-admin user stats --uid=mptester {
"stats": {
"size": 94371840,
"size_actual": 94371840,
"size_kb": 92160,
"size_kb_actual": 92160,
"num_objects": 18
},
"last_stats_sync": "0.000000",
"last_stats_update": "2022-10-11T12:34:01.393017Z"
}
#And the stats are still wrong #

#on the client: #==============
rclone lsl mptester:/mptest
<nothing>

rclone mkdir mptester:/mptest

rclone lsl mptester:/mptest
2022/10/11 14:37:14 Failed to lsl: directory not found

clone -P -v sync mptestfiles mptester:/mptest
2022-10-11 14:39:04 INFO : S3 bucket mptest: Bucket "mptest" created with ACL "private"
2022-10-11 14:40:07 INFO : file1: Copied (new)
Transferred: 8 GiB / 8 GiB, 100%, 161.382 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1m3.0s
2022/10/11 14:40:07 INFO :
Transferred: 8 GiB / 8 GiB, 100%, 161.382 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1m3.0s

rclone lsl mptester:/mptest
2022/10/11 14:40:21 Failed to lsl: directory not found

#Looks like the pool/placement group is screwed now. Better get rid of it.

#On Ceph: #========
radosgw-admin user rm --uid=mptester --purge-data
could not remove user: unable to remove user, unable to delete user data
#Oh, well ...
radosgw-admin user rm --uid=mptester
could not remove user: unable to remove user, must specify purge data to remove user with buckets
#So, we're left with a ghost user :)
#At least deleting placement group and pools does still work
#But syncing also doesn't work anymore, always 1 shards behind on meta and data sync

#
#So, the "delete orphans" only works if the orphans are not "associated" with an existing bucket, or so ...
#Otherwise the system gets screwed in mysterious ways.

Looking at the code. In `MultipartObjectProcessor::process_first_chunk`, if writing the multipart object first chunk returns that it exists already, we generate a new random name for this part and write with this name [1]. And then in `MultipartObjectProcessor::complete` we just set "partX -> new_part_name" omap key value in the upload meta object [2], and the previously uploaded part becomes orphan.

One option to resolve this that I can imagine is to provide and use something like `omap_exchange_val_by_key` (instead of `omap_set_val_by_key`), so we could set atomically the part link, and check if it had not been set already, and if it had we could initiate removal of that previous part. A complication is that the rados backend seems not to provide something like `omap_exchange_val_by_key` and it should be implemented, and additionally we have several new backends, where this should be implemented too.

Another option could seem to be instead of "partNum -> part_name" omap values to store "part_name -> partNum" omap values, so the links would not be overwritten. And then when completing the upload (in `RGWCompleteMultipart::execute`) process the list of all uploaded parts and remove duplicates.

[1] https://github.com/ceph/ceph/blob/v17.2.5/src/rgw/rgw_putobj_processor.cc#L352
[2] https://github.com/ceph/ceph/blob/v17.2.5/src/rgw/rgw_putobj_processor.cc#L495

It is very big problem for us.

We have a lot of big buckets with orphaned parts which use hundreds TBs of space.

And second problem that bucket check can't fix it if we have sharded bucket.
We have to reshard big buckets to 0 shards and fix it. But we can't reshard very big buckets (200-500m objects) to 0 shards because it can lead to another problems with osd crash...
and fix will eat a lot of memory..

Will this ever get fixed? Or remain in the state that solutions are discussed and discarded?

I just repeated my little test from comment #6 on a single-site 17.2.6.
Result was exactly the same as for all tests before. So, I guess there had been no fix for this problem in 17.2.6.
Will there ever ...?

Hi Ulrich, this was merged to main
Thanks for point to this, I was updated status, bot will create backport tickets soon

I just repeated my little test from comment #6 on a single-site 18.2.0
The uploads were noticeable faster, but the result was exactly the same as for all tests before. So, I guess there has been no fix for this problem in 18.2.0.

Hi all, we had a user trigger this behavior, and we were able to come up with a clean up process using only radosgw-admin commands. This allowed us to delete the data so that the user could actually reuse the bucket.

1. Have the user backup all of their data.
2. `radosgw-admin bucket check --fix --bucket=$BUCKET` (bucket stats showed a bad rgw.none num_objects count)
3. `radosgw-admin bucket reshard --bucket=$BUCKET` --num-shards=29`
4. `radosgw-admin bucket rm --bucket=$BUCKET --purge-objects` to purge everything. Note that I had to run this a couple of times as it still appeared to get stuck.
5. Now that the bucket is gone, the user was able to recreate the bucket and put their data back.

We found that without the reshard and bucket check --fix, the bucket rm would get stuck and make no progress. In that situation it would infinitely loop over the same object.

We discovered this behaviour in one of our users' buckets and tried the solution given by Nathan above. Unfortunately it didn't work: `radosgw-admin bucket rm --bucket=$BUCKET --purge-objects` got stuck in a loop (it ran for ages and judging by debug logs it was spinning on a subset of objects without managing to actually delete them).

1. List all the rados objects storing data for out bucket with:

   radsogw-admin bucket radoslist --bucket=$BUCKET

2. Get the number of shards in our bucket with:

   radosgw-admin bucket stats --bucket=$BUCKET

3. Loop over each shard, deleting all metadata/omaps for our bucket by running:

   rados -p $POOL.rgw.buckets.index clearomap $BUCKET_ID.$SHARD_ID

4. At this point `bucket stats` reports our bucket as empty
5. Now that the bucket thinks it's empty it can be deleted as usual with:

   radosgw-admin bucket rm --bucket=$BUCKET

6. Finally, clear up the data objects for our former-bucket by looping over each object listed by `radoslist` above and running:

   rados -p $POOL.rgw.buckets rm $OBJ_NAME