Ceph

Each OSD probably needs some concurrency target (max concurrent scrubs). And a counter that indicates how many are in progress, and how many are pending (planned).

for each pg {
++pending
ask replicas to ++pending
if any decline, --pending (locally and on any replicas that replied), and go to next pg
start scrub locally and on replicas
gather results
compare
}

Something along those lines.

The tricky part is marking sure that any of this state gets cleaned up and thrown out if the PG acting set changes. So probably there is an OSD global pending/active scrub count, and also a per-pg flag, so that when we reset a given pg (in advance_map()), we know whether to decrement the global counter or not.

Probably a Mutex in OSD that protects the shared scrub accounting (active/pending counters).