Ceph » CephFS

Ah, the problem is recovery of the MDS takes too long. (from follow-up posts to "[ceph-users] cephfs mds millions of caps")

I'd like to see measurements to see how long it takes the MDS to load metadata during recovery. Is there a way we could make that faster?

tracking the rate we add new cap to the client

Do we really care about the rate we add caps? If the client is bumping into its cache limit then it will begin trimming. We should only care if the caps are old and unused?

no about recovery time, clients already trim their cache aggressively when mds recovers.

Idle client holds so many caps is wasteful, because it increase the chance that mds trim other relatively hot objects from the cache.

Zheng Yan wrote:

Idle client holds so many caps is wasteful, because it increase the chance that mds trim other relatively hot objects from the cache.

Okay, that makes sense. I think the decay counter idea is good; let's do it.

Glad to see this :)

- Backport set to mimic,luminous

Thanks.

Can I get few implementation specific details to get started working on this issue?

And for clarity on my side, we would trim caps when the count reaches zero (and the counter reaches zero when there are no ops on the session for a certain while i.e. it is idle), right?

Rishabh Dave wrote:

Can I get few implementation specific details to get started working on this issue?

Copying from your email:

• The issue description talks about "decay counter". I haven't found
  to a source to verify what I understand by the term, but in my
  understanding we'll increase the count with the every cap that the
  client acquires and then if there are no ops for a certain period of
  time, the count must naturally decrease until either there are no more
  caps or there an op is conducted on the filesystem. Please correct me
  if I am wrong here.

DecayCounter is a data structure used in the MDS in various places to track popularity of metadata (for the purposes of subtree migration).

• The issue description says "tracking the rate we add new cap to
  client". What is the purpose behind knowing the rate? In my
  understanding, idle clients must trim caps irrespectively. Is it to
  vary the timeout for each session accordingly?

I'm not sure having a DecayCounter for each session is quite right. We'd probably want a DecayCounter per-cap and trim caps that haven't been used in a long time (like 1 week?).

• Should the code to implement this feature be implemented with MDS
  (idle sessions are request to trim caps) or with client (sessions
  trims the caps after the certain period of time)?

In the client. Only the client knows how often it uses its capabilities (i.e. how often some file I/O relies on a capability).

• Currently, on the MDS side we already trim caps for idle sessions
  after `session_timeout`[2]. So, how is this feature be different from
  that?

Idle sessions are not the same as idle clients in this context. An idle client would be a client that hasn't done any significant file I/O recently (recently to-be-defined). Whereas an idle session is a concept in the MDS where the client has not spoken to the MDS in session_timeout seconds (e.g. hasn't renewed caps as required).