Bug #213
closed
non-idempotent transactions (clone) under ext3 may not replay correct result
Added by Sage Weil almost 14 years ago.
Updated over 12 years ago.
Description
The writeahead journaling will restore the store to a known state regardless of which operations have committed, but only if the transactions are idempotent. i.e. can be repeated and still end up at a known result. This is true of all the normal operations (remove, write, truncate, setxattr, etc., but not for clone and clone_range. E.g.,
clone foo_head -> foo_2
write foo_head
will put the old head content in _2, but if it is replayed _2 will contain the new _head content. Meh!
How to fix? We could mark which transaction are non-idempotent, and sync the store before applying them.. that's expensive, but possibly the price you pay for not using btrfs! :)
- Target version changed from v0.21 to v0.22
- Target version changed from v0.22 to v0.23
- Target version changed from v0.23 to v0.24
- Target version deleted (
v0.24)
- Priority changed from High to Normal
- Translation missing: en.field_position deleted (
614)
- Translation missing: en.field_position set to 1
Isn't the idempotency in that case "clone foo_head -> foo_2 IFF foo_2 does not exist" ?
Tommi Virtanen wrote:
Isn't the idempotency in that case "clone foo_head -> foo_2 IFF foo_2 does not exist" ?
That's almost enough for clone() (if we add O_EXCL and whitelist EEXIST for non-btrfs). It wouldn't catch something like
1 clone A->B
2 modify A
...
3 delete B
4 <crash>
<replay from 1>
That trick also wouldn't work for clone_range(), which doesn't create a file.
It may be that we need to make transactions idempotent at a higher level, but it'd be dependent on what you clone to, and whether it is ever modified/removed... it'd be dependent on the particular, though, and hard to analyze/verify.
FWIW even if we know what not to replay, we could still be screwed with ext4 (which does not commit everything in order):
clone A->B
modify A
<fs commits A (before B)>
<crash>
On replay, we don't actually have the old A to clone to B. :(
I think the simplest solution would be:
- for all operations, set an xattr with the last op_seq to write to that file.
- for any operation that is potentially non-idempotent, fsync(2) after doing it.
- on replay, verify the xattr isn't == or newer to avoid re-doing the operation.
Those operations would be:
- create collection
- clone
- clone range
We need to set the attr on all operations to avoid something like
1 truncate B
2 clone A->B
3 modify A
<crash>
<replay 1>
<skip 2 due to xattr>
<replay 3>
- Translation missing: en.field_story_points deleted (
0)
- Translation missing: en.field_position deleted (
6)
- Translation missing: en.field_position set to 6
- Translation missing: en.field_position deleted (
9)
- Translation missing: en.field_position set to 7
- Translation missing: en.field_story_points set to 5
- Translation missing: en.field_position deleted (
7)
- Translation missing: en.field_position set to 7
- Translation missing: en.field_position deleted (
7)
- Translation missing: en.field_position set to 4
- Target version set to v0.39
- Translation missing: en.field_position deleted (
1)
- Translation missing: en.field_position set to 971
Update: the current first pass plan is to initiate a FileStore sync after any non-idempotent operation. This updates commit_op_seq on disk and ensures that it won't be replayed.
It's also heavyweight as it calls sync(2). So it's a big hammer, but at least it's correct.
We can set a non-idempotent bool in the do_transaction method on CLONE or anything similar and then do the commit at the end (before any other operations occur under the
current OpSequencer).
- Status changed from New to 7
- Assignee set to Sage Weil
- Status changed from 7 to Resolved
Also available in: Atom
PDF