Bug #2825
closedFile lock doesn't work properly
0%
Description
Hi,
I've been testing the locking feature on cephfs and seems that the locking doesn't work as expected when locking/writing on the same file from 2 different client. I've been testing this on version 0.48
Here are the steps to reproduce the problem.
3 servers used: ceph01, ceph02, ceph03 running each mon and osd + ceph01 running mds
ceph01,ceph02 mount cephfs as follow: (in fstab)
ceph01,ceph02:/ /mnt/ceph ceph name=admin,secretfile=/etc/ceph/secret,noexec,nodev,noatime 0 2
the code used to reproduce the problem:
#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <err.h>
int
main(int argc, char **argv)
{
int fd;
char *buf;
struct flock fl;
int count, i;
if (argc < 4) {
printf("Usage: %s <file> <count> <str>\n", argv[0]);
exit(1);
}
if ((fd = open(argv[1], O_RDWR | O_CREAT | O_APPEND, 0644)) == -1)
err(1, "open");
count = atoi(argv[2]);
asprintf(&buf, "%s\n", argv[3]);
if (buf == NULL)
err(1, "asprintf");
fl.l_type = F_WRLCK;
fl.l_whence = SEEK_SET;
fl.l_start = 0;
fl.l_len = 0;
fl.l_pid = 0;
if (fcntl(fd, F_SETLKW, &fl) -1)
err(1, "fcntl");
for (i = 0; i < count || count 0; i++) {
if (write(fd, buf, strlen(buf)) == -1)
err(1, "write");
printf("%d - %s", i + 1, buf);
sleep(1);
}
close(fd);
return 0;
}
when it works:
1- on the local fs not using local file system (ext4)
steps: on ceph01 open 2 ssh shells and run the following commands:
first shell: ceph01$ ./mr_lock /home/myuser/bla 3 foo; ./mr_lock /home/myuser/bla 3 foo
second shell: ceph01$ ./mr_lock /home/myuser/bla 3 bar; ./mr_lock /home/myuser/bla 3 bar
result: cat bla
foo
foo
foo
bar
bar
bar
foo
foo
foo
bar
bar
bar
2- on the same client using ceph file system
steps: open 2 shells on ceph01 and run the following commands:
first shell: ceph01$ ./mr_lock /mnt/ceph/bla 3 foo; ./mr_lock /mnt/ceph/bla 3 foo
second shell: ceph01$ ./mr_lock /mnt/ceph/bla 3 bar; ./mr_lock /mnt/ceph/bla 3 bar
result: cat /mnt/ceph/bla
foo
foo
foo
bar
bar
bar
foo
foo
foo
bar
bar
bar
when it doesn't work:
1- when 2 clients are connected to ceph
steps: open 1 shell on each ceph01 and ceph02 and run the following commands:
first shell: ceph01$ ./mr_lock /mnt/ceph/bla 3 foo; ./mr_lock /mnt/ceph/bla 3 foo
second shell: ceph02$ ./mr_lock /mnt/ceph/bla 3 bar; ./mr_lock /mnt/ceph/bla 3 bar
result:
ceph01$ cat /mnt/ceph/bla
bar
bar
bar
bar
bar
bar
another try shows this result:
cat /mnt/ceph/bla
foo
bar
bar
foo
bar
bar
bar
Let me know if you need anything additionnal info.
Updated by Jean-Sébastien Frerot almost 12 years ago
Here is the code included in pre ;)
#define _GNU_SOURCE #include <sys/types.h> #include <sys/stat.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <stdlib.h> #include <string.h> #include <err.h> int main(int argc, char **argv) { int fd; char *buf; struct flock fl; int count, i; if (argc < 4) { printf("Usage: %s <file> <count> <str>\n", argv[0]); exit(1); } if ((fd = open(argv[1], O_RDWR | O_CREAT | O_APPEND, 0644)) == -1) err(1, "open"); count = atoi(argv[2]); asprintf(&buf, "%s\n", argv[3]); if (buf == NULL) err(1, "asprintf"); fl.l_type = F_WRLCK; fl.l_whence = SEEK_SET; fl.l_start = 0; fl.l_len = 0; fl.l_pid = 0; if (fcntl(fd, F_SETLKW, &fl) == -1) err(1, "fcntl"); for (i = 0; i < count || count == 0; i++) { if (write(fd, buf, strlen(buf)) == -1) err(1, "write"); printf("%d - %s", i + 1, buf); sleep(1); } close(fd); return 0; }
Updated by Anonymous over 10 years ago
@Jean-Sébastien Frerot: Please retest with the following fix:
commit 476e4902907dfadb3709ba820453299ececf990b
Author: David Disseldorp <ddiss@suse.de>
Date: Mon Jul 29 17:05:44 2013 +0200
mds: remove waiting lock before merging with neighbours
Updated by David Pippenger almost 10 years ago
I've been using 0.80.1 on a vanilla 3.10.33 kernel. I am seeing this issue and can reproduce it reliably using the test scenarios and code provided above. I've produced the results using both the native kernel client and fuse-ceph client for mounts. I'll copy the contents of this into a new bug if it's preferred.
Updated by Greg Farnum almost 10 years ago
Yeah, please open a new ticket; this one's been closed for a while. :)
Updated by Zheng Yan almost 10 years ago
the file was opened in O_APPEND mode. client needs to ask for file size before each write