Ceph

This assert means that either a malloc or a call to pthread_create failed. It's probably resource exhaustion of some type; if you can provide more details we might be able to suggest something.

This is weird, if the problem is caused by resource exhaustion. I run this app on a machine with i7 CPU (with 8 cores), 16GB of RAM. Right before the crash, ps showed the VSZ of the application around 142MB. I just had a few other apps running at the time, free showed I still have at least 8GB of RAM not being used. When it crashed, I only had about 20 connections. This is still far from the thousands and tens of thousands that we are expecting.

What would you suggest I do to try to reproduce that? Right now, I've found no pattern. I can help with testing this.

Ah, formatting... sorry

xp@china:~/workspace-xp/stress.rados$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 128005
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 50000
cpu time               (seconds, -t) unlimited
max user processes              (-u) 128005
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

xp@china:~/workspace-xp/stress.rados$ free
             total       used       free     shared    buffers     cached
Mem:      16402252    2697460   13704792          0      70864     689208
-/+ buffers/cache:    1937388   14464864
Swap:     15625212          0   15625212

BTW, I think it would great if the API returns an error code nicely instead of just crashing.

can you cat /proc/sys/kernel/threads-max ? on my system it's only 127837.

Sage Weil wrote:

can you cat /proc/sys/kernel/threads-max ? on my system it's only 127837.

Yeah, for each librados 'instance' (via rados_create) there are several threads that run (3-4 minimum, but more depending on how many osds you talk to). on my system, this will exhaust the kernel's per-process thread limit well before -n 40000.

This is on my system:

xp@china:~/workspace-xp/stress.rados$ cat /proc/sys/kernel/threads-max 
256010

Does it create a thread to every configured osd or only one thread to every osd where the data is located? If it's the first case, and if we have a lot of osd, that could be quite an issue. For testing, we only have 3 osd.

I bumped up the threads-max to:

xp@china:~/workspace-xp/stress.rados$ cat /proc/sys/kernel/threads-max 
768030

Xiaopong Tran wrote:

This is on my system:
[...]

Does it create a thread to every configured osd or only one thread to every osd where the data is located? If it's the first case, and if we have a lot of osd, that could be quite an issue. For testing, we only have 3 osd.

There are two threads per peer it is communicating with, plus several in overhead for each instance. How many you really consume depends somewhat on the size of the cluster and how much io each instance does.

The client library is really not intended to be instantiated thousands of times in the same process... that is horribly inefficient because you are duplicating all of the cluster session state, TCP connections, etc. You're better off sharing a single instance. librados is thread safe.

If you're just trying to do load testing, generate more workload on each client instance, and spawn multiple processes. That's much closer to what would happen in a real environment anyway.

Thanks for the update. Yes, we do have different models, including a pool of set number of rados_t instances, etc. But since we are still doing testing, we'd like to know how it behaves with different ways of connecting to the cluster. And there's an issue (#2525) which Josh opened right after this one, which we might have to check as well.