[SPDK] Query on scaling SPDK threads

[SPDK] Query on scaling SPDK threads

[lokesharo]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

Hello

In the SPDK source code, the message pool is of size 262143 ( taken from _thread_lib_init() ) and the per thread cache size is SPDK_MSG_MEMPOOL_CACHE_SIZE (1024). What this means is the first 255 threads created will have the cache size as 1024 and after that the threads will be created but the cache will be NULL and the threads have to refer the global pool to get the msge object to send the messages.

In my application, we create around 300 threads during init. So, 256th thread onward, global pool is referenced for the msge object. 
Now, when the program is run, some of the threads that got 1024 entries in the cache while creation (i.e. threads between 0 - 255) remain unused. What this means is that some of the entries from the global pool remain unused always and the threads (from 256 - 300) are not able to get the free entries from the global pool and this leads to spdk_thread_send_msg() failing for the threads in range of 256 - 300.

I have the following doubts here:
1. How do we scale the number of threads in the SPDK environment with the hardcoded value of cache size and pool size.
2. Why is the pool size set to 262143 and cache size to 1024. Is there some logical explanation behind the numbers?
3. If the user changes the above two numbers, are there any issues with it, be it performance or memory or any thing else?
4. Is there any maximum limit that the SPDK suggests for the scaling of threads? If yes, what is the scaling model that SPDK suggests the user should implement?

Thanks
Lokesh