Too Many Open Files

Too Many Open Files

[Gruher, Joseph R]

I have a 72-core server running CentOS 8, kernel 4.18.0-240, with FIO 3.27.  I have 12 NVMe drives, each with 12 partitions, and am trying to run multiple jobs for each partition, with numjobs=16 and iodepth=32.  I get a too many open files error from FIO.  Which is fine, that's a lot of threads.  If I change numjobs to 8, same error, if I change it to 4, FIO runs.  Is there a hard or dynamic limit in FIO, and if so, what is it, or how is it calculated?  Thanks!

[global]
ioengine=libaio
direct=1
randrepeat=0
thread=1
time_based=1
ramp_time=300
runtime=900
group_reporting=1
norandommap
iodepth=32
numjobs=4
bs=4k
rw=randread
[job-2-1]
filename=/dev/nvme2n1p1
[job-2-2]
filename=/dev/nvme2n1p2
[job-2-3]
filename=/dev/nvme2n1p3
[job-2-4]
filename=/dev/nvme2n1p4
[job-2-5]
filename=/dev/nvme2n1p5
[job-2-6]
filename=/dev/nvme2n1p6

(goes on like that for 12 drives * 12 partitions per drive)

Re: Too Many Open Files

[Sitsofe Wheeler]

Hi,

On Sat, 29 May 2021 at 16:03, Gruher, Joseph R
<joseph.r.gruher@intel.com> wrote:
>
> I have a 72-core server running CentOS 8, kernel 4.18.0-240, with FIO 3.27.  I have 12 NVMe drives, each with 12 partitions, and am trying to run multiple jobs for each partition, with numjobs=16 and iodepth=32.  I get a too many open files error from FIO.  Which is fine, that's a lot of threads.  If I change numjobs to 8, same error, if I change it to 4, FIO runs.  Is there a hard or dynamic limit in FIO, and if so, what is it, or how is it calculated?  Thanks!
>
>
> [global]
> ioengine=libaio
> direct=1
> randrepeat=0
> thread=1

In this case I don't think you're hitting an fio limit but your system
likely has limits (e.g. a global maximum and a seperate per process
limit). I'm guessing 12 (disks) * 12 (partitions) * 16 (jobs) = 2304
files. 12 * 12 * 8 = 1152 files. The fact that 4 worked makes me think
you have some per user or per process limit which is a number like
1024. Does the information in https://unix.stackexchange.com/q/84227
help?


--
Sitsofe