[lttng-dev] Deleted files held open by LTTng

[lttng-dev] Deleted files held open by LTTng

[nils.doering--- via lttng-dev]

Hi all,

I have recently started using LTTng and stumbled upon the fact, that
LTTng is keeping a lot of open file pointers to a single deleted file.

Specifically the process that holds these open is the lttng-consumerd and its
threads. The file that is held open, but is deleted is
/dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
number of open file descriptors is dependent on the number of CPUs the
system has and results to 16 + (#CPUs * 16). Is this behavior expected?

# lsof | grep lttng | grep DEL
lttng-con  2280             root  DEL       REG               0,18               11128
        /dev/shm/ust-shm-consumer-2280
lttng-con  2280             root  DEL       REG               0,18               11127
        /dev/shm/ust-shm-consumer-2280
lttng-con  2280             root  DEL       REG               0,18               11126
        /dev/shm/ust-shm-consumer-2280
[...]

I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
SLES12 SP5 without Xen.

Best regards,
Nils Döring
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Deleted files held open by LTTng

[Mathieu Desnoyers via lttng-dev]

----- On Oct 12, 2020, at 5:04 AM, lttng-dev lttng-dev@lists.lttng.org wrote:

> Hi all,
> 
> I have recently started using LTTng and stumbled upon the fact, that
> LTTng is keeping a lot of open file pointers to a single deleted file.
> 
> Specifically the process that holds these open is the lttng-consumerd and its
> threads. The file that is held open, but is deleted is
> /dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
> number of open file descriptors is dependent on the number of CPUs the
> system has and results to 16 + (#CPUs * 16). Is this behavior expected?
> 
> # lsof | grep lttng | grep DEL
> lttng-con  2280             root  DEL       REG               0,18
> 11128
>        /dev/shm/ust-shm-consumer-2280
> lttng-con  2280             root  DEL       REG               0,18
> 11127
>        /dev/shm/ust-shm-consumer-2280
> lttng-con  2280             root  DEL       REG               0,18
> 11126
>        /dev/shm/ust-shm-consumer-2280
> [...]
> 
> I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
> SLES12 SP5 without Xen.

Yes, it works exactly as designed. The inodes to each of those files stay
allocated as along as there is at least one file descriptor referencing them.
We use this scheme to facilitate cleaning up when the consumer daemon is
killed (including with SIGKILL) so it does not leave stale files hanging
around.

Note that on Linux, and starting from Linux kernel 3.17, we could instead
use the memfd_create system call to create those shared mappings, which AFAIU
would remove the need for unlinking the shm files after their creation.
However, considering that LTTng-tools and LTTng-UST also works on older Linux
kernels, and on BSDs, using the POSIX shm still appears to be appropriate.

On the reason for having one shm file per cpu (per channel/per uid/per session):
having one file per cpu is mainly done to facilitate NUMA-local memory allocation.
That being said, I wonder if as a future improvement we could combine all per-cpu
buffers into a single shm file, and issue numa_set_preferred piecewise when
mmapping regions of the files. This would lessen the number of file descriptors
needed by a significant amount. I suspect it would work, but we'd need to do some
prototyping to validate this approach first.

Thanks,

Mathieu

> 
> Best regards,
> Nils Döring
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Deleted files held open by LTTng

[nils.doering--- via lttng-dev]

>> Hi all,
>> 
>> I have recently started using LTTng and stumbled upon the fact, that
>> LTTng is keeping a lot of open file pointers to a single deleted file.
>> 
>> Specifically the process that holds these open is the lttng-consumerd and its
>> threads. The file that is held open, but is deleted is
>> /dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
>> number of open file descriptors is dependent on the number of CPUs the
>> system has and results to 16 + (#CPUs * 16). Is this behavior expected?
>> 
>> # lsof | grep lttng | grep DEL
>> lttng-con  2280             root  DEL       REG               0,18
>> 11128
>>        /dev/shm/ust-shm-consumer-2280
>> lttng-con  2280             root  DEL       REG               0,18
>> 11127
>>        /dev/shm/ust-shm-consumer-2280
>> lttng-con  2280             root  DEL       REG               0,18
>> 11126
>>        /dev/shm/ust-shm-consumer-2280
>> [...]
>> 
>> I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
>> SLES12 SP5 without Xen.
> 
> Yes, it works exactly as designed. The inodes to each of those files stay
> allocated as along as there is at least one file descriptor referencing them.
> We use this scheme to facilitate cleaning up when the consumer daemon is
> killed (including with SIGKILL) so it does not leave stale files hanging
> around.
> 
> 
> Note that on Linux, and starting from Linux kernel 3.17, we could instead
> use the memfd_create system call to create those shared mappings, which AFAIU
> would remove the need for unlinking the shm files after their creation.
> However, considering that LTTng-tools and LTTng-UST also works on older Linux
> kernels, and on BSDs, using the POSIX shm still appears to be appropriate. >
>
> On the reason for having one shm file per cpu (per channel/per uid/per session):
> having one file per cpu is mainly done to facilitate NUMA-local memory allocation.
> That being said, I wonder if as a future improvement we could combine all per-cpu
> buffers into a single shm file, and issue numa_set_preferred piecewise when
> mmapping regions of the files. This would lessen the number of file descriptors
> needed by a significant amount. I suspect it would work, but we'd need to do some
> prototyping to validate this approach first.
>
> Thanks,
>
> Mathieu
>

Thank you for your explanation. This helps a lot in understanding.

Best wishes,
Nils Döring

>> 
>> Best regards,
>> Nils Döring
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev