lttng-dev.lists.lttng.org archive mirror
 help / color / mirror / Atom feed
* [lttng-dev] Deleted files held open by LTTng
@ 2020-10-12  9:04 nils.doering--- via lttng-dev
  2020-10-12 13:17 ` Mathieu Desnoyers via lttng-dev
  0 siblings, 1 reply; 3+ messages in thread
From: nils.doering--- via lttng-dev @ 2020-10-12  9:04 UTC (permalink / raw)
  To: lttng-dev

Hi all,

I have recently started using LTTng and stumbled upon the fact, that
LTTng is keeping a lot of open file pointers to a single deleted file.

Specifically the process that holds these open is the lttng-consumerd and its
threads. The file that is held open, but is deleted is
/dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
number of open file descriptors is dependent on the number of CPUs the
system has and results to 16 + (#CPUs * 16). Is this behavior expected?

# lsof | grep lttng | grep DEL
lttng-con  2280             root  DEL       REG               0,18               11128
        /dev/shm/ust-shm-consumer-2280
lttng-con  2280             root  DEL       REG               0,18               11127
        /dev/shm/ust-shm-consumer-2280
lttng-con  2280             root  DEL       REG               0,18               11126
        /dev/shm/ust-shm-consumer-2280
[...]

I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
SLES12 SP5 without Xen.

Best regards,
Nils Döring
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [lttng-dev] Deleted files held open by LTTng
  2020-10-12  9:04 [lttng-dev] Deleted files held open by LTTng nils.doering--- via lttng-dev
@ 2020-10-12 13:17 ` Mathieu Desnoyers via lttng-dev
  2020-10-13  6:47   ` nils.doering--- via lttng-dev
  0 siblings, 1 reply; 3+ messages in thread
From: Mathieu Desnoyers via lttng-dev @ 2020-10-12 13:17 UTC (permalink / raw)
  To: nils doering; +Cc: lttng-dev

----- On Oct 12, 2020, at 5:04 AM, lttng-dev lttng-dev@lists.lttng.org wrote:

> Hi all,
> 
> I have recently started using LTTng and stumbled upon the fact, that
> LTTng is keeping a lot of open file pointers to a single deleted file.
> 
> Specifically the process that holds these open is the lttng-consumerd and its
> threads. The file that is held open, but is deleted is
> /dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
> number of open file descriptors is dependent on the number of CPUs the
> system has and results to 16 + (#CPUs * 16). Is this behavior expected?
> 
> # lsof | grep lttng | grep DEL
> lttng-con  2280             root  DEL       REG               0,18
> 11128
>        /dev/shm/ust-shm-consumer-2280
> lttng-con  2280             root  DEL       REG               0,18
> 11127
>        /dev/shm/ust-shm-consumer-2280
> lttng-con  2280             root  DEL       REG               0,18
> 11126
>        /dev/shm/ust-shm-consumer-2280
> [...]
> 
> I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
> SLES12 SP5 without Xen.

Yes, it works exactly as designed. The inodes to each of those files stay
allocated as along as there is at least one file descriptor referencing them.
We use this scheme to facilitate cleaning up when the consumer daemon is
killed (including with SIGKILL) so it does not leave stale files hanging
around.

Note that on Linux, and starting from Linux kernel 3.17, we could instead
use the memfd_create system call to create those shared mappings, which AFAIU
would remove the need for unlinking the shm files after their creation.
However, considering that LTTng-tools and LTTng-UST also works on older Linux
kernels, and on BSDs, using the POSIX shm still appears to be appropriate.

On the reason for having one shm file per cpu (per channel/per uid/per session):
having one file per cpu is mainly done to facilitate NUMA-local memory allocation.
That being said, I wonder if as a future improvement we could combine all per-cpu
buffers into a single shm file, and issue numa_set_preferred piecewise when
mmapping regions of the files. This would lessen the number of file descriptors
needed by a significant amount. I suspect it would work, but we'd need to do some
prototyping to validate this approach first.

Thanks,

Mathieu

> 
> Best regards,
> Nils Döring
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [lttng-dev] Deleted files held open by LTTng
  2020-10-12 13:17 ` Mathieu Desnoyers via lttng-dev
@ 2020-10-13  6:47   ` nils.doering--- via lttng-dev
  0 siblings, 0 replies; 3+ messages in thread
From: nils.doering--- via lttng-dev @ 2020-10-13  6:47 UTC (permalink / raw)
  To: mathieu.desnoyers; +Cc: lttng-dev

>> Hi all,
>> 
>> I have recently started using LTTng and stumbled upon the fact, that
>> LTTng is keeping a lot of open file pointers to a single deleted file.
>> 
>> Specifically the process that holds these open is the lttng-consumerd and its
>> threads. The file that is held open, but is deleted is
>> /dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
>> number of open file descriptors is dependent on the number of CPUs the
>> system has and results to 16 + (#CPUs * 16). Is this behavior expected?
>> 
>> # lsof | grep lttng | grep DEL
>> lttng-con  2280             root  DEL       REG               0,18
>> 11128
>>        /dev/shm/ust-shm-consumer-2280
>> lttng-con  2280             root  DEL       REG               0,18
>> 11127
>>        /dev/shm/ust-shm-consumer-2280
>> lttng-con  2280             root  DEL       REG               0,18
>> 11126
>>        /dev/shm/ust-shm-consumer-2280
>> [...]
>> 
>> I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
>> SLES12 SP5 without Xen.
> 
> Yes, it works exactly as designed. The inodes to each of those files stay
> allocated as along as there is at least one file descriptor referencing them.
> We use this scheme to facilitate cleaning up when the consumer daemon is
> killed (including with SIGKILL) so it does not leave stale files hanging
> around.
> 
> 
> Note that on Linux, and starting from Linux kernel 3.17, we could instead
> use the memfd_create system call to create those shared mappings, which AFAIU
> would remove the need for unlinking the shm files after their creation.
> However, considering that LTTng-tools and LTTng-UST also works on older Linux
> kernels, and on BSDs, using the POSIX shm still appears to be appropriate. >
>
> On the reason for having one shm file per cpu (per channel/per uid/per session):
> having one file per cpu is mainly done to facilitate NUMA-local memory allocation.
> That being said, I wonder if as a future improvement we could combine all per-cpu
> buffers into a single shm file, and issue numa_set_preferred piecewise when
> mmapping regions of the files. This would lessen the number of file descriptors
> needed by a significant amount. I suspect it would work, but we'd need to do some
> prototyping to validate this approach first.
>
> Thanks,
>
> Mathieu
>

Thank you for your explanation. This helps a lot in understanding.

Best wishes,
Nils Döring

>> 
>> Best regards,
>> Nils Döring
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-10-13  6:48 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-12  9:04 [lttng-dev] Deleted files held open by LTTng nils.doering--- via lttng-dev
2020-10-12 13:17 ` Mathieu Desnoyers via lttng-dev
2020-10-13  6:47   ` nils.doering--- via lttng-dev

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).