From: "nils.doering--- via lttng-dev" <lttng-dev@lists.lttng.org>
To: "mathieu.desnoyers@efficios.com" <mathieu.desnoyers@efficios.com>
Cc: "lttng-dev@lists.lttng.org" <lttng-dev@lists.lttng.org>
Subject: Re: [lttng-dev] Deleted files held open by LTTng
Date: Tue, 13 Oct 2020 06:47:08 +0000 [thread overview]
Message-ID: <DB8P191MB10162024A94A720E02C3EC399D040@DB8P191MB1016.EURP191.PROD.OUTLOOK.COM> (raw)
In-Reply-To: <67256246.16573.1602508668109.JavaMail.zimbra@efficios.com>
>> Hi all,
>>
>> I have recently started using LTTng and stumbled upon the fact, that
>> LTTng is keeping a lot of open file pointers to a single deleted file.
>>
>> Specifically the process that holds these open is the lttng-consumerd and its
>> threads. The file that is held open, but is deleted is
>> /dev/shm/ust-shm-consumer-${PID} with the PID of the lttng-comsumerd. The
>> number of open file descriptors is dependent on the number of CPUs the
>> system has and results to 16 + (#CPUs * 16). Is this behavior expected?
>>
>> # lsof | grep lttng | grep DEL
>> lttng-con 2280 root DEL REG 0,18
>> 11128
>> /dev/shm/ust-shm-consumer-2280
>> lttng-con 2280 root DEL REG 0,18
>> 11127
>> /dev/shm/ust-shm-consumer-2280
>> lttng-con 2280 root DEL REG 0,18
>> 11126
>> /dev/shm/ust-shm-consumer-2280
>> [...]
>>
>> I have tested this on a Xen-Dom0 system running SLES 12 SP5 and on a
>> SLES12 SP5 without Xen.
>
> Yes, it works exactly as designed. The inodes to each of those files stay
> allocated as along as there is at least one file descriptor referencing them.
> We use this scheme to facilitate cleaning up when the consumer daemon is
> killed (including with SIGKILL) so it does not leave stale files hanging
> around.
>
>
> Note that on Linux, and starting from Linux kernel 3.17, we could instead
> use the memfd_create system call to create those shared mappings, which AFAIU
> would remove the need for unlinking the shm files after their creation.
> However, considering that LTTng-tools and LTTng-UST also works on older Linux
> kernels, and on BSDs, using the POSIX shm still appears to be appropriate. >
>
> On the reason for having one shm file per cpu (per channel/per uid/per session):
> having one file per cpu is mainly done to facilitate NUMA-local memory allocation.
> That being said, I wonder if as a future improvement we could combine all per-cpu
> buffers into a single shm file, and issue numa_set_preferred piecewise when
> mmapping regions of the files. This would lessen the number of file descriptors
> needed by a significant amount. I suspect it would work, but we'd need to do some
> prototyping to validate this approach first.
>
> Thanks,
>
> Mathieu
>
Thank you for your explanation. This helps a lot in understanding.
Best wishes,
Nils Döring
>>
>> Best regards,
>> Nils Döring
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
prev parent reply other threads:[~2020-10-13 6:48 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-12 9:04 [lttng-dev] Deleted files held open by LTTng nils.doering--- via lttng-dev
2020-10-12 13:17 ` Mathieu Desnoyers via lttng-dev
2020-10-13 6:47 ` nils.doering--- via lttng-dev [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DB8P191MB10162024A94A720E02C3EC399D040@DB8P191MB1016.EURP191.PROD.OUTLOOK.COM \
--to=lttng-dev@lists.lttng.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=nils.doering@fujitsu.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).