From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.lttng.org (lists.lttng.org [167.114.26.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E8022C433F5 for ; Wed, 9 Mar 2022 16:37:17 +0000 (UTC) Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4KDHsm0yBBzB0Z; Wed, 9 Mar 2022 11:37:16 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1646843836; bh=hd21WHXe/JsbvpkQovERIUDPta+SY0Hfl50sMgaLktM=; h=Date:To:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=y+ZKiFyoNMuMBV9BoePqebzz3S5MIzmzBZahsoNyCcUPV9zED71EV4N4CbszKTtxH TrD8qn4kH2QaCw8zqLE6n530gExvaqYXlLo/g6uq6EFmlUcgbbTlFgCy8HsHFVROzx Dt6ZvPsnRFWqUIrZ7PjOqzPJLwczvx0xmVUuLicDUz43YWX0jtbF+JXVwOX0onzHW0 5SIaMcGwG+pqKpqb85mO95XnF4NsPtu8dtdFgFyGyeKBUoG3qmOiWHf7MlpU8A70Fq ftsG3tBDpOvM/a1a5JdeSR+iaQrUXvSkyf39UGWrOQ521NHrTsw1ccvTRwbX1zM2sG ZOb4BeuVVBmhg== Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lists.lttng.org (Postfix) with ESMTPS id 4KDHsk3WzMzBKH for ; Wed, 9 Mar 2022 11:37:09 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 293C9393FDC; Wed, 9 Mar 2022 11:37:03 -0500 (EST) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 76kX7oR4PFiD; Wed, 9 Mar 2022 11:37:02 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 604C7393FDA; Wed, 9 Mar 2022 11:37:02 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 604C7393FDA X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 9ogwJaS3nWHh; Wed, 9 Mar 2022 11:37:02 -0500 (EST) Received: from mail03.efficios.com (mail03.efficios.com [167.114.26.124]) by mail.efficios.com (Postfix) with ESMTP id 4F43D393EDB; Wed, 9 Mar 2022 11:37:02 -0500 (EST) Date: Wed, 9 Mar 2022 11:37:02 -0500 (EST) To: "zhenyu.ren" Message-ID: <816104861.134170.1646843822208.JavaMail.zimbra@efficios.com> In-Reply-To: <1a87b3ee-9983-4db6-b569-e6e6c1ab8411.zhenyu.ren@aliyun.com> References: <20220225142111.GC1861057@x> <26341add-b962-4027-8c5e-28d940e8f4dc.zhenyu.ren@aliyun.com> <2119663162.129405.1646752637020.JavaMail.zimbra@efficios.com> <1a87b3ee-9983-4db6-b569-e6e6c1ab8411.zhenyu.ren@aliyun.com> MIME-Version: 1.0 X-Originating-IP: [167.114.26.124] X-Mailer: Zimbra 8.8.15_GA_4203 (ZimbraWebClient - FF97 (Linux)/8.8.15_GA_4232) Thread-Topic: =?utf-8?B?5Zue5aSN77yaW2x0dG5nLWRldl0g5Zue5aSN77yaIOWbnuWkje+8miDlm57lpI3vvJo=?= shm leak in traced application? Thread-Index: jw7LRmFAcVsMJPBj4cp2PGx88gfkXA== Subject: Re: [lttng-dev] =?utf-8?b?5Zue5aSN77yaIOWbnuWkje+8miDlm57lpI3vvJog?= =?utf-8?b?5Zue5aSN77yaIHNobSBsZWFrIGluIHRyYWNlZCBhcHBsaWNhdGlvbj8=?= X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mathieu Desnoyers via lttng-dev Reply-To: Mathieu Desnoyers Cc: lttng-dev Content-Type: multipart/mixed; boundary="===============2971732014250284935==" Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --===============2971732014250284935== Content-Type: multipart/alternative; boundary="=_918eaa43-d5ec-4c71-91d5-a88e8d29ff10" --=_918eaa43-d5ec-4c71-91d5-a88e8d29ff10 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable When this happpens, is the process holding a single (or very few) shm file = references, or references to many=20 shm files ?=20 I wonder if you end up in a scenario where an application very frequently p= erforms exec(), and therefore=20 sometimes the exec() will happen in the window between the unix socket file= descriptor reception and=20 call to fcntl FD_CLOEXEC.=20 Thanks,=20 Mathieu=20 ----- On Mar 8, 2022, at 8:29 PM, zhenyu.ren wrote:= =20 > Thanks a lot for reply. I do not reply it in bug tracker since I have not= gotten > a reliable way to reproduce the leak case. >> ------------------------------------------------------------------ >> =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9AMathieu Desnoyers >> =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A2022=E5=B9=B43=E6=9C=888=E6= =97=A5(=E6=98=9F=E6=9C=9F=E4=BA=8C) 23:26 >> =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9Azhenyu.ren >> =E6=8A=84 =E9=80=81=EF=BC=9AJonathan Rajotte ; lttng-dev >> >> =E4=B8=BB =E9=A2=98=EF=BC=9ARe: [lttng-dev] =E5=9B=9E=E5=A4=8D=EF=BC=9A = =E5=9B=9E=E5=A4=8D=EF=BC=9A =E5=9B=9E=E5=A4=8D=EF=BC=9A shm leak in traced = application? >> ----- On Mar 8, 2022, at 12:18 AM, lttng-dev lttng-dev@lists.lttng.org w= rote: >> > Hi, >> > In shm_object_table_append_shm()/alloc_shm()=EF=BC=8C why not calling = FD_CLOEXEC fcntl() >> > to shmfds? I guess this omission leads to shm fds leak. >> Those file descriptors are created when received by ustcomm_recv_fds_uni= x_sock, >> and >> immediately after creation they are set as FD_CLOEXEC. >> We should continue this discussion in the bug tracker as suggested by Jo= nathan. >> It would greatly help if you can provide a small reproducer. >> Thanks, >> Mathieu >> > Thanks >> > zhenyu.ren >> >> ------------------------------------------------------------------ >> >> =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9AJonathan Rajotte-Julien >> >> =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A2022=E5=B9=B42=E6=9C=882= 5=E6=97=A5(=E6=98=9F=E6=9C=9F=E4=BA=94) 22:31 >> >> =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9Azhenyu.ren >> >> =E6=8A=84 =E9=80=81=EF=BC=9Alttng-dev >> >> =E4=B8=BB =E9=A2=98=EF=BC=9ARe: [lttng-dev] =E5=9B=9E=E5=A4=8D=EF=BC= =9A =E5=9B=9E=E5=A4=8D=EF=BC=9A shm leak in traced application? >> >> Hi zhenyu.ren, >> >> Please open a bug on our bug tracker and provide a reproducer against= the latest >> >> stable version (2.13.x). >> >> https://bugs.lttng.org/ >> >> Please follow the guidelines: https://bugs.lttng.org/#Bug-reporting-g= uidelines >> >> Cheers >> >> On Fri, Feb 25, 2022 at 12:47:34PM +0800, zhenyu.ren via lttng-dev wr= ote: >> >> > Hi, lttng-dev team >> >>> When lttng-sessiond exits, the ust applications should call >> >>> lttng_ust_objd_table_owner_cleanup() and clean up all shm resource(u= nmap and >> >>> close). Howerver I do find that the ust applications keep opening "a= ll" of the >> >> > shm fds("/dev/shm/ust-shm-consumer-81132 (deleted)") and do NOT fre= e shm. >> >>> If we run lttng-sessiond again, ust applications can get a new piece= of shm and >> >>> a new list of shm fds so double shm usages. Then if we kill lttng-se= ssiond, >> >>> what the mostlikely happened is ust applications close the new list = of shm fds >> >>> and free new shm resource but keeping old shm still. In other word, = we can not >> >> > free this piece of shm unless we killing ust applications!!! >> >>> So Is there any possilbe that ust applications failed calling >> >>> lttng_ust_objd_table_owner_cleanup()? Do you have ever see this prob= lem? Do you >> >>> have any advice to free the shm without killling ust applications(I = tried to >> >> > dig into kernel shm_open and /dev/shm, but not found any ideas)? >> >> > Thanks in advance >> >> > zhenyu.ren >> >> > ------------------------------------------------------------------ >> >> > =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9Azhenyu.ren via lttng-dev >> >> > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A2022=E5=B9=B42=E6=9C= =8823=E6=97=A5(=E6=98=9F=E6=9C=9F=E4=B8=89) 23:09 >> >> > =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9Alttng-dev >> >> > =E4=B8=BB =E9=A2=98=EF=BC=9A[lttng-dev] =E5=9B=9E=E5=A4=8D=EF=BC=9A= shm leak in traced application? >> >>> >"I found these items also exist in a traced application which is a = long-time >> >> > >running daemon" >> >> > Even if lttng-sessiond has been killed!! >> >> > Thanks >> >> > zhenyu.ren >> >> > ------------------------------------------------------------------ >> >> > =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9Azhenyu.ren via lttng-dev >> >> > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A2022=E5=B9=B42=E6=9C= =8823=E6=97=A5(=E6=98=9F=E6=9C=9F=E4=B8=89) 22:44 >> >> > =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9Alttng-dev >> >> > =E4=B8=BB =E9=A2=98=EF=BC=9A[lttng-dev] shm leak in traced applicat= ion? >> >> > Hi, >> >>> There are many items such as "/dev/shm/ust-shm-consumer-81132 (delet= ed)" exist >> >>> in lttng-sessiond fd spaces. I know it is the result of shm_open() a= nd >> >> > shm_unlnik() in create_posix_shm(). >> >>> However, today, I found these items also exist in a traced applicati= on which is >> >>> a long-time running daemon. The most important thing I found is that= there >> >> > seems no reliable way to release share memory. >> >>> I tried to kill lttng-sessiond but not always release share memory. = Sometimes I >> >>> need to kill the traced application to free share memory....But it i= s not a >> >> > good idea to kill these applications. >> >> > My questions are: >> >>> 1. Is there any way to release share memory without killing any trac= ed >> >> > application? >> >>> 2. Is it normal that many items such as "/dev/shm/ust-shm-consumer-8= 1132 >> >> > (deleted)" exist in the traced application? >> >> > Thanks >> >> > zhenyu.ren >> >> > _______________________________________________ >> >> > lttng-dev mailing list >> >> > lttng-dev@lists.lttng.org >> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> -- >> >> Jonathan Rajotte-Julien >> >> EfficiOS >> > _______________________________________________ >> > lttng-dev mailing list >> > lttng-dev@lists.lttng.org >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> http://www.efficios.com --=20 Mathieu Desnoyers=20 EfficiOS Inc.=20 http://www.efficios.com=20 --=_918eaa43-d5ec-4c71-91d5-a88e8d29ff10 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
When this happpens, is the process holding = a single (or very few) shm file references, or references to many
shm files ?
I wonder if you end up in a scenario wher= e an application very frequently performs exec(), and therefore
sometimes the exec() will happen in the window betw= een the unix socket file descriptor reception and
<= /div>
call to fcntl FD_CLOEXEC.

Thanks,
=
Mathieu
<= div>
----- On = Mar 8, 2022, at 8:29 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
Thanks a  lot for reply. I do not reply = it in bug tracker since I have not gotten a reliable way to reproduce the l= eak case. 
---------------------------= ---------------------------------------
=E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9AMathieu Desnoyers <mathieu.des= noyers@efficios.com>
=E5=8F=91= =E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A2022=E5=B9=B43=E6=9C=888=E6=97=A5(=E6= =98=9F=E6=9C=9F=E4=BA=8C) 23:26
= =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9Azhenyu.ren <zhenyu.ren@aliyun.com>= ;
=E6=8A=84=E3=80=80=E9=80=81=EF= =BC=9AJonathan Rajotte <jonathan.rajotte-julien@efficios.com>; lttng-= dev <lttng-dev@lists.lttng.org>
=E4=B8=BB=E3=80=80=E9=A2=98=EF=BC=9ARe: [lttng-dev] =E5=9B=9E=E5=A4=8D= =EF=BC=9A =E5=9B=9E=E5=A4=8D=EF=BC=9A =E5=9B=9E=E5=A4=8D=EF=BC=9A shm leak = in traced application?

=


----- On Mar 8, 2022, at 12:18&= nbsp;AM, lttng-dev lttng-dev@lists.lttng.org wrote:

&= gt; Hi,
> In shm_object_table_append_shm()/alloc_shm()= =EF=BC=8C why not calling FD_CLOEXEC fcntl()
&g= t; to shmfds? I guess this omission lead= s to shm fds leak.

Those file descript= ors are created when received by ustcomm_recv= _fds_unix_sock, and
immediately after creation they&= nbsp;are set as FD_CLOEXEC.

We should conti= nue this discussion in the bug tracker a= s suggested by Jonathan.
It would greatly = help if you can provide a small reproduc= er.

Thanks,

Mathieu


> Thanks
> = zhenyu.ren

>> -------------------------------------------= -----------------------
>> =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC= =9AJonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.c= om>
>> =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A2022= =E5=B9=B42=E6=9C=8825=E6=97=A5(=E6=98=9F=E6=9C=9F=E4=BA=94) 22:31
&= gt;> =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9Azhenyu.ren <zhenyu= .ren@aliyun.com>
>> =E6=8A=84 =E9=80=81=EF=BC=9Alttng= -dev <lttng-dev@lists.lttng.org>
>> =E4=B8=BB = ;=E9=A2=98=EF=BC=9ARe: [lttng-dev] =E5=9B=9E=E5=A4=8D=EF=BC=9A&nb= sp;=E5=9B=9E=E5=A4=8D=EF=BC=9A shm leak in traced = application?

>> Hi zhenyu.ren,

>> = Please open a bug on our bug tracker&nbs= p;and provide a reproducer against the latest=
>> stable version (2.13.x).

>> h= ttps://bugs.lttng.org/

>> Please follow the&nbs= p;guidelines: https://bugs.lttng.org/#Bug-reporting-guidelines

= >> Cheers

>> On Fri, Feb 25,&nb= sp;2022 at 12:47:34PM +0800, zhenyu.ren via l= ttng-dev wrote:
>> > Hi, lttng-dev tea= m
>>> When lttng-sessiond exits, the u= st applications should call
>>> lttng_ust_o= bjd_table_owner_cleanup() and clean up all shm&nbs= p;resource(unmap and
>>> close). Howerver I=  do find that the ust applications keep&= nbsp;opening "all" of the
>> > shm&nbs= p;fds("/dev/shm/ust-shm-consumer-81132 (deleted)") and do&nb= sp;NOT free shm.
>>> If we run lt= tng-sessiond again, ust applications can get = a new piece of shm and
>>> a = ;new list of shm fds so double shm = usages. Then if we kill lttng-sessiond,
>>= ;> what the mostlikely happened is ust&nbs= p;applications close the new list of shm = ;fds
>>> and free new shm resource&nbs= p;but keeping old shm still. In other wo= rd, we can not
>> > free this&nbs= p;piece of shm unless we killing ust app= lications!!!
>>> So Is there any possi= lbe that ust applications failed calling
>&g= t;> lttng_ust_objd_table_owner_cleanup()? Do you hav= e ever see this problem? Do you
>>&g= t; have any advice to free the shm = without killling ust applications(I tried to
&g= t;> > dig into kernel shm_open and = ;/dev/shm, but not found any ideas)?

>&g= t; > Thanks in advance
>> > zh= enyu.ren



>> > ---------------------------= ---------------------------------------
>> > =E5=8F= =91=E4=BB=B6=E4=BA=BA=EF=BC=9Azhenyu.ren via lttng-dev <l= ttng-dev@lists.lttng.org>
>> > =E5=8F=91=E9=80=81= =E6=97=B6=E9=97=B4=EF=BC=9A2022=E5=B9=B42=E6=9C=8823=E6=97=A5(=E6=98=9F=E6= =9C=9F=E4=B8=89) 23:09
>> > =E6=94=B6=E4=BB=B6= =E4=BA=BA=EF=BC=9Alttng-dev <lttng-dev@lists.lttng.org>
>&= gt; > =E4=B8=BB =E9=A2=98=EF=BC=9A[lttng-dev] =E5=9B= =9E=E5=A4=8D=EF=BC=9A shm leak in traced applicati= on?

>>> >"I found these items a= lso exist in a traced application which = is a long-time
>> > >running daemon= "
>> > Even if lttng-sessiond has = ;been killed!!

>> > Thanks
>> = > zhenyu.ren
>> > ---------------------------= ---------------------------------------
>> > =E5=8F= =91=E4=BB=B6=E4=BA=BA=EF=BC=9Azhenyu.ren via lttng-dev <l= ttng-dev@lists.lttng.org>
>> > =E5=8F=91=E9=80=81= =E6=97=B6=E9=97=B4=EF=BC=9A2022=E5=B9=B42=E6=9C=8823=E6=97=A5(=E6=98=9F=E6= =9C=9F=E4=B8=89) 22:44
>> > =E6=94=B6=E4=BB=B6= =E4=BA=BA=EF=BC=9Alttng-dev <lttng-dev@lists.lttng.org>
>&= gt; > =E4=B8=BB =E9=A2=98=EF=BC=9A[lttng-dev] shm&nb= sp;leak in traced application?

>> >&nbs= p;Hi,
>>> There are many items such&nb= sp;as "/dev/shm/ust-shm-consumer-81132 (deleted)" exist
&= gt;>> in lttng-sessiond fd spaces. I kn= ow it is the result of shm_open() and>> > shm_unlnik() in create_posix_shm().
&= gt;>> However, today, I found these ite= ms also exist in a traced application wh= ich is
>>> a long-time running daemon.=  The most important thing I found is&nbs= p;that there
>> > seems no reliable&nb= sp;way to release share memory.
>>> I&= nbsp;tried to kill lttng-sessiond but not alw= ays release share memory. Sometimes I
>>&= gt; need to kill the traced application = to free share memory....But it is not a<= br>>> > good idea to kill these = ;applications.
>> > My questions are:
&g= t;>> 1. Is there any way to releas= e share memory without killing any traced
= >> > application?
>>> 2. Is i= t normal that many items such as "/dev/s= hm/ust-shm-consumer-81132
>> > (deleted)" exist&= nbsp;in the traced application?

>> >&nb= sp;Thanks
>> > zhenyu.ren



>>&nbs= p;> _______________________________________________
>>&nbs= p;> lttng-dev mailing list
>> > ltt= ng-dev@lists.lttng.org
>> > https://lists.lttng.org/c= gi-bin/mailman/listinfo/lttng-dev

>> --
>> = Jonathan Rajotte-Julien
>> EfficiOS
> _______= ________________________________________
> lttng-dev mailin= g list
> lttng-dev@lists.lttng.org
> https://lis= ts.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
-- 
Mathieu = ;Desnoyers
EfficiOS Inc.
http://www.efficios.com


= --
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios= .com
--=_918eaa43-d5ec-4c71-91d5-a88e8d29ff10-- --===============2971732014250284935== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============2971732014250284935==--