* Re: Problem with application changing UID [not found] <SN4PR0601MB3614967ABCE12621B316C89D91840@SN4PR0601MB3614.namprd06.prod.outlook.com> @ 2019-09-24 15:07 ` Jonathan Rajotte-Julien 2019-09-24 15:15 ` Jonathan Rajotte-Julien ` (2 subsequent siblings) 3 siblings, 0 replies; 11+ messages in thread From: Jonathan Rajotte-Julien @ 2019-09-24 15:07 UTC (permalink / raw) To: Kramer, Zach; +Cc: lttng-dev Hi Zach, Thanks for reaching out. lttng-ust does not support the change of uid once the application is registered to the lttng-sessiond daemon. I think that we use the uid received on registration for all subsequent operations. Gabriel Pollo-Guilbert actually worked on this this summer. You can check out the proposed wrapper for setuid here [1]. You will need to LD_PRELOAD it on the start of you application. It basically unregister the application and re-register it. [1] https://lists.lttng.org/pipermail/lttng-dev/2019-June/029035.html This should be applied on master of lttng-ust. Make sure to use lttng-tools master also. Same for lttng-modules if necessary. Would you be interested in giving it a try? Cheers On Tue, Sep 24, 2019 at 02:32:41PM +0000, Kramer, Zach wrote: > Hi, > > Is LTTng intended to support userspace applications that change their UID at run-time? As in, is there an expected behavior for when this happens? > > For example: > > 1. Embedded device boots > 2. My daemon is launched as root via systemd > 3. Runs privileged code > 4. Changes UID to a less privileged user (500) > 5. Creates LTTng session > * If session already exists, destroy it first > 6. <if ‘systemctl stop’ is called>: Destroy session > * Otherwise it will be destroyed next daemon launch in step 5 > > This causes many conflicts with the trace folders that are created. Most of the time, LTTng creates a folder + metadata for both root and the user, then puts traces in the user folder. Other times, it may create a folder just for the user. This is seemingly random, since it’s a fresh device boot each time. If the daemon is launched directly (i.e. not from systemd), then the root folder gets the traces and the user folder gets the metadata. Here is a more detailed explanation: > > > Case > Command > Result > > Comments > 1 > [Device boot] > systemctl start daemon > …../uid/500/32-bit --> has metadata and trace logs > > Most times, this also happens: > …../uid/0/32-bit --> has metadata but no trace logs > > [cid:image001.png@01D56F9D.AF158770] > Occasionally, the uid/0 folder is not created at all. This seems random, since this is tested by rebooting the device several times. > 2 > [Device boot] > systemctl start daemon > systemctl stop daemon > > * This uses LTTng C-API to destroy session > …../uid/500/32-bit --> has metadata but the logs were cleared > > Most times, this also happens: > …../uid/0/32-bit --> has metadata but no trace logs > > [cid:image001.png@01D56F9D.AF158770] > The logs are cleared when ‘lttng destroy sess’ is called via the LTTng C-API. From my understanding, this should not happen. > > Destroying the daemon from command line behaves properly. From my understanding, this should be practically the exact same command. > 3 > root@device: daemon > > (launching via command-line) > When daemon is killed or session is stopped: mimics case 1 > > When daemon is alive: > …./uid/0/32-bit --> has trace logs but empty metadata > > …./uid/500/32-bit --> has metadata but empty trace logs > [cid:image002.png@01D56F9D.AF158770] > > > Is this use-case supported? > > Unfortunately, the logs are huge and contain sensitive information. If they can help a substantial amount, I can prune them. > > Thanks and best regards, > Zach > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Jonathan Rajotte-Julien EfficiOS _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Problem with application changing UID [not found] <SN4PR0601MB3614967ABCE12621B316C89D91840@SN4PR0601MB3614.namprd06.prod.outlook.com> 2019-09-24 15:07 ` Problem with application changing UID Jonathan Rajotte-Julien @ 2019-09-24 15:15 ` Jonathan Rajotte-Julien [not found] ` <20190924151515.GH187569@joraj-alpa> [not found] ` <20190924150754.GG187569@joraj-alpa> 3 siblings, 0 replies; 11+ messages in thread From: Jonathan Rajotte-Julien @ 2019-09-24 15:15 UTC (permalink / raw) To: Kramer, Zach; +Cc: lttng-dev On Tue, Sep 24, 2019 at 02:32:41PM +0000, Kramer, Zach wrote: > Hi, > > Is LTTng intended to support userspace applications that change their UID at run-time? As in, is there an expected behavior for when this happens? > > For example: > > 1. Embedded device boots > 2. My daemon is launched as root via systemd > 3. Runs privileged code > 4. Changes UID to a less privileged user (500) > 5. Creates LTTng session > * If session already exists, destroy it first > 6. <if ‘systemctl stop’ is called>: Destroy session > * Otherwise it will be destroyed next daemon launch in step 5 When in this chain of operations is lttng-sessiond started? > > [cid:image001.png@01D56F9D.AF158770] > The logs are cleared when ‘lttng destroy sess’ is called via the LTTng C-API. From my understanding, this should not happen. I would tend to agree with you here. Would you be able to provide a small reproducer for this? What is the version of lttng-* -- Jonathan Rajotte-Julien EfficiOS _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <20190924151515.GH187569@joraj-alpa>]
* Re: [EXTERNAL] Re: Problem with application changing UID [not found] ` <20190924151515.GH187569@joraj-alpa> @ 2019-09-24 15:52 ` Kramer, Zach 0 siblings, 0 replies; 11+ messages in thread From: Kramer, Zach @ 2019-09-24 15:52 UTC (permalink / raw) To: Jonathan Rajotte-Julien; +Cc: lttng-dev Hi, lttng-sessiond is launched in step 1 (with adequate delay before my daemon). The versions of LTTng are from poky/sumo: * lttng-modules 2.10.9 * lttng-tools 2.9.11 * lttng-ust 2.10.3 Unfortunately, the destroy session call that clears my logs is very likely tied to the systemd service file and the changing of user ID, so reproducing is not so simple. I will have to get back to you on whether or not I can easily make a reproduceable example. Regards, Zach -----Original Message----- From: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> Sent: Tuesday, September 24, 2019 5:15 PM To: Kramer, Zach <Zach.Kramer@cognex.com> Cc: lttng-dev@lists.lttng.org Subject: [EXTERNAL] Re: [lttng-dev] Problem with application changing UID On Tue, Sep 24, 2019 at 02:32:41PM +0000, Kramer, Zach wrote: > Hi, > > Is LTTng intended to support userspace applications that change their UID at run-time? As in, is there an expected behavior for when this happens? > > For example: > > 1. Embedded device boots > 2. My daemon is launched as root via systemd > 3. Runs privileged code > 4. Changes UID to a less privileged user (500) > 5. Creates LTTng session > * If session already exists, destroy it first > 6. <if ‘systemctl stop’ is called>: Destroy session > * Otherwise it will be destroyed next daemon launch in step 5 When in this chain of operations is lttng-sessiond started? > > [cid:image001.png@01D56F9D.AF158770] > The logs are cleared when ‘lttng destroy sess’ is called via the LTTng C-API. From my understanding, this should not happen. I would tend to agree with you here. Would you be able to provide a small reproducer for this? What is the version of lttng-* -- Jonathan Rajotte-Julien EfficiOS _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <20190924150754.GG187569@joraj-alpa>]
* Re: [EXTERNAL] Re: Problem with application changing UID [not found] ` <20190924150754.GG187569@joraj-alpa> @ 2019-09-24 16:01 ` Kramer, Zach [not found] ` <SN4PR0601MB36143DD719CB75559F9ABDD891840@SN4PR0601MB3614.namprd06.prod.outlook.com> 1 sibling, 0 replies; 11+ messages in thread From: Kramer, Zach @ 2019-09-24 16:01 UTC (permalink / raw) To: Jonathan Rajotte-Julien; +Cc: lttng-dev Hi, Thanks for the quick response! This looks interesting -- I will experiment with it and come back with any results. Cheers, Zach -----Original Message----- From: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> Sent: Tuesday, September 24, 2019 5:08 PM To: Kramer, Zach <Zach.Kramer@cognex.com> Cc: lttng-dev@lists.lttng.org Subject: [EXTERNAL] Re: [lttng-dev] Problem with application changing UID Hi Zach, Thanks for reaching out. lttng-ust does not support the change of uid once the application is registered to the lttng-sessiond daemon. I think that we use the uid received on registration for all subsequent operations. Gabriel Pollo-Guilbert actually worked on this this summer. You can check out the proposed wrapper for setuid here [1]. You will need to LD_PRELOAD it on the start of you application. It basically unregister the application and re-register it. [1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fpipermail%2Flttng-dev%2F2019-June%2F029035.html&data=02%7C01%7CZach.Kramer%40cognex.com%7C0fa610945b6a4857ce0408d74100fa70%7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637049344808702442&sdata=mkbvRXQ9GDSDIfJlD%2BL1CVqG18TDz9kxsNgMFdMIvsI%3D&reserved=0 This should be applied on master of lttng-ust. Make sure to use lttng-tools master also. Same for lttng-modules if necessary. Would you be interested in giving it a try? Cheers On Tue, Sep 24, 2019 at 02:32:41PM +0000, Kramer, Zach wrote: > Hi, > > Is LTTng intended to support userspace applications that change their UID at run-time? As in, is there an expected behavior for when this happens? > > For example: > > 1. Embedded device boots > 2. My daemon is launched as root via systemd > 3. Runs privileged code > 4. Changes UID to a less privileged user (500) > 5. Creates LTTng session > * If session already exists, destroy it first > 6. <if ‘systemctl stop’ is called>: Destroy session > * Otherwise it will be destroyed next daemon launch in step 5 > > This causes many conflicts with the trace folders that are created. Most of the time, LTTng creates a folder + metadata for both root and the user, then puts traces in the user folder. Other times, it may create a folder just for the user. This is seemingly random, since it’s a fresh device boot each time. If the daemon is launched directly (i.e. not from systemd), then the root folder gets the traces and the user folder gets the metadata. Here is a more detailed explanation: > > > Case > Command > Result > > Comments > 1 > [Device boot] > systemctl start daemon > …../uid/500/32-bit --> has metadata and trace logs > > Most times, this also happens: > …../uid/0/32-bit --> has metadata but no trace logs > > [cid:image001.png@01D56F9D.AF158770] > Occasionally, the uid/0 folder is not created at all. This seems random, since this is tested by rebooting the device several times. > 2 > [Device boot] > systemctl start daemon > systemctl stop daemon > > * This uses LTTng C-API to destroy session > …../uid/500/32-bit --> has metadata but the logs were cleared > > Most times, this also happens: > …../uid/0/32-bit --> has metadata but no trace logs > > [cid:image001.png@01D56F9D.AF158770] > The logs are cleared when ‘lttng destroy sess’ is called via the LTTng C-API. From my understanding, this should not happen. > > Destroying the daemon from command line behaves properly. From my understanding, this should be practically the exact same command. > 3 > root@device: daemon > > (launching via command-line) > When daemon is killed or session is stopped: mimics case 1 > > When daemon is alive: > …./uid/0/32-bit --> has trace logs but empty metadata > > …./uid/500/32-bit --> has metadata but empty trace logs > [cid:image002.png@01D56F9D.AF158770] > > > Is this use-case supported? > > Unfortunately, the logs are huge and contain sensitive information. If they can help a substantial amount, I can prune them. > > Thanks and best regards, > Zach > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > s.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=02%7C0 > 1%7CZach.Kramer%40cognex.com%7C0fa610945b6a4857ce0408d74100fa70%7Cc120 > 07a4882b4381b05ab783431570c7%7C1%7C0%7C637049344808702442&sdata=mA > uYCBhxqSdNuYSecmk4FmlaujH8yEX7HGEi0Ysw4ko%3D&reserved=0 -- Jonathan Rajotte-Julien EfficiOS _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <SN4PR0601MB36143DD719CB75559F9ABDD891840@SN4PR0601MB3614.namprd06.prod.outlook.com>]
* Re: [EXTERNAL] Re: Problem with application changing UID [not found] ` <SN4PR0601MB36143DD719CB75559F9ABDD891840@SN4PR0601MB3614.namprd06.prod.outlook.com> @ 2019-09-24 16:06 ` Jonathan Rajotte-Julien [not found] ` <20190924160604.GK187569@joraj-alpa> ` (2 subsequent siblings) 3 siblings, 0 replies; 11+ messages in thread From: Jonathan Rajotte-Julien @ 2019-09-24 16:06 UTC (permalink / raw) To: Kramer, Zach; +Cc: lttng-dev Hi Zach, Forgot to add that you might want to look at available UST wrappers depending on the nature of your daemon application. https://lttng.org/docs/v2.10/#doc-using-lttng-ust-with-daemons https://lttng.org/docs/v2.10/#doc-liblttng-ust-fd Cheers -- Jonathan Rajotte-Julien EfficiOS ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <20190924160604.GK187569@joraj-alpa>]
* Re: [EXTERNAL] Re: Problem with application changing UID [not found] ` <20190924160604.GK187569@joraj-alpa> @ 2019-09-24 16:13 ` Kramer, Zach [not found] ` <SN4PR0601MB361479BD3396866D29AA39F791840@SN4PR0601MB3614.namprd06.prod.outlook.com> 1 sibling, 0 replies; 11+ messages in thread From: Kramer, Zach @ 2019-09-24 16:13 UTC (permalink / raw) To: Jonathan Rajotte-Julien; +Cc: lttng-dev Hi Jonathan, Thanks for the extra info. We do indeed fork without a following exec call and I experimented with this preloading recently. When I experimented, I noticed no difference in the tracing when I preload liblttng-ust-fork.so. What exactly is the consequence of not doing this? Thanks! -----Original Message----- From: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> Sent: Tuesday, September 24, 2019 6:06 PM To: Kramer, Zach <Zach.Kramer@cognex.com> Cc: lttng-dev@lists.lttng.org Subject: Re: [EXTERNAL] Re: [lttng-dev] Problem with application changing UID Hi Zach, Forgot to add that you might want to look at available UST wrappers depending on the nature of your daemon application. https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.10%2F%23doc-using-lttng-ust-with-daemons&data=02%7C01%7CZach.Kramer%40cognex.com%7C5bd45c30c5bd4c48ad4c08d741091b32%7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637049379681260582&sdata=R8O%2Be9j9O4O7pUAX5TAmLGQLGoIRKgg5k9vOixhTDHg%3D&reserved=0 https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.10%2F%23doc-liblttng-ust-fd&data=02%7C01%7CZach.Kramer%40cognex.com%7C5bd45c30c5bd4c48ad4c08d741091b32%7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637049379681270586&sdata=iNJnFMPDqCUzZS6a2hUrN4tGfpoInT5MgQYnANk6Y4g%3D&reserved=0 Cheers -- Jonathan Rajotte-Julien EfficiOS ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <SN4PR0601MB361479BD3396866D29AA39F791840@SN4PR0601MB3614.namprd06.prod.outlook.com>]
* Re: [EXTERNAL] Re: Problem with application changing UID [not found] ` <SN4PR0601MB361479BD3396866D29AA39F791840@SN4PR0601MB3614.namprd06.prod.outlook.com> @ 2019-09-24 18:45 ` Jonathan Rajotte-Julien 0 siblings, 0 replies; 11+ messages in thread From: Jonathan Rajotte-Julien @ 2019-09-24 18:45 UTC (permalink / raw) To: Kramer, Zach; +Cc: lttng-dev On Tue, Sep 24, 2019 at 04:13:13PM +0000, Kramer, Zach wrote: > Hi Jonathan, > > Thanks for the extra info. We do indeed fork without a following exec call and > I experimented with this preloading recently. When I experimented, I noticed > no difference in the tracing when I preload liblttng-ust-fork.so. What exactly > is the consequence of not doing this? It mostly ensure that the state of lttng-ust for the child is valid and the child will register to lttng-sessiond. It allows the child to trace between the fork call and the exec call if any. https://github.com/lttng/lttng-ust/blob/f596de62e69d1942ec545b8ba6b8f8b7244f8fb4/liblttng-ust/lttng-ust-comm.c#L2097 So in your case you definitely need it. The immediate consequence is that some context information will be completely wrong (pid/vtid). Mathieu Desnoyers could provide more information on the deeper side effects of not using the fork wrapper given that no exec call is done. I'm not completely sure of the possible side effects on the tracing buffers. Cheers -- Jonathan Rajotte-Julien EfficiOS ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [EXTERNAL] Re: Problem with application changing UID [not found] ` <SN4PR0601MB36143DD719CB75559F9ABDD891840@SN4PR0601MB3614.namprd06.prod.outlook.com> 2019-09-24 16:06 ` Jonathan Rajotte-Julien [not found] ` <20190924160604.GK187569@joraj-alpa> @ 2019-09-27 14:07 ` Kramer, Zach [not found] ` <SN4PR0601MB36145C3D44B796739CD7AF0691810@SN4PR0601MB3614.namprd06.prod.outlook.com> 3 siblings, 0 replies; 11+ messages in thread From: Kramer, Zach @ 2019-09-27 14:07 UTC (permalink / raw) To: Kramer, Zach, Jonathan Rajotte-Julien; +Cc: lttng-dev Hi, Following up on this. Unfortunately I was not able to test the patch because our build system relies on using the lttng version provided by poky -- updating is not something we can easily do. I seemingly solved this problem in another manner, though. I built the tracepoint provider package into a shared object and dlopen() it after the root privileges have been dropped. This seems to have solved the UID problem, because the app only registers itself under the UID that it will trace under. This resulted in another problem, which is that the metadata does not exist until the session is stopped, and once the session is started again it is no longer tracing. In the lttng documentation I found: "Important: You need to stop tracing to make LTTng flush the remaining trace data and make the trace readable." I interpreted this as "the metadata will not exist until the session is stopped." Is this correct? Regardless, I enabled the channel 'metadata' and configured it to flush via the switch timer, which then gives my traces metadata. My problem thus is seemingly solved. Do you see any flaws on the lttng side of this approach? Best, Zach -----Original Message----- From: lttng-dev <lttng-dev-bounces@lists.lttng.org> On Behalf Of Kramer, Zach Sent: Tuesday, September 24, 2019 6:01 PM To: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> Cc: lttng-dev@lists.lttng.org Subject: Re: [lttng-dev] [EXTERNAL] Re: Problem with application changing UID Hi, Thanks for the quick response! This looks interesting -- I will experiment with it and come back with any results. Cheers, Zach -----Original Message----- From: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> Sent: Tuesday, September 24, 2019 5:08 PM To: Kramer, Zach <Zach.Kramer@cognex.com> Cc: lttng-dev@lists.lttng.org Subject: [EXTERNAL] Re: [lttng-dev] Problem with application changing UID Hi Zach, Thanks for reaching out. lttng-ust does not support the change of uid once the application is registered to the lttng-sessiond daemon. I think that we use the uid received on registration for all subsequent operations. Gabriel Pollo-Guilbert actually worked on this this summer. You can check out the proposed wrapper for setuid here [1]. You will need to LD_PRELOAD it on the start of you application. It basically unregister the application and re-register it. [1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fpipermail%2Flttng-dev%2F2019-June%2F029035.html&data=02%7C01%7Czach.kramer%40cognex.com%7C515e8022a83848c7e97f08d741086bce%7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637049376731748051&sdata=XXMusNtcLMynO8QhHU220Plq06P7qvrhenIFBFcTu0A%3D&reserved=0 This should be applied on master of lttng-ust. Make sure to use lttng-tools master also. Same for lttng-modules if necessary. Would you be interested in giving it a try? Cheers On Tue, Sep 24, 2019 at 02:32:41PM +0000, Kramer, Zach wrote: > Hi, > > Is LTTng intended to support userspace applications that change their UID at run-time? As in, is there an expected behavior for when this happens? > > For example: > > 1. Embedded device boots > 2. My daemon is launched as root via systemd > 3. Runs privileged code > 4. Changes UID to a less privileged user (500) > 5. Creates LTTng session > * If session already exists, destroy it first > 6. <if ‘systemctl stop’ is called>: Destroy session > * Otherwise it will be destroyed next daemon launch in step 5 > > This causes many conflicts with the trace folders that are created. Most of the time, LTTng creates a folder + metadata for both root and the user, then puts traces in the user folder. Other times, it may create a folder just for the user. This is seemingly random, since it’s a fresh device boot each time. If the daemon is launched directly (i.e. not from systemd), then the root folder gets the traces and the user folder gets the metadata. Here is a more detailed explanation: > > > Case > Command > Result > > Comments > 1 > [Device boot] > systemctl start daemon > …../uid/500/32-bit --> has metadata and trace logs > > Most times, this also happens: > …../uid/0/32-bit --> has metadata but no trace logs > > [cid:image001.png@01D56F9D.AF158770] > Occasionally, the uid/0 folder is not created at all. This seems random, since this is tested by rebooting the device several times. > 2 > [Device boot] > systemctl start daemon > systemctl stop daemon > > * This uses LTTng C-API to destroy session > …../uid/500/32-bit --> has metadata but the logs were cleared > > Most times, this also happens: > …../uid/0/32-bit --> has metadata but no trace logs > > [cid:image001.png@01D56F9D.AF158770] > The logs are cleared when ‘lttng destroy sess’ is called via the LTTng C-API. From my understanding, this should not happen. > > Destroying the daemon from command line behaves properly. From my understanding, this should be practically the exact same command. > 3 > root@device: daemon > > (launching via command-line) > When daemon is killed or session is stopped: mimics case 1 > > When daemon is alive: > …./uid/0/32-bit --> has trace logs but empty metadata > > …./uid/500/32-bit --> has metadata but empty trace logs > [cid:image002.png@01D56F9D.AF158770] > > > Is this use-case supported? > > Unfortunately, the logs are huge and contain sensitive information. If they can help a substantial amount, I can prune them. > > Thanks and best regards, > Zach > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > s.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=02%7C0 > 1%7CZach.Kramer%40cognex.com%7C0fa610945b6a4857ce0408d74100fa70%7Cc120 > 07a4882b4381b05ab783431570c7%7C1%7C0%7C637049344808702442&sdata=mA > uYCBhxqSdNuYSecmk4FmlaujH8yEX7HGEi0Ysw4ko%3D&reserved=0 -- Jonathan Rajotte-Julien EfficiOS _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=02%7C01%7Czach.kramer%40cognex.com%7C515e8022a83848c7e97f08d741086bce%7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637049376731758042&sdata=m2f45UtNBgI2PPF%2BqKxOlSokbZnTyYDfAJeULiAFhv4%3D&reserved=0 _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <SN4PR0601MB36145C3D44B796739CD7AF0691810@SN4PR0601MB3614.namprd06.prod.outlook.com>]
* Re: [EXTERNAL] Re: Problem with application changing UID [not found] ` <SN4PR0601MB36145C3D44B796739CD7AF0691810@SN4PR0601MB3614.namprd06.prod.outlook.com> @ 2019-09-30 19:56 ` Mathieu Desnoyers [not found] ` <1505194855.4263.1569873368480.JavaMail.zimbra@efficios.com> 1 sibling, 0 replies; 11+ messages in thread From: Mathieu Desnoyers @ 2019-09-30 19:56 UTC (permalink / raw) To: Kramer, Zach; +Cc: lttng-dev ----- On Sep 27, 2019, at 10:07 AM, Kramer, Zach Zach.Kramer@cognex.com wrote: > Hi, > > Following up on this. > > Unfortunately I was not able to test the patch because our build system relies > on using the lttng version provided by poky -- updating is not something we can > easily do. > > I seemingly solved this problem in another manner, though. I built the > tracepoint provider package into a shared object and dlopen() it after the root > privileges have been dropped. This seems to have solved the UID problem, > because the app only registers itself under the UID that it will trace under. > > This resulted in another problem, which is that the metadata does not exist > until the session is stopped, This is expected. > and once the session is started again it is no longer tracing. Well that means your solution does not completely work. > In the lttng documentation I found: > > "Important: You need to stop tracing to make LTTng flush the remaining trace > data and make the trace readable." > > I interpreted this as "the metadata will not exist until the session is > stopped." Is this correct? > > Regardless, I enabled the channel 'metadata' and configured it to flush via the > switch timer, which then gives my traces metadata. > > My problem thus is seemingly solved. Do you see any flaws on the lttng side of > this approach? If you want to consume your traces without stopping tracing, you need to use the lttng "live" mode or the new "session rotation" feature introduced in lttng 2.11. Without those features, reading a trace while it's being produced is racy and may yield to corrupted traces. Thanks, Mathieu > > Best, > Zach > > -----Original Message----- > From: lttng-dev <lttng-dev-bounces@lists.lttng.org> On Behalf Of Kramer, Zach > Sent: Tuesday, September 24, 2019 6:01 PM > To: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> > Cc: lttng-dev@lists.lttng.org > Subject: Re: [lttng-dev] [EXTERNAL] Re: Problem with application changing UID > > Hi, > > Thanks for the quick response! This looks interesting -- I will experiment with > it and come back with any results. > > Cheers, > Zach > > > -----Original Message----- > From: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> > Sent: Tuesday, September 24, 2019 5:08 PM > To: Kramer, Zach <Zach.Kramer@cognex.com> > Cc: lttng-dev@lists.lttng.org > Subject: [EXTERNAL] Re: [lttng-dev] Problem with application changing UID > > Hi Zach, > > Thanks for reaching out. > > lttng-ust does not support the change of uid once the application is registered > to the lttng-sessiond daemon. I think that we use the uid received on > registration for all subsequent operations. > > Gabriel Pollo-Guilbert actually worked on this this summer. You can check out > the proposed wrapper for setuid here [1]. You will need to LD_PRELOAD it on the > start of you application. It basically unregister the application and > re-register it. > > [1] > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fpipermail%2Flttng-dev%2F2019-June%2F029035.html&data=02%7C01%7Czach.kramer%40cognex.com%7C515e8022a83848c7e97f08d741086bce%7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637049376731748051&sdata=XXMusNtcLMynO8QhHU220Plq06P7qvrhenIFBFcTu0A%3D&reserved=0 > This should be applied on master of lttng-ust. > Make sure to use lttng-tools master also. Same for lttng-modules if necessary. > > Would you be interested in giving it a try? > > Cheers > > On Tue, Sep 24, 2019 at 02:32:41PM +0000, Kramer, Zach wrote: >> Hi, >> >> Is LTTng intended to support userspace applications that change their UID at >> run-time? As in, is there an expected behavior for when this happens? >> >> For example: >> >> 1. Embedded device boots >> 2. My daemon is launched as root via systemd >> 3. Runs privileged code >> 4. Changes UID to a less privileged user (500) >> 5. Creates LTTng session >> * If session already exists, destroy it first >> 6. <if ‘systemctl stop’ is called>: Destroy session >> * Otherwise it will be destroyed next daemon launch in step 5 >> >> This causes many conflicts with the trace folders that are created. Most of the >> time, LTTng creates a folder + metadata for both root and the user, then puts >> traces in the user folder. Other times, it may create a folder just for the >> user. This is seemingly random, since it’s a fresh device boot each time. If >> the daemon is launched directly (i.e. not from systemd), then the root folder >> gets the traces and the user folder gets the metadata. Here is a more detailed >> explanation: >> >> >> Case >> Command >> Result >> >> Comments >> 1 >> [Device boot] >> systemctl start daemon >> …../uid/500/32-bit --> has metadata and trace logs >> >> Most times, this also happens: >> …../uid/0/32-bit --> has metadata but no trace logs >> >> [cid:image001.png@01D56F9D.AF158770] >> Occasionally, the uid/0 folder is not created at all. This seems random, since >> this is tested by rebooting the device several times. >> 2 >> [Device boot] >> systemctl start daemon >> systemctl stop daemon >> >> * This uses LTTng C-API to destroy session >> …../uid/500/32-bit --> has metadata but the logs were cleared >> >> Most times, this also happens: >> …../uid/0/32-bit --> has metadata but no trace logs >> >> [cid:image001.png@01D56F9D.AF158770] >> The logs are cleared when ‘lttng destroy sess’ is called via the LTTng C-API. >> From my understanding, this should not happen. >> >> Destroying the daemon from command line behaves properly. From my understanding, >> this should be practically the exact same command. >> 3 >> root@device: daemon >> >> (launching via command-line) >> When daemon is killed or session is stopped: mimics case 1 >> >> When daemon is alive: >> …./uid/0/32-bit --> has trace logs but empty metadata >> >> …./uid/500/32-bit --> has metadata but empty trace logs >> [cid:image002.png@01D56F9D.AF158770] >> >> >> Is this use-case supported? >> >> Unfortunately, the logs are huge and contain sensitive information. If they can >> help a substantial amount, I can prune them. >> >> Thanks and best regards, >> Zach > > > > >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev@lists.lttng.org >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist >> s.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=02%7C0 >> 1%7CZach.Kramer%40cognex.com%7C0fa610945b6a4857ce0408d74100fa70%7Cc120 >> 07a4882b4381b05ab783431570c7%7C1%7C0%7C637049344808702442&sdata=mA >> uYCBhxqSdNuYSecmk4FmlaujH8yEX7HGEi0Ysw4ko%3D&reserved=0 > > > -- > Jonathan Rajotte-Julien > EfficiOS > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=02%7C01%7Czach.kramer%40cognex.com%7C515e8022a83848c7e97f08d741086bce%7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637049376731758042&sdata=m2f45UtNBgI2PPF%2BqKxOlSokbZnTyYDfAJeULiAFhv4%3D&reserved=0 > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <1505194855.4263.1569873368480.JavaMail.zimbra@efficios.com>]
* Re: [EXTERNAL] Re: Problem with application changing UID [not found] ` <1505194855.4263.1569873368480.JavaMail.zimbra@efficios.com> @ 2019-10-07 13:25 ` Kramer, Zach 0 siblings, 0 replies; 11+ messages in thread From: Kramer, Zach @ 2019-10-07 13:25 UTC (permalink / raw) To: Mathieu Desnoyers; +Cc: lttng-dev Thanks for the tips. The stopping/starting issue (resulting in no more events) still exists with a live session, but it is related to Trace Compass reading the logs, not LTTng writing them. The logs are still written to and Babeltrace can view them fine, but Trace Compass cannot view past the point where the session was stopped. Thus, this is now outside the realm of this mailing list. Best, Zach -----Original Message----- From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Sent: Monday, September 30, 2019 9:56 PM To: Kramer, Zach <Zach.Kramer@cognex.com> Cc: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>; lttng-dev <lttng-dev@lists.lttng.org> Subject: Re: [lttng-dev] [EXTERNAL] Re: Problem with application changing UID ----- On Sep 27, 2019, at 10:07 AM, Kramer, Zach Zach.Kramer@cognex.com wrote: > Hi, > > Following up on this. > > Unfortunately I was not able to test the patch because our build > system relies on using the lttng version provided by poky -- updating > is not something we can easily do. > > I seemingly solved this problem in another manner, though. I built the > tracepoint provider package into a shared object and dlopen() it after > the root privileges have been dropped. This seems to have solved the > UID problem, because the app only registers itself under the UID that it will trace under. > > This resulted in another problem, which is that the metadata does not > exist until the session is stopped, This is expected. > and once the session is started again it is no longer tracing. Well that means your solution does not completely work. > In the lttng documentation I found: > > "Important: You need to stop tracing to make LTTng flush the remaining > trace data and make the trace readable." > > I interpreted this as "the metadata will not exist until the session > is stopped." Is this correct? > > Regardless, I enabled the channel 'metadata' and configured it to > flush via the switch timer, which then gives my traces metadata. > > My problem thus is seemingly solved. Do you see any flaws on the lttng > side of this approach? If you want to consume your traces without stopping tracing, you need to use the lttng "live" mode or the new "session rotation" feature introduced in lttng 2.11. Without those features, reading a trace while it's being produced is racy and may yield to corrupted traces. Thanks, Mathieu > > Best, > Zach > > -----Original Message----- > From: lttng-dev <lttng-dev-bounces@lists.lttng.org> On Behalf Of > Kramer, Zach > Sent: Tuesday, September 24, 2019 6:01 PM > To: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> > Cc: lttng-dev@lists.lttng.org > Subject: Re: [lttng-dev] [EXTERNAL] Re: Problem with application > changing UID > > Hi, > > Thanks for the quick response! This looks interesting -- I will > experiment with it and come back with any results. > > Cheers, > Zach > > > -----Original Message----- > From: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> > Sent: Tuesday, September 24, 2019 5:08 PM > To: Kramer, Zach <Zach.Kramer@cognex.com> > Cc: lttng-dev@lists.lttng.org > Subject: [EXTERNAL] Re: [lttng-dev] Problem with application changing > UID > > Hi Zach, > > Thanks for reaching out. > > lttng-ust does not support the change of uid once the application is > registered to the lttng-sessiond daemon. I think that we use the uid > received on registration for all subsequent operations. > > Gabriel Pollo-Guilbert actually worked on this this summer. You can > check out the proposed wrapper for setuid here [1]. You will need to > LD_PRELOAD it on the start of you application. It basically unregister > the application and re-register it. > > [1] > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > s.lttng.org%2Fpipermail%2Flttng-dev%2F2019-June%2F029035.html&data > =02%7C01%7CZach.Kramer%40cognex.com%7C0a0ab1502ae24e4a4a9708d745e03d3a > %7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637054701722761406&s > data=%2FQCW1bbnLXEPcs1PQKKoa1v9D5ZSQSiGBwbO6khwhYI%3D&reserved=0 > This should be applied on master of lttng-ust. > Make sure to use lttng-tools master also. Same for lttng-modules if necessary. > > Would you be interested in giving it a try? > > Cheers > > On Tue, Sep 24, 2019 at 02:32:41PM +0000, Kramer, Zach wrote: >> Hi, >> >> Is LTTng intended to support userspace applications that change their >> UID at run-time? As in, is there an expected behavior for when this happens? >> >> For example: >> >> 1. Embedded device boots >> 2. My daemon is launched as root via systemd >> 3. Runs privileged code >> 4. Changes UID to a less privileged user (500) >> 5. Creates LTTng session >> * If session already exists, destroy it first >> 6. <if ‘systemctl stop’ is called>: Destroy session >> * Otherwise it will be destroyed next daemon launch in step 5 >> >> This causes many conflicts with the trace folders that are created. >> Most of the time, LTTng creates a folder + metadata for both root and >> the user, then puts traces in the user folder. Other times, it may >> create a folder just for the user. This is seemingly random, since >> it’s a fresh device boot each time. If the daemon is launched >> directly (i.e. not from systemd), then the root folder gets the >> traces and the user folder gets the metadata. Here is a more detailed >> explanation: >> >> >> Case >> Command >> Result >> >> Comments >> 1 >> [Device boot] >> systemctl start daemon >> …../uid/500/32-bit --> has metadata and trace logs >> >> Most times, this also happens: >> …../uid/0/32-bit --> has metadata but no trace logs >> >> [cid:image001.png@01D56F9D.AF158770] >> Occasionally, the uid/0 folder is not created at all. This seems >> random, since this is tested by rebooting the device several times. >> 2 >> [Device boot] >> systemctl start daemon >> systemctl stop daemon >> >> * This uses LTTng C-API to destroy session >> …../uid/500/32-bit --> has metadata but the logs were cleared >> >> Most times, this also happens: >> …../uid/0/32-bit --> has metadata but no trace logs >> >> [cid:image001.png@01D56F9D.AF158770] >> The logs are cleared when ‘lttng destroy sess’ is called via the LTTng C-API. >> From my understanding, this should not happen. >> >> Destroying the daemon from command line behaves properly. From my >> understanding, this should be practically the exact same command. >> 3 >> root@device: daemon >> >> (launching via command-line) >> When daemon is killed or session is stopped: mimics case 1 >> >> When daemon is alive: >> …./uid/0/32-bit --> has trace logs but empty metadata >> >> …./uid/500/32-bit --> has metadata but empty trace logs >> [cid:image002.png@01D56F9D.AF158770] >> >> >> Is this use-case supported? >> >> Unfortunately, the logs are huge and contain sensitive information. >> If they can help a substantial amount, I can prune them. >> >> Thanks and best regards, >> Zach > > > > >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev@lists.lttng.org >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis >> t >> s.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=02%7C >> 0 >> 1%7CZach.Kramer%40cognex.com%7C0fa610945b6a4857ce0408d74100fa70%7Cc12 >> 0 >> 07a4882b4381b05ab783431570c7%7C1%7C0%7C637049344808702442&sdata=m >> A >> uYCBhxqSdNuYSecmk4FmlaujH8yEX7HGEi0Ysw4ko%3D&reserved=0 > > > -- > Jonathan Rajotte-Julien > EfficiOS > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > s.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=02%7C0 > 1%7CZach.Kramer%40cognex.com%7C0a0ab1502ae24e4a4a9708d745e03d3a%7Cc120 > 07a4882b4381b05ab783431570c7%7C1%7C0%7C637054701722761406&sdata=ZN > 1%2FNSV2Gsxx88tbkSrQFn5LR0Kyc%2FfidPKTkPgP4Ag%3D&reserved=0 > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > s.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=02%7C0 > 1%7CZach.Kramer%40cognex.com%7C0a0ab1502ae24e4a4a9708d745e03d3a%7Cc120 > 07a4882b4381b05ab783431570c7%7C1%7C0%7C637054701722761406&sdata=ZN > 1%2FNSV2Gsxx88tbkSrQFn5LR0Kyc%2FfidPKTkPgP4Ag%3D&reserved=0 -- Mathieu Desnoyers EfficiOS Inc. https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.efficios.com&data=02%7C01%7CZach.Kramer%40cognex.com%7C0a0ab1502ae24e4a4a9708d745e03d3a%7Cc12007a4882b4381b05ab783431570c7%7C1%7C0%7C637054701722761406&sdata=s3hl9M6zXFQQQD4VOGlAUe4U6E5ecZHCerTeQIfq%2FEk%3D&reserved=0 _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 11+ messages in thread
* Problem with application changing UID
@ 2019-09-24 14:32 Kramer, Zach
0 siblings, 0 replies; 11+ messages in thread
From: Kramer, Zach @ 2019-09-24 14:32 UTC (permalink / raw)
To: lttng-dev
[-- Attachment #1.1.1: Type: text/plain, Size: 2493 bytes --]
Hi,
Is LTTng intended to support userspace applications that change their UID at run-time? As in, is there an expected behavior for when this happens?
For example:
1. Embedded device boots
2. My daemon is launched as root via systemd
3. Runs privileged code
4. Changes UID to a less privileged user (500)
5. Creates LTTng session
* If session already exists, destroy it first
6. <if ‘systemctl stop’ is called>: Destroy session
* Otherwise it will be destroyed next daemon launch in step 5
This causes many conflicts with the trace folders that are created. Most of the time, LTTng creates a folder + metadata for both root and the user, then puts traces in the user folder. Other times, it may create a folder just for the user. This is seemingly random, since it’s a fresh device boot each time. If the daemon is launched directly (i.e. not from systemd), then the root folder gets the traces and the user folder gets the metadata. Here is a more detailed explanation:
Case
Command
Result
Comments
1
[Device boot]
systemctl start daemon
…../uid/500/32-bit --> has metadata and trace logs
Most times, this also happens:
…../uid/0/32-bit --> has metadata but no trace logs
[cid:image001.png@01D56F9D.AF158770]
Occasionally, the uid/0 folder is not created at all. This seems random, since this is tested by rebooting the device several times.
2
[Device boot]
systemctl start daemon
systemctl stop daemon
* This uses LTTng C-API to destroy session
…../uid/500/32-bit --> has metadata but the logs were cleared
Most times, this also happens:
…../uid/0/32-bit --> has metadata but no trace logs
[cid:image001.png@01D56F9D.AF158770]
The logs are cleared when ‘lttng destroy sess’ is called via the LTTng C-API. From my understanding, this should not happen.
Destroying the daemon from command line behaves properly. From my understanding, this should be practically the exact same command.
3
root@device: daemon
(launching via command-line)
When daemon is killed or session is stopped: mimics case 1
When daemon is alive:
…./uid/0/32-bit --> has trace logs but empty metadata
…./uid/500/32-bit --> has metadata but empty trace logs
[cid:image002.png@01D56F9D.AF158770]
Is this use-case supported?
Unfortunately, the logs are huge and contain sensitive information. If they can help a substantial amount, I can prune them.
Thanks and best regards,
Zach
[-- Attachment #1.1.2: Type: text/html, Size: 18835 bytes --]
[-- Attachment #1.2: image003.png --]
[-- Type: image/png, Size: 3141 bytes --]
[-- Attachment #1.3: image004.png --]
[-- Type: image/png, Size: 6313 bytes --]
[-- Attachment #2: Type: text/plain, Size: 156 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 11+ messages in threadend of thread, other threads:[~2019-10-07 13:25 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <SN4PR0601MB3614967ABCE12621B316C89D91840@SN4PR0601MB3614.namprd06.prod.outlook.com>
2019-09-24 15:07 ` Problem with application changing UID Jonathan Rajotte-Julien
2019-09-24 15:15 ` Jonathan Rajotte-Julien
[not found] ` <20190924151515.GH187569@joraj-alpa>
2019-09-24 15:52 ` [EXTERNAL] " Kramer, Zach
[not found] ` <20190924150754.GG187569@joraj-alpa>
2019-09-24 16:01 ` Kramer, Zach
[not found] ` <SN4PR0601MB36143DD719CB75559F9ABDD891840@SN4PR0601MB3614.namprd06.prod.outlook.com>
2019-09-24 16:06 ` Jonathan Rajotte-Julien
[not found] ` <20190924160604.GK187569@joraj-alpa>
2019-09-24 16:13 ` Kramer, Zach
[not found] ` <SN4PR0601MB361479BD3396866D29AA39F791840@SN4PR0601MB3614.namprd06.prod.outlook.com>
2019-09-24 18:45 ` Jonathan Rajotte-Julien
2019-09-27 14:07 ` Kramer, Zach
[not found] ` <SN4PR0601MB36145C3D44B796739CD7AF0691810@SN4PR0601MB3614.namprd06.prod.outlook.com>
2019-09-30 19:56 ` Mathieu Desnoyers
[not found] ` <1505194855.4263.1569873368480.JavaMail.zimbra@efficios.com>
2019-10-07 13:25 ` Kramer, Zach
2019-09-24 14:32 Kramer, Zach
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.