Is there a way to make LTTng use the same subbuffers for multiple processes?

Is there a way to make LTTng use the same subbuffers for multiple processes?

[Amit Margalit]

[-- Attachment #1.1: Type: text/plain, Size: 651 bytes --]

Hi,

We're considering using LTTng as a logging / tracing solution for a 
multi-process application, which spawns many sub-processes.

Our system is a 12-core x86 (6-core + HT). If I take 4 subbuffers of 1MB, 
this means 4MB per process. As the number of processes rises, this becomes 
prohibitive.

For example, we may need to support > 500 sub-processes ==> 500 * 4MB = 
2GB - which is a bit too much.

Is there a way to tell LTTng to share the sub-buffers with forked 
processes?

If not - how difficult would it be to add such support?

Thanks,

Amit Margalit
IBM XIV - Storage Reinvented
XIV-NAS Development Team
Tel. 03-689-7774
Fax. 03-689-7230

[-- Attachment #1.2: Type: text/html, Size: 1281 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Is there a way to make LTTng use the same subbuffers for multiple processes?

[Amit Margalit]

[-- Attachment #1.1: Type: text/plain, Size: 3575 bytes --]

Security trumps most things, IMHO, so for me this is (one of the few) 
valid reasons to change backward compatibility.

An environment variable is a good solution to allow controlling this. 
Thank you.

Amit Margalit
IBM XIV - Storage Reinvented
XIV-NAS Development Team
Tel. 03-689-7774
Fax. 03-689-7230



From:   Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To:     Amit Margalit/Israel/IBM@IBMIL
Cc:     lttng-dev@lists.lttng.org, Michel Dagenais 
<michel.dagenais@polymtl.ca>
Date:   05/10/2013 05:50 PM
Subject:        Re: [lttng-dev] Is there a way to make LTTng use the same 
subbuffers for multiple processes?



I expect we could let an environment variable control this.

I would be tempted to make the default behavior switch to the new user
ID buffers in the future though, for security reasons.

Thoughts ?

Thanks,

Mathieu

* Amit Margalit (AMITM@il.ibm.com) wrote:
> This (current) behavior you describe is very suitable for our needs.
> 
> Is there an intention to insert a hook into setuid() in the future?
> If yes - will the existing behavior remain as a default and the hook be 
> used only if I choose to use it?
> 
> Thanks,
> 
> Amit Margalit
> IBM XIV - Storage Reinvented
> XIV-NAS Development Team
> Tel. 03-689-7774
> Fax. 03-689-7230
> 
> 
> 
> From:   Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> To:     Amit Margalit/Israel/IBM@IBMIL
> Cc:     Michel Dagenais <michel.dagenais@polymtl.ca>, 
> lttng-dev@lists.lttng.org
> Date:   04/29/2013 07:27 PM
> Subject:        Re: [lttng-dev] Is there a way to make LTTng use the 
same 
> subbuffers for multiple processes?
> 
> 
> 
> * Amit Margalit (AMITM@il.ibm.com) wrote:
> > Another 2.2 question -
> > 
> > What if I have a single process, that forks, and each child process 
then 
> 
> > changes its own UID. Will the tracing continue on the parent UID 
> buffers?
> 
> Yes, we indeed currently keep the user ID that was effective at process
> creation or last exec() encountered, because we don't have hooks in
> libc's setuid() at the moment.
> 
> Thanks,
> 
> Mathieu
> 
> > 
> > Thanks,
> > 
> > Amit Margalit
> > IBM XIV - Storage Reinvented
> > XIV-NAS Development Team
> > Tel. 03-689-7774
> > Fax. 03-689-7230
> > 
> > 
> > 
> > From:   Michel Dagenais <michel.dagenais@polymtl.ca>
> > To:     Amit Margalit/Israel/IBM@IBMIL
> > Cc:     lttng-dev@lists.lttng.org
> > Date:   04/29/2013 04:32 AM
> > Subject:        Re: [lttng-dev] Is there a way to make LTTng use the 
> same 
> > subbuffers for multiple processes?
> > 
> > 
> > 
> > 
> > > Our system is a 12-core x86 (6-core + HT). If I take 4 subbuffers of 

> > 1MB, this means 4MB per
> > > process. As the number of processes rises, this becomes prohibitive. 

> > > For example, we may need to support > 500 sub-processes ==> 500 * 
4MB 
> = 
> > 2GB - 
> > > which is a bit too much.  Is there a way to tell LTTng to share the 
> > sub-buffers with forked processes? 
> > 
> > This is the number one feature listed in the changelog for lttng-tools 

> and 
> > UST for version 2.2. The first release candidate is out so you can try 

> the 
> > feature, and the final 2.2 version should not be too far away.
> > 
> > http://lttng.org/lttng-tools-220-rc1
> > - Add UST per UID buffers support
> > 
> 
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
> 
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com



[-- Attachment #1.2: Type: text/html, Size: 6101 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Is there a way to make LTTng use the same subbuffers for multiple processes?

[Mathieu Desnoyers]

I expect we could let an environment variable control this.

I would be tempted to make the default behavior switch to the new user
ID buffers in the future though, for security reasons.

Thoughts ?

Thanks,

Mathieu

* Amit Margalit (AMITM@il.ibm.com) wrote:
> This (current) behavior you describe is very suitable for our needs.
> 
> Is there an intention to insert a hook into setuid() in the future?
> If yes - will the existing behavior remain as a default and the hook be 
> used only if I choose to use it?
> 
> Thanks,
> 
> Amit Margalit
> IBM XIV - Storage Reinvented
> XIV-NAS Development Team
> Tel. 03-689-7774
> Fax. 03-689-7230
> 
> 
> 
> From:   Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> To:     Amit Margalit/Israel/IBM@IBMIL
> Cc:     Michel Dagenais <michel.dagenais@polymtl.ca>, 
> lttng-dev@lists.lttng.org
> Date:   04/29/2013 07:27 PM
> Subject:        Re: [lttng-dev] Is there a way to make LTTng use the same 
> subbuffers for multiple processes?
> 
> 
> 
> * Amit Margalit (AMITM@il.ibm.com) wrote:
> > Another 2.2 question -
> > 
> > What if I have a single process, that forks, and each child process then 
> 
> > changes its own UID. Will the tracing continue on the parent UID 
> buffers?
> 
> Yes, we indeed currently keep the user ID that was effective at process
> creation or last exec() encountered, because we don't have hooks in
> libc's setuid() at the moment.
> 
> Thanks,
> 
> Mathieu
> 
> > 
> > Thanks,
> > 
> > Amit Margalit
> > IBM XIV - Storage Reinvented
> > XIV-NAS Development Team
> > Tel. 03-689-7774
> > Fax. 03-689-7230
> > 
> > 
> > 
> > From:   Michel Dagenais <michel.dagenais@polymtl.ca>
> > To:     Amit Margalit/Israel/IBM@IBMIL
> > Cc:     lttng-dev@lists.lttng.org
> > Date:   04/29/2013 04:32 AM
> > Subject:        Re: [lttng-dev] Is there a way to make LTTng use the 
> same 
> > subbuffers for multiple processes?
> > 
> > 
> > 
> > 
> > > Our system is a 12-core x86 (6-core + HT). If I take 4 subbuffers of 
> > 1MB, this means 4MB per
> > > process. As the number of processes rises, this becomes prohibitive. 
> > > For example, we may need to support > 500 sub-processes ==> 500 * 4MB 
> = 
> > 2GB - 
> > > which is a bit too much.  Is there a way to tell LTTng to share the 
> > sub-buffers with forked processes? 
> > 
> > This is the number one feature listed in the changelog for lttng-tools 
> and 
> > UST for version 2.2. The first release candidate is out so you can try 
> the 
> > feature, and the final 2.2 version should not be too far away.
> > 
> > http://lttng.org/lttng-tools-220-rc1
> > - Add UST per UID buffers support
> > 
> 
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
> 
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: Is there a way to make LTTng use the same subbuffers for multiple processes?

[Amit Margalit]

[-- Attachment #1.1: Type: text/plain, Size: 2427 bytes --]

This (current) behavior you describe is very suitable for our needs.

Is there an intention to insert a hook into setuid() in the future?
If yes - will the existing behavior remain as a default and the hook be 
used only if I choose to use it?

Thanks,

Amit Margalit
IBM XIV - Storage Reinvented
XIV-NAS Development Team
Tel. 03-689-7774
Fax. 03-689-7230



From:   Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To:     Amit Margalit/Israel/IBM@IBMIL
Cc:     Michel Dagenais <michel.dagenais@polymtl.ca>, 
lttng-dev@lists.lttng.org
Date:   04/29/2013 07:27 PM
Subject:        Re: [lttng-dev] Is there a way to make LTTng use the same 
subbuffers for multiple processes?



* Amit Margalit (AMITM@il.ibm.com) wrote:
> Another 2.2 question -
> 
> What if I have a single process, that forks, and each child process then 

> changes its own UID. Will the tracing continue on the parent UID 
buffers?

Yes, we indeed currently keep the user ID that was effective at process
creation or last exec() encountered, because we don't have hooks in
libc's setuid() at the moment.

Thanks,

Mathieu

> 
> Thanks,
> 
> Amit Margalit
> IBM XIV - Storage Reinvented
> XIV-NAS Development Team
> Tel. 03-689-7774
> Fax. 03-689-7230
> 
> 
> 
> From:   Michel Dagenais <michel.dagenais@polymtl.ca>
> To:     Amit Margalit/Israel/IBM@IBMIL
> Cc:     lttng-dev@lists.lttng.org
> Date:   04/29/2013 04:32 AM
> Subject:        Re: [lttng-dev] Is there a way to make LTTng use the 
same 
> subbuffers for multiple processes?
> 
> 
> 
> 
> > Our system is a 12-core x86 (6-core + HT). If I take 4 subbuffers of 
> 1MB, this means 4MB per
> > process. As the number of processes rises, this becomes prohibitive. 
> > For example, we may need to support > 500 sub-processes ==> 500 * 4MB 
= 
> 2GB - 
> > which is a bit too much.  Is there a way to tell LTTng to share the 
> sub-buffers with forked processes? 
> 
> This is the number one feature listed in the changelog for lttng-tools 
and 
> UST for version 2.2. The first release candidate is out so you can try 
the 
> feature, and the final 2.2 version should not be too far away.
> 
> http://lttng.org/lttng-tools-220-rc1
> - Add UST per UID buffers support
> 

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com



[-- Attachment #1.2: Type: text/html, Size: 4397 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Is there a way to make LTTng use the same subbuffers for multiple processes?

[Mathieu Desnoyers]

* Amit Margalit (AMITM@il.ibm.com) wrote:
> Another 2.2 question -
> 
> What if I have a single process, that forks, and each child process then 
> changes its own UID. Will the tracing continue on the parent UID buffers?

Yes, we indeed currently keep the user ID that was effective at process
creation or last exec() encountered, because we don't have hooks in
libc's setuid() at the moment.

Thanks,

Mathieu

> 
> Thanks,
> 
> Amit Margalit
> IBM XIV - Storage Reinvented
> XIV-NAS Development Team
> Tel. 03-689-7774
> Fax. 03-689-7230
> 
> 
> 
> From:   Michel Dagenais <michel.dagenais@polymtl.ca>
> To:     Amit Margalit/Israel/IBM@IBMIL
> Cc:     lttng-dev@lists.lttng.org
> Date:   04/29/2013 04:32 AM
> Subject:        Re: [lttng-dev] Is there a way to make LTTng use the same 
> subbuffers for multiple processes?
> 
> 
> 
> 
> > Our system is a 12-core x86 (6-core + HT). If I take 4 subbuffers of 
> 1MB, this means 4MB per
> > process. As the number of processes rises, this becomes prohibitive. 
> > For example, we may need to support > 500 sub-processes ==> 500 * 4MB = 
> 2GB - 
> > which is a bit too much.  Is there a way to tell LTTng to share the 
> sub-buffers with forked processes? 
> 
> This is the number one feature listed in the changelog for lttng-tools and 
> UST for version 2.2. The first release candidate is out so you can try the 
> feature, and the final 2.2 version should not be too far away.
> 
> http://lttng.org/lttng-tools-220-rc1
> - Add UST per UID buffers support
> 

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: Is there a way to make LTTng use the same subbuffers for multiple processes?

[Michel Dagenais]

[-- Attachment #1.1: Type: text/plain, Size: 379 bytes --]


> Another 2.2 question - 
> What if I have a single process, that forks, and each child process then changes its own UID. 
> Will the tracing continue on the parent UID buffers? 

The parent is running as root and the child processes are using seteuid? I am not sure about how this is handled. David or Mathieu can tell you more about the new architecture for sharing buffers. 

[-- Attachment #1.2: Type: text/html, Size: 674 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Is there a way to make LTTng use the same subbuffers for multiple processes?

[Amit Margalit]

[-- Attachment #1.1: Type: text/plain, Size: 1201 bytes --]

Another 2.2 question -

What if I have a single process, that forks, and each child process then 
changes its own UID. Will the tracing continue on the parent UID buffers?

Thanks,

Amit Margalit
IBM XIV - Storage Reinvented
XIV-NAS Development Team
Tel. 03-689-7774
Fax. 03-689-7230



From:   Michel Dagenais <michel.dagenais@polymtl.ca>
To:     Amit Margalit/Israel/IBM@IBMIL
Cc:     lttng-dev@lists.lttng.org
Date:   04/29/2013 04:32 AM
Subject:        Re: [lttng-dev] Is there a way to make LTTng use the same 
subbuffers for multiple processes?




> Our system is a 12-core x86 (6-core + HT). If I take 4 subbuffers of 
1MB, this means 4MB per
> process. As the number of processes rises, this becomes prohibitive. 
> For example, we may need to support > 500 sub-processes ==> 500 * 4MB = 
2GB - 
> which is a bit too much.  Is there a way to tell LTTng to share the 
sub-buffers with forked processes? 

This is the number one feature listed in the changelog for lttng-tools and 
UST for version 2.2. The first release candidate is out so you can try the 
feature, and the final 2.2 version should not be too far away.

http://lttng.org/lttng-tools-220-rc1
- Add UST per UID buffers support


[-- Attachment #1.2: Type: text/html, Size: 2630 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Is there a way to make LTTng use the same subbuffers for multiple processes?

[Amit Margalit]

[-- Attachment #1.1: Type: text/plain, Size: 1071 bytes --]

This is excellent. We'll give it a try. Thank you,

Amit Margalit
IBM XIV - Storage Reinvented
XIV-NAS Development Team
Tel. 03-689-7774
Fax. 03-689-7230



From:   Michel Dagenais <michel.dagenais@polymtl.ca>
To:     Amit Margalit/Israel/IBM@IBMIL
Cc:     lttng-dev@lists.lttng.org
Date:   04/29/2013 04:32 AM
Subject:        Re: [lttng-dev] Is there a way to make LTTng use the same 
subbuffers for multiple processes?




> Our system is a 12-core x86 (6-core + HT). If I take 4 subbuffers of 
1MB, this means 4MB per
> process. As the number of processes rises, this becomes prohibitive. 
> For example, we may need to support > 500 sub-processes ==> 500 * 4MB = 
2GB - 
> which is a bit too much.  Is there a way to tell LTTng to share the 
sub-buffers with forked processes? 

This is the number one feature listed in the changelog for lttng-tools and 
UST for version 2.2. The first release candidate is out so you can try the 
feature, and the final 2.2 version should not be too far away.

http://lttng.org/lttng-tools-220-rc1
- Add UST per UID buffers support


[-- Attachment #1.2: Type: text/html, Size: 2409 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Is there a way to make LTTng use the same subbuffers for multiple processes?

[Michel Dagenais]

[-- Attachment #1.1: Type: text/plain, Size: 645 bytes --]


> Our system is a 12-core x86 (6-core + HT). If I take 4 subbuffers of 1MB, this means 4MB per 
> process. As the number of processes rises, this becomes prohibitive. 
> For example, we may need to support > 500 sub-processes ==> 500 * 4MB = 2GB - 
> which is a bit too much. Is there a way to tell LTTng to share the sub-buffers with forked processes? 

This is the number one feature listed in the changelog for lttng-tools and UST for version 2.2. The first release candidate is out so you can try the feature, and the final 2.2 version should not be too far away. 

http://lttng.org/lttng-tools-220-rc1 
- Add UST per UID buffers support 


[-- Attachment #1.2: Type: text/html, Size: 1088 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev