Re: Making 32-bit user-space events on a 64-bit Linux system

Re: Making 32-bit user-space events on a 64-bit Linux system

[Alexandre Montplaisir]

>> The Debian/Ubuntu packages for liblttng-ust and liburcu are multiarch-enabled, so you can install the :i386 and :amd64 versions in
>> parallel. That way they will be tracked by the package manager, and won't "linger around".
>    Except that the Ubuntu LTTng packages are hopelessly behind right now.  I wanted to do everything with the 2.3.0 release.

The packages from the official repo are often behind, yes. But we have
an Ubuntu PPA which follows the stable releases:
https://launchpad.net/~lttng/+archive/ppa

It's updated daily with the contents of the stable-* git branches.

>    On a different note, why does LTTng have 32- and 64-bit consumers anyway?  The only possible difference between a 32-bit tracer and a 
> 64-bit tracer is that the latter could generate event record fields that are 64-bit longs and floats, while the former cannot (but will never 
> have to, since the application it is tracing won't have such beasts anyway).  So the 64-bit consumer should be backward compatible with the
> 32-bit consumer, because it deals with a super-set of the 32-bit consumer requirements.  If this were the only difference, LTTng would not
> need a separate 32-bit consumer daemon.  Is there some sort of context-switching overhead associated with a 64-bit daemon accessing a
> buffer that exists in a 32-bit process context?  The performance gain would justify the existence of a 32-bit consumer, I suppose.

The consumer shares memory space with the traced process, so they have
to use the same ABI. If they talked only through let's say a pipe, that
wouldn't be necessary, but performance would indeed be much worse.


Cheers,
Alex

Re: Making 32-bit user-space events on a 64-bit Linux system

[Mathieu Desnoyers]

----- Original Message -----
> From: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>
> To: "Daniel Thibault" <Daniel.Thibault@drdc-rddc.gc.ca>, lttng-dev@lists.lttng.org
> Sent: Thursday, October 3, 2013 11:54:15 AM
> Subject: Re: [lttng-dev] Making 32-bit user-space events on a 64-bit Linux system
> 
> >> The Debian/Ubuntu packages for liblttng-ust and liburcu are
> >> multiarch-enabled, so you can install the :i386 and :amd64 versions in
> >> parallel. That way they will be tracked by the package manager, and won't
> >> "linger around".
> >    Except that the Ubuntu LTTng packages are hopelessly behind right now.
> >    I wanted to do everything with the 2.3.0 release.
> 
> The packages from the official repo are often behind, yes. But we have
> an Ubuntu PPA which follows the stable releases:
> https://launchpad.net/~lttng/+archive/ppa
> 
> It's updated daily with the contents of the stable-* git branches.
> 
> >    On a different note, why does LTTng have 32- and 64-bit consumers
> >    anyway?  The only possible difference between a 32-bit tracer and a
> > 64-bit tracer is that the latter could generate event record fields that
> > are 64-bit longs and floats, while the former cannot (but will never
> > have to, since the application it is tracing won't have such beasts
> > anyway).

No, this is not it at all. 32-bit apps can generate 64-bit integers too, and double-precision floats.

The difference between 32 and 64-bit consumers mainly comes for atomic operations: on 64-bit, consumer and application can interact through atomic operations on 64-bit integers, which allow detection of free-running counter overflow much larger than with 32-bit counter.

Thanks,

Mathieu


  So the 64-bit consumer should be backward compatible with the
> > 32-bit consumer, because it deals with a super-set of the 32-bit consumer
> > requirements.  If this were the only difference, LTTng would not
> > need a separate 32-bit consumer daemon.  Is there some sort of
> > context-switching overhead associated with a 64-bit daemon accessing a
> > buffer that exists in a 32-bit process context?  The performance gain would
> > justify the existence of a 32-bit consumer, I suppose.
> 
> The consumer shares memory space with the traced process, so they have
> to use the same ABI. If they talked only through let's say a pipe, that
> wouldn't be necessary, but performance would indeed be much worse.
> 
> 
> Cheers,
> Alex
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: Making 32-bit user-space events on a 64-bit Linux system

[Thibault, Daniel]

-----Message d'origine-----
De : Mathieu Desnoyers [mailto:mathieu.desnoyers@efficios.com] 
Envoyé : 7 octobre 2013 08:57

----- Original Message -----
> From: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>
> Sent: Thursday, October 3, 2013 11:54:15 AM
> 
> >    On a different note, why does LTTng have 32- and 64-bit consumers
> >    anyway?  The only possible difference between a 32-bit tracer and 
> > a 64-bit tracer is that the latter could generate event record 
> > fields that are 64-bit longs and floats, while the former cannot 
> > (but will never have to, since the application it is tracing won't 
> > have such beasts anyway).
>
> No, this is not it at all. 32-bit apps can generate 64-bit integers too, and double-precision floats.

   I realised that soon after having written the above, when I looked up the C spec.  Clearly, the event records are precisely the same in 32- and 64-bit modes.  But if the event buffers don't have intrinsic bitness, something else must be responsible for the requirement for 32-bit consumers.  Does mmap have a bitness barrier?  Or do the event buffers have intrinsic bitness in the sense that the buffer management fields change?

> The difference between 32 and 64-bit consumers mainly comes for atomic operations: on 64-bit,
> consumer and application can interact through atomic operations on 64-bit integers, which allow
> detection of free-running counter overflow much larger than with 32-bit counter.
>
> Mathieu

   If I understand you correctly, the 64-bit atomic operations needed are not available in the 32-bit assembly code.  Makes sense.  One also sees that the dynamic loader has 32- and 64-bit operating modes, due mostly to different ELF formats.  But is there anything in the processor architecture that makes it impossible (or undesirable because of, say, some sort of context-switching) for a 64-bit processor running 32-bit code to insert 64-bit instructions in the code flow?  Alternately, how hard is it for 32-bit code to call a 64-bit library?  (In some ways, this is similar to inserting an external call or a piece of assembly code in high-level language code such as Object Pascal)  Taking yet another step back, what is it about the event buffer lying in a 32-bit process's space that makes it undesirable/difficult/impossible for a 64-bit consumer to access it?
   Obviously, when running on a 32-bit processor LTTng has no choice but to be 32-bit wide throughout, and therefore it cannot use the desirable 64-bit integer atomic comparisons.  Running on a 64-bit processor should be a wholly different kettle of fish, however.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Making 32-bit user-space events on a 64-bit Linux system

[Mathieu Desnoyers]

----- Original Message -----
> From: "Daniel Thibault" <Daniel.Thibault@drdc-rddc.gc.ca>
> To: "Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>, "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>
> Cc: lttng-dev@lists.lttng.org
> Sent: Monday, October 7, 2013 10:40:28 AM
> Subject: RE: [lttng-dev] Making 32-bit user-space events on a 64-bit Linux system
> 
> -----Message d'origine-----
> De : Mathieu Desnoyers [mailto:mathieu.desnoyers@efficios.com]
> Envoyé : 7 octobre 2013 08:57
> 
> ----- Original Message -----
> > From: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>
> > Sent: Thursday, October 3, 2013 11:54:15 AM
> > 
> > >    On a different note, why does LTTng have 32- and 64-bit consumers
> > >    anyway?  The only possible difference between a 32-bit tracer and
> > > a 64-bit tracer is that the latter could generate event record
> > > fields that are 64-bit longs and floats, while the former cannot
> > > (but will never have to, since the application it is tracing won't
> > > have such beasts anyway).
> >
> > No, this is not it at all. 32-bit apps can generate 64-bit integers too,
> > and double-precision floats.
> 
>    I realised that soon after having written the above, when I looked up the
>    C spec.  Clearly, the event records are precisely the same in 32- and
>    64-bit modes.  But if the event buffers don't have intrinsic bitness,
>    something else must be responsible for the requirement for 32-bit
>    consumers.  Does mmap have a bitness barrier?  Or do the event buffers
>    have intrinsic bitness in the sense that the buffer management fields
>    change?
> 
> > The difference between 32 and 64-bit consumers mainly comes for atomic
> > operations: on 64-bit,
> > consumer and application can interact through atomic operations on 64-bit
> > integers, which allow
> > detection of free-running counter overflow much larger than with 32-bit
> > counter.
> >
> > Mathieu
> 
>    If I understand you correctly, the 64-bit atomic operations needed are not
>    available in the 32-bit assembly code.  Makes sense.  One also sees that
>    the dynamic loader has 32- and 64-bit operating modes, due mostly to
>    different ELF formats.  But is there anything in the processor
>    architecture that makes it impossible (or undesirable because of, say,
>    some sort of context-switching) for a 64-bit processor running 32-bit
>    code to insert 64-bit instructions in the code flow?  Alternately, how
>    hard is it for 32-bit code to call a 64-bit library?  (In some ways, this
>    is similar to inserting an external call or a piece of assembly code in
>    high-level language code such as Object Pascal)  Taking yet another step
>    back, what is it about the event buffer lying in a 32-bit process's space
>    that makes it undesirable/difficult/impossible for a 64-bit consumer to
>    access it?
>    Obviously, when running on a 32-bit processor LTTng has no choice but to
>    be 32-bit wide throughout, and therefore it cannot use the desirable
>    64-bit integer atomic comparisons.  Running on a 64-bit processor should
>    be a wholly different kettle of fish, however.

Typically OSes don't allow 32-bit compatibility processes to use the upper register bits (those are cleared upon entry into the OS). So 32-bit processes typically never call a 64-bit library.

We chose to have a consumer daemon that match the traced application ABI bitness so the consumer can link to the liblttng-ust ring buffer implementation, so we don't have to duplicate code across multiple code-bases, and don't have to partially implement pieces of the ring buffer into different code-bases, compiled across different ABIs, talking to each other through shared data structures over shared memory (posix shm). In summary, this decision has been done for simplicity of implementation.

Having a 64-bit consumer that could deal with both 32-bit and 64-bit processes could eventually be done. The main reason why it has not been done originally was that we expected a large change in the UST consumer (it came in the 2.2 release with per-UID UST buffers), which was much easier to do if the ring buffer was sitting in its own library, used by the consumer daemon. We could revisit this eventually if there is sufficient interest for this and funding available.

Thanks,

Mathieu


> 
> Daniel U. Thibault
> Protection des systèmes et contremesures (PSC) | Systems Protection &
> Countermeasures (SPC)
> Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber
> Security (MCCS)
> R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D
> Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Québec QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada | Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Making 32-bit user-space events on a 64-bit Linux system

[Thibault, Daniel]

-----Message d'origine-----
Date: Wed, 2 Oct 2013 19:07:56 +0000
>>    Closing observations:
>>
>>    What's the best, easiest way to get the 32-bit versions of popt and uuid?
>
> Some -dev packages are also multiarch, for example I can install libpopt-dev:amd64 and libpopt-dev:i386 at the same time (on Ubuntu
> 13.10) and get the "real" .so symlink from the package.
>
> It doesn't seem to be the case with uuid-dev though, I can only have one version installed at a time. But you can install one, compile what you
> need, then switch to the other one.
>
> ia32-libs has long been deprecated, it now simply depends on a bunch of :i386 libraries. What's sad is that it used to contain both the 
> libraries and the headers (so the equivalent of the -dev packages) it shipped, whereas now the cross-arch dev files are a bit harder to get.

   Yeah, that situation is an absolute mess right now.  There apparently was a ppa with a 'getlibs' package that did precisely what is needed here: 
bolting 32-bit libraries onto a x86-64 system.  Sadly, the original ppa is gone, and I can only find scattered updates that go no further than Ubuntu 
Natty, apparently.

>>    My lttng-tools-32 make ran into trouble with the 'tests/unit' subdirectory (error below).  Is this an actual bug?
>>
>>    Why did the lttng-tools make not restrict itself to the lttng-consumerd daemon as requested?
>
> These could be bugs in the build system. It's quite an obscure feature, so it's not as well-tested...

   I tried this in a pure 64-bit context and the same error occurred.  I'm filing a bug.

>>    Couldn't the lttng-tools build detect it's running in a 64-bit system and at least try to make a proper 32-bit consumer daemon (and supporting 
>> libraries)?  Maybe this could be a configure switch, off by default.
>
> This is hard to do with the current build system. Autotools basically lets you "build this one thing". If you want to build something 64 bits,
> and then something 32 bits, you have to run 'make' twice. And packaging the result is a nightmare.

   Agreed, not worth the trouble.

>>    In building the 32-bit consumer daemon, could we skip the with-consumerd64-* options?
>
> You can probably skip those. They are used for the opposite case, where you want to compile a 32-bit daemon that will use a 64-bit consumer.

> The Debian/Ubuntu packages for liblttng-ust and liburcu are multiarch-enabled, so you can install the :i386 and :amd64 versions in
> parallel. That way they will be tracked by the package manager, and won't "linger around".

   Except that the Ubuntu LTTng packages are hopelessly behind right now.  I wanted to do everything with the 2.3.0 release.

#################################################

   On a different note, why does LTTng have 32- and 64-bit consumers anyway?  The only possible difference between a 32-bit tracer and a 
64-bit tracer is that the latter could generate event record fields that are 64-bit longs and floats, while the former cannot (but will never 
have to, since the application it is tracing won't have such beasts anyway).  So the 64-bit consumer should be backward compatible with the
32-bit consumer, because it deals with a super-set of the 32-bit consumer requirements.  If this were the only difference, LTTng would not
need a separate 32-bit consumer daemon.  Is there some sort of context-switching overhead associated with a 64-bit daemon accessing a
buffer that exists in a 32-bit process context?  The performance gain would justify the existence of a 32-bit consumer, I suppose.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Re: Making 32-bit user-space events on a 64-bit Linux system

[Alexandre Montplaisir]

Hi Daniel,

Interesting, thanks for documenting your adventure! Some little notes:

On 13-10-02 03:07 PM, Thibault, Daniel wrote:
>    [...]
>
>    Closing observations:
>
>    What's the best, easiest way to get the 32-bit versions of popt and uuid?


Some -dev packages are also multiarch, for example I can install
libpopt-dev:amd64 and libpopt-dev:i386 at the same time (on Ubuntu
13.10) and get the "real" .so symlink from the package.

It doesn't seem to be the case with uuid-dev though, I can only have one
version installed at a time. But you can install one, compile what you
need, then switch to the other one.

ia32-libs has long been deprecated, it now simply depends on a bunch of
:i386 libraries. What's sad is that it used to contain both the
libraries and the headers (so the equivalent of the -dev packages) it
shipped, whereas now the cross-arch dev files are a bit harder to get.

>
>    My lttng-tools-32 make ran into trouble with the 'tests/unit' subdirectory (error below).  Is this an actual bug?
>
>   CC     ../../src/bin/lttng-sessiond/session.o
> ../../src/bin/lttng-sessiond/session.c:249:1: erreur fatale: ouverture du fichier de dépendances .deps/../../src/bin/lttng-sessiond/session.Tpo: Aucun fichier ou dossier de ce type
>
>    Why did the lttng-tools make not restrict itself to the lttng-consumerd daemon as requested?

These could be bugs in the build system. It's quite an obscure feature,
so it's not as well-tested...

>    Couldn't the lttng-tools build detect it's running in a 64-bit system and at least try to make a proper 32-bit consumer daemon (and supporting libraries)?  Maybe this could be a configure switch, off by default.

This is hard to do with the current build system. Autotools basically
lets you "build this one thing". If you want to build something 64 bits,
and then something 32 bits, you have to run 'make' twice. And packaging
the result is a nightmare.

>
>    In building the 32-bit consumer daemon, could we skip the with-consumerd64-* options?

You can probably skip those. They are used for the opposite case, where
you want to compile a 32-bit daemon that will use a 64-bit consumer.

>    How many of the lib32 lttng libraries (apart from liblttng-ust, of course) are deletable?  sample_static32 needs liburcu-bp, liburcu-cds, liblttng-ust-tracepoint in order to run; lttng-consumerd needs liburcu and liblttng-ust-ctl.  That's as far as ldd can detect: there could be dlopen calls lurking in there.  Are liblttng-ctl, liblttng-ust-cyg-profile, liblttng-ust-cyg-profile-fast, liblttng-ust-fork, liblttng-ust-java, liblttng-ust-libc-wrapper, liburcu-common, liburcu-mb, liburcu-qsbr, and liburcu-signal superfluous?

The different liblttng-ust-* libraries are used for different features
(cyg-profile, java JNI interface, fork() tracing). If you want the
feature to be available for 32-bit applications, you need the library.
The liburcu ones that are really unused could be deleted, if you want to
take the risk. They are quite small though, is disk space that much of a
problem?

The Debian/Ubuntu packages for liblttng-ust and liburcu are
multiarch-enabled, so you can install the :i386 and :amd64 versions in
parallel. That way they will be tracked by the package manager, and
won't "linger around".


Hope this helps,
Alexandre

Re: Making 32-bit user-space events on a 64-bit Linux system

[Thibault, Daniel]

   Thanks for the help, I think I've cracked it now.  There are a few minor nagging points, however.  I'll get to those at the end of this post.

   Recap: How does one generate and capture 32-bit user-space events on a 64-bit system?

   The process consists essentially of making a 32-bit lttng-consumerd that the 64-bit lttng-sessiond can use.

   First off, you will need 32-bit development libraries.  This is mostly achieved by installing the gcc-multilib and g++-multilib packages.  The ia32-libs package could also be of use, as are a scattering of lib32* packages, but I did not need them.  I did have a spot of difficulty getting the 32-bit versions of popt and uuid, however (they're not in ia32-libs as of Ubuntu 12.04 Precise).  Synaptic does not make that easy, although command-line 'apt get' may be a better approach.  The important thing is to get /usr/local/lib32 populated with libpopt.so and libuuid.so (in the libuuid case, I installed libuuid1:i386 and then linked from /usr/lib/i386-linux-gnu/libuuid.so to /lib/i386-linux-gnu/libuuid.so.1.3.0, which serves the same purpose).

   Next, deploy the sources for userspace-rcu, lttng-ust and lttng-tools to a 32-bit working directory.  We will build a partial 32-bit LTTng toolchain, being careful not to overwrite the installed 64-bit toolchain.

  The build procedure is as before, except that the ./configure line changes:

userspace-rcu-32:

$ ./bootstrap 2>&1 | tee bootstrap.log
$ ./configure --libdir=/usr/local/lib32 CFLAGS="-m32 -O2" 2>&1 | tee configure.log
$ make 2>&1 | tee make.log
$ sudo make install 2>&1 | tee install.log
$ sudo ldconfig

   Instead of CFLAGS="-m32 -O2", you could use CC="gcc -m32 -O2" and LD="gcc -m32 -O2", which should yield the same results.

lttng-ust-32:

$ ./bootstrap 2>&1 | tee bootstrap.log
$ ./configure LDFLAGS=-L/usr/local/lib32 --libdir=/usr/local/lib32 CFLAGS="-m32 -O2" 2>&1 | tee configure.log
$ make 2>&1 | tee make.log
$ sudo make install 2>&1 | tee install.log
$ sudo ldconfig

   If you have Java installed and/or SystemTap, you may want to add to the ./configure line "--with-jni-interface --with-java-jdk=<path to JDK> --with-sdt", although I suspect this isn't really necessary (we only want a working 32-bit lttng-consumerd, after all).

lttng-tools-32:

$ ./bootstrap 2>&1 | tee bootstrap.log
$ ./configure LDFLAGS=-L/usr/local/lib32 --bindir=/usr/local/bin32 --libdir=/usr/local/lib32 CFLAGS="-m32 -O2" --with-consumerd-only --with-consumerd64-bin=/usr/local/lib/lttng/libexec/lttng-consumerd --with-consumerd64-libdir=/usr/local/lib 2>&1 | tee configure.log
$ make 2>&1 | tee make.log
$ sudo make install 2>&1 | tee install.log
$ sudo ldconfig

   After the configure step, you may need/want to edit the Makefile in order to remove 'tests' from the SUBDIRS (see below for the reason why).
   Note how we ask lttng-tools to make just lttng-consumerd (--with-consumerd-only) and how we hooked the 64-bit consumer in (--with-consumerd64-bin and --with-consumerd64-libdir).

   Finally, we redo the 64-bit lttng-tools to hook in our 32-bit lttng-consumerd:

lttng-tools-64:

$ ./bootstrap 2>&1 | tee bootstrap.log
$ ./configure LDFLAGS=-L/usr/local/lib --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd --with-consumerd32-libdir=/usr/local/lib32 2>&1 | tee configure.log
$ make 2>&1 | tee make.log
$ sudo make install 2>&1 | tee install.log
$ sudo ldconfig

   Henceforth, the lttng-sessiond daemon will run with the correct 32-bit lttng-consumerd.

   Meanwhile, we need a simple source of 32-bit user-space events.  The simplest can be made from the lttng-ust/doc/examples/easy-ust.  Edit the Makefile to add a static32 target:

static32: static32.o tp32.o
        @echo "~~~~~~Linking sample_$@:"
        $(CC) -m32 -o sample_$@ $^ $(LDFLAGS) $(LIBDL) $(LIBUST)
        @echo "   Use 'LD_PRELOAD=/usr/local/lib32/liblttng-ust.so ./sample_$@' to run sample_$@"

static32.o: sample.c tp.h
        @echo "~~~~~~Compiling $@:"
        $(CC) -m32 $(CPPFLAGS) $(LOCAL_CPPFLAGS) $(TP_DEFINE) -c -o $@ $<

tp32.o: tp.c tp.h
        @echo "~~~~~~Compiling $@:"
        $(CC) -m32 $(CPPFLAGS) $(LOCAL_CPPFLAGS) -c -o $@ $<

   From lttng-ust/doc/examples/easy-ust , call:

$ LDFLAGS=-L/usr/local/lib32 make static32

   You can then invoke sample_static32 with:
$ LD_PRELOAD=/usr/local/lib32/liblttng-ust.so ./sample_static32

   Running an lttng session with all user-space events enabled, I did indeed capture ust/uid/1000/32-bit/channel0_0 and ust/uid/1000/64-bit/channel0_0.

   Closing observations:

   What's the best, easiest way to get the 32-bit versions of popt and uuid?

   My lttng-tools-32 make ran into trouble with the 'tests/unit' subdirectory (error below).  Is this an actual bug?

  CC     ../../src/bin/lttng-sessiond/session.o
../../src/bin/lttng-sessiond/session.c:249:1: erreur fatale: ouverture du fichier de dépendances .deps/../../src/bin/lttng-sessiond/session.Tpo: Aucun fichier ou dossier de ce type

   Why did the lttng-tools make not restrict itself to the lttng-consumerd daemon as requested?

   Couldn't the lttng-tools build detect it's running in a 64-bit system and at least try to make a proper 32-bit consumer daemon (and supporting libraries)?  Maybe this could be a configure switch, off by default.

   In building the 32-bit consumer daemon, could we skip the with-consumerd64-* options?
   How many of the lib32 lttng libraries (apart from liblttng-ust, of course) are deletable?  sample_static32 needs liburcu-bp, liburcu-cds, liblttng-ust-tracepoint in order to run; lttng-consumerd needs liburcu and liblttng-ust-ctl.  That's as far as ldd can detect: there could be dlopen calls lurking in there.  Are liblttng-ctl, liblttng-ust-cyg-profile, liblttng-ust-cyg-profile-fast, liblttng-ust-fork, liblttng-ust-java, liblttng-ust-libc-wrapper, liburcu-common, liburcu-mb, liburcu-qsbr, and liburcu-signal superfluous?

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Re: Making 32-bit user-space events on a 64-bit Linux system

[Woegerer, Paul]

ling On 10/02/2013 04:35 PM, Thibault, Daniel wrote:
> -----Message d'origine-----
> De : Woegerer, Paul [mailto:Paul_Woegerer@mentor.com] 
> Envoyé : 2 octobre 2013 03:38
>
>> The 64-bit version of lttng-tools works just fine with both, 32-bit and 64-bit userspace applications. No need to compile it as 32-bit variant.
>> Instrumented 32-bit applications automatically use the 32-bit lttng-consumerd (from the lttng-ust 32-bit build), instrumented 64-bit 
>> applications automatically use the 64-bit lttng-consumerd (from the lttng-ust 64-bit build).
>    But lttng-consumerd is in the lttng-tools package, not lttng-ust ?!  This prevents my lttng-sessiond from using a 32-bit lttng-consumerd, as you'll see below.

Right. Now that you say it, I DO remember compiling lttng-tools twice. I
took only the lttng-consumerd from the 32-bit variant and copied it over
into the 64-bit install tree and recompiled for 64-bit with
--with-consumerd32-bin=<lttng_install_root>/lib/lttng/libexec/lttng-consumerd
--with-consumerd64-bin=<lttng_install_root>/lib64/lttng/libexec/lttng-consumerd.
I think, this did the trick (if I remember correctly).

--
Paul

>
>> BTW, when you compile for the "-m32" multilib, it's better to use the following:
>> $ ./configure CC="gcc -m32"
>> This way also the linking steps (performed by libtool) of your build will know that you are building for m32 (on your x86_64 system) and you don't need e.g. "--libdir ....".
>    That's what I did, but I still felt I needed to specify libdir so the install did not overwrite the 64-bit libraries (liblttng-ctl.so, liblttng-ust.so, etc.).
>
>> For more information see:
>> http://doc.opensuse.org/documentation/html/openSUSE/opensuse-reference/cha.64bit.html#sec.64bit.biarch
>> or
>> https://developer.mozilla.org/en/docs/Compiling_32-bit_Firefox_on_a_Linux_64-bit_OS
>    I have seen the second one, which was moderately useul.  I'll look into the first one, thanks.
>
>>> More generally, how does one add tracepoints to a 32-bit application running on a 64-bit system?
>> You simply compile your instrumented application to 32-bit the same way you did it for the lttng-ust package (CC="gcc -m32").
>>
>> Paul Woegerer, SW Development Engineer
>    Indeed, once lttng-ust has been done in 32-bits, I can use the 'make static32' I quoted at the beginning of the thread like this:
>
> $ LDFLAGS=-L/usr/local/lib32 make static 32
>
>    And then run the sample app like this:
>
> $ LD_PRELOAD=/usr/local/lib32/liblttng-ust.so ./sample_static32
>
>    (I could dump the LD_PRELOAD by adding /usr/local/lib32 to my PATH)
>
>    But my running lttng-sessiond (enable-event -u -a) complains:
>
> Error: Consumer socket fd -1 not found in consumer obj 0x7ffd4000b650
>
>    My guess is I'll need to launch lttng-sessiond with the -u and -U options to point it to the 32-bit lttng-consumerd...Once I can get the 32-bit lttng-tools to compile and install it.
>
> Daniel U. Thibault
> Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
> Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
> R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Québec QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada | Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>


-- 
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

Re: Making 32-bit user-space events on a 64-bit Linux system

[Thibault, Daniel]

-----Message d'origine-----
De : Woegerer, Paul [mailto:Paul_Woegerer@mentor.com] 
Envoyé : 2 octobre 2013 03:38

> The 64-bit version of lttng-tools works just fine with both, 32-bit and 64-bit userspace applications. No need to compile it as 32-bit variant.
> Instrumented 32-bit applications automatically use the 32-bit lttng-consumerd (from the lttng-ust 32-bit build), instrumented 64-bit 
> applications automatically use the 64-bit lttng-consumerd (from the lttng-ust 64-bit build).

   But lttng-consumerd is in the lttng-tools package, not lttng-ust ?!  This prevents my lttng-sessiond from using a 32-bit lttng-consumerd, as you'll see below.

> BTW, when you compile for the "-m32" multilib, it's better to use the following:
> $ ./configure CC="gcc -m32"
> This way also the linking steps (performed by libtool) of your build will know that you are building for m32 (on your x86_64 system) and you don't need e.g. "--libdir ....".

   That's what I did, but I still felt I needed to specify libdir so the install did not overwrite the 64-bit libraries (liblttng-ctl.so, liblttng-ust.so, etc.).

> For more information see:
> http://doc.opensuse.org/documentation/html/openSUSE/opensuse-reference/cha.64bit.html#sec.64bit.biarch
> or
> https://developer.mozilla.org/en/docs/Compiling_32-bit_Firefox_on_a_Linux_64-bit_OS

   I have seen the second one, which was moderately useul.  I'll look into the first one, thanks.

> > More generally, how does one add tracepoints to a 32-bit application running on a 64-bit system?
>
> You simply compile your instrumented application to 32-bit the same way you did it for the lttng-ust package (CC="gcc -m32").
>
> Paul Woegerer, SW Development Engineer

   Indeed, once lttng-ust has been done in 32-bits, I can use the 'make static32' I quoted at the beginning of the thread like this:

$ LDFLAGS=-L/usr/local/lib32 make static 32

   And then run the sample app like this:

$ LD_PRELOAD=/usr/local/lib32/liblttng-ust.so ./sample_static32

   (I could dump the LD_PRELOAD by adding /usr/local/lib32 to my PATH)

   But my running lttng-sessiond (enable-event -u -a) complains:

Error: Consumer socket fd -1 not found in consumer obj 0x7ffd4000b650

   My guess is I'll need to launch lttng-sessiond with the -u and -U options to point it to the 32-bit lttng-consumerd...Once I can get the 32-bit lttng-tools to compile and install it.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Re: Making 32-bit user-space events on a 64-bit Linux system

[Woegerer, Paul]

Hi Daniel, Hi Jeremie

I tried this on my x86_64 system some time ago (with 2.2.0-rc2) and it
worked just fine.

The only thing that was needed was to build 32-bit variants of urcu and
lttng-ust (and also having all dependend libs as 32-bit variants on the
system).

The 64-bit version of lttng-tools works just fine with both, 32-bit and
64-bit userspace applications. No need to compile it as 32-bit variant.

Instrumented 32-bit applications automatically use the 32-bit
lttng-consumerd (from the lttng-ust 32-bit build),
instrumented 64-bit applications automatically use the 64-bit
lttng-consumerd (from the lttng-ust 64-bit build).

BTW, when you compile for the "-m32" multilib, it's better to use the
following:

$ ./configure CC="gcc -m32"

This way also the linking steps (performed by libtool) of your build
will know that you are building for m32 (on your x86_64 system) and you
don't need e.g. "--libdir ....".
For more information see:
http://doc.opensuse.org/documentation/html/openSUSE/opensuse-reference/cha.64bit.html#sec.64bit.biarch
or
https://developer.mozilla.org/en/docs/Compiling_32-bit_Firefox_on_a_Linux_64-bit_OS

> More generally, how does one add tracepoints to a 32-bit application running on a 64-bit system?

You simply compile your instrumented application to 32-bit the same way
you did it for the lttng-ust package (CC="gcc -m32").


Best,
Paul

-- 
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

Re: Making 32-bit user-space events on a 64-bit Linux system

[Thibault, Daniel]

-----Message d'origine-----
De : jeremie.galarneau@gmail.com [mailto:jeremie.galarneau@gmail.com]
Envoyé : 30 septembre 2013 18:34

>>    The problem is the "incompatible /usr/local/lib/liblttng-ust.so".  I even tried rebuilding the entire lttng suite (originally built *before* gcc-multilib was added), but that changed nothing.  How do I do this?  More generally, how does one add tracepoints to a 32-bit application running on a 64-bit system?
>
> The complete LTTng toolchain must be built for 32-bit (including the dependencies). Please refer to your distribution's documentation on how to build or install 32-bit packages.

   I'm assuming the lttng-modules are to be excluded from the 32-bit build (since I'm concerned with just 32-bit user-space events anyway).  I manage to rebuild userspace-rcu and lttng-ust with no apparent trouble, changing just the ./configure line in each case:

userspace-rcu-32bits:
$ ./configure --libdir=/usr/local/lib32 CFLAGS="-m32 -g -O2"
$ make
$ sudo make install
$ sudo ldconfig

lttng-ust-32bits:

$ ./configure LDFLAGS=-L/usr/local/lib32 --with-jni-interface --with-java-jdk=`./doc/examples/drdc/findjdk` --with-sdt --libdir=/usr/local/lib32 CFLAGS="-m32 -g -O2"
$ make
$ sudo make install
$ sudo ldconfig

lttng-tools is more challenging.  I figure I'll need to build just the 32-bit lttng-consumerd, then rebuild the 64-bit lttng-tools with --with-consumerd32-libdir and --with-consumerd32-bin properly set:

lttng-tools-32bits:
$ ./configure LDFLAGS=-L/usr/local/lib32 --enable-python-bindings --bindir=/usr/local/bin32 --libdir=/usr/local/lib32 CFLAGS="-m32 -g -O2" --with-consumerd-only
$ make
$ sudo make install
$ sudo ldconfig

lttng-tools-64bits:
$ ./configure LDFLAGS=-L/usr/local/lib --enable-python-bindings --with-consumerd32-bin=/usr/local/bin32/lttng/lttng-consumerd --with-consumerd32-libdir=/usr/local/lib32
$ make
$ sudo make install
$ sudo ldconfig

Haven't tried this yet.  Am I on the right track to get 64-bit lttng-tools to capture 32-bit user-space events?

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Making 32-bit user-space events on a 64-bit Linux system

[Thibault, Daniel]

   I wanted to test the 32-bit UST event stream (to see if one can indeed have two consumer daemons serving the session, one for 64-bit user-space events, another for 32-bit user-space events), so I adapted the lttng-ust/doc/examples/easy-ust Makefile by making a new target:

static32: static32.o tp32.o
	@echo "~~~~~~Linking sample_$@:"
	$(CC) -m32 -o sample_$@ $^ $(LDFLAGS) $(LIBDL) $(LIBUST)
	@echo "   Use './sample_$@' to run sample_$@"

static32.o: sample.c tp.h
	@echo "~~~~~~Compiling $@:"
	$(CC) -m32 $(CPPFLAGS) $(LOCAL_CPPFLAGS) $(TP_DEFINE) -c -o $@ $<

tp32.o: tp.c tp.h
	@echo "~~~~~~Compiling $@:"
	$(CC) -m32 $(CPPFLAGS) $(LOCAL_CPPFLAGS) -c -o $@ $<

   Simply put, the .c files are compiled and linked with an additional -m32 gcc option.  The gcc-multilib package is installed.

   However, I get:

$ make static32
~~~~~~Compiling static32.o:
gcc -m32  -I. -D TRACEPOINT_DEFINE -c -o static32.o sample.c
~~~~~~Compiling tp32.o:
gcc -m32  -I. -c -o tp32.o tp.c
~~~~~~Linking sample_static32:
gcc -m32 -o sample_static32 static32.o tp32.o -L/usr/local/lib -ldl  -llttng-ust
/usr/bin/ld: warning: skipping incompatible /usr/local/lib/liblttng-ust.so while searching for lttng-ust
/usr/bin/ld: error: cannot find -llttng-ust
tp32.o:tp.c:function __lttng_events_init__sample_component: error: undefined reference to 'lttng_probe_register'
tp32.o:tp.c:function __lttng_events_exit__sample_component: error: undefined reference to 'lttng_probe_unregister'
collect2: ld a retourné 1 code d'état d'exécution
make: *** [static32] Erreur 1

   The problem is the "incompatible /usr/local/lib/liblttng-ust.so".  I even tried rebuilding the entire lttng suite (originally built *before* gcc-multilib was added), but that changed nothing.  How do I do this?  More generally, how does one add tracepoints to a 32-bit application running on a 64-bit system?

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>