Re: Using LTTng for existing Qt-based tracepoints?

Re: Using LTTng for existing Qt-based tracepoints?

[Milian Wolff]

On Donnerstag, 19. Oktober 2017 12:00:01 CEST Arnaud Clère wrote:
> Hi all,
> I am evaluating the feasibility of using LTTng as a backend for existing Qt
> tracepoints following the discussions at Qt Contributors Summit :
> https://wiki.qt.io/QtCS2017_Discuss_Qt_Logging_enhancements . The idea
> would be to get less performance impact when tracing and more structured
> data to analyse without changing Qt users' code.
> 
> I see that instead of describing all tracepoints, we could use a generic LTT
> tracepoint like in Python:
> http://git.lttng.org/?p=lttng-ust.git;a=blob;f=liblttng-ust-python-agent/lt
> tng_ust_python.h
> 
> Except, I would like to structure the tracepoint data at run-time following
> a JSON-based data model. I think I could encode that part of CTF
> dynamically by appending binary user data, CTF variants, dynamically-sized
> CTF structures and sequences in a local buffer.
> 
> But I am wondering how to pass this buffer along with its size to LTTng.
> I am afraid a TP_ARGS(void*, buffer) and TP_FIELDS(buffer) would not work
> since it would require LTTng to read the CTF in "buffer" to discover where
> it stops...
> 
> Or should I use some other API?

Hey Arnaud,

Why do we want to use the generic LTTng tracepoint instead of the concrete 
ones? See also the existing work that went into https://codereview.qt-project.org/#/c/185287/ and the dependent changesets. We hope to be able to 
revive this patchset soon.

Cheers

-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts


_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Using LTTng for existing Qt-based tracepoints?

[Arnaud Clère]

Hi,
I am exploring the feasibility of getting part of LTTng performance and structure for existing qDebug() tracepoints without having to rework them at all.
The idea would be to interpret QDebug::operator<< differently to:
1/ collect static information such as the tracepoint message "format" without having to formalize and "freeze" it in external description files
2/ assemble event data in a binary but self-describing format that could be decoded by generic tools without having to "sync" the trace and tracepoint definitions.

Maybe it is not so useful to device/system developers but I think it would be highly valuable to application developers to get decent performance and trace analysis tools for free (almost).
So, what is this "generic" LTTng tracepoint you were talking about? 

Cheers,
Arnaud

-----Original Message-----
From: milian On Behalf Of Milian Wolff
Sent: vendredi 20 octobre 2017 21:55
To: lttng-dev@lists.lttng.org
Cc: Arnaud Clère <arnaud.clere@minmaxmedical.com>
Subject: Re: [lttng-dev] Using LTTng for existing Qt-based tracepoints?

On Donnerstag, 19. Oktober 2017 12:00:01 CEST Arnaud Clère wrote:
> Hi all,
> I am evaluating the feasibility of using LTTng as a backend for 
> existing Qt tracepoints following the discussions at Qt Contributors Summit :
> https://wiki.qt.io/QtCS2017_Discuss_Qt_Logging_enhancements . The idea 
> would be to get less performance impact when tracing and more 
> structured data to analyse without changing Qt users' code.
> 
> I see that instead of describing all tracepoints, we could use a 
> generic LTT tracepoint like in Python:
> http://git.lttng.org/?p=lttng-ust.git;a=blob;f=liblttng-ust-python-age
> nt/lt
> tng_ust_python.h
> 
> Except, I would like to structure the tracepoint data at run-time 
> following a JSON-based data model. I think I could encode that part of 
> CTF dynamically by appending binary user data, CTF variants, 
> dynamically-sized CTF structures and sequences in a local buffer.
> 
> But I am wondering how to pass this buffer along with its size to LTTng.
> I am afraid a TP_ARGS(void*, buffer) and TP_FIELDS(buffer) would not 
> work since it would require LTTng to read the CTF in "buffer" to 
> discover where it stops...
> 
> Or should I use some other API?

Hey Arnaud,

Why do we want to use the generic LTTng tracepoint instead of the concrete ones? See also the existing work that went into https://codereview.qt-project.org/#/c/185287/ and the dependent changesets. We hope to be able to revive this patchset soon.

Cheers

--
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts


_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Using LTTng for existing Qt-based tracepoints?

[Milian Wolff]

On Montag, 23. Oktober 2017 17:19:47 CEST Arnaud Clère wrote:
> Hi,
> I am exploring the feasibility of getting part of LTTng performance and
> structure for existing qDebug() tracepoints without having to rework them
> at all. The idea would be to interpret QDebug::operator<< differently to:
> 1/ collect static information such as the tracepoint message "format"
> without having to formalize and "freeze" it in external description files
> 2/ assemble event data in a binary but self-describing format that could be
> decoded by generic tools without having to "sync" the trace and tracepoint
> definitions.
> 
> Maybe it is not so useful to device/system developers but I think it would
> be highly valuable to application developers to get decent performance and
> trace analysis tools for free (almost).

Have you ever benchmarked qDebug? The main performance issue isn't so much 
writing the data out, but rather it is assembling the string representation in 
the first place. Replacing the output system with LTTng won't make this 
magically faster.

> So, what is this "generic" LTTng
> tracepoint you were talking about?

You mentioned that one, you said: "we could use a generic LTT tracepoint like 
in Python".

Bye

> 
> -----Original Message-----
> From: milian On Behalf Of Milian Wolff
> Sent: vendredi 20 octobre 2017 21:55
> To: lttng-dev@lists.lttng.org
> Cc: Arnaud Clère <arnaud.clere@minmaxmedical.com>
> Subject: Re: [lttng-dev] Using LTTng for existing Qt-based tracepoints?
> 
> On Donnerstag, 19. Oktober 2017 12:00:01 CEST Arnaud Clère wrote:
> > Hi all,
> > I am evaluating the feasibility of using LTTng as a backend for
> > existing Qt tracepoints following the discussions at Qt Contributors
> > Summit : https://wiki.qt.io/QtCS2017_Discuss_Qt_Logging_enhancements .
> > The idea would be to get less performance impact when tracing and more
> > structured data to analyse without changing Qt users' code.
> > 
> > I see that instead of describing all tracepoints, we could use a
> > generic LTT tracepoint like in Python:
> > http://git.lttng.org/?p=lttng-ust.git;a=blob;f=liblttng-ust-python-age
> > nt/lt
> > tng_ust_python.h
> > 
> > Except, I would like to structure the tracepoint data at run-time
> > following a JSON-based data model. I think I could encode that part of
> > CTF dynamically by appending binary user data, CTF variants,
> > dynamically-sized CTF structures and sequences in a local buffer.
> > 
> > But I am wondering how to pass this buffer along with its size to LTTng.
> > I am afraid a TP_ARGS(void*, buffer) and TP_FIELDS(buffer) would not
> > work since it would require LTTng to read the CTF in "buffer" to
> > discover where it stops...
> > 
> > Or should I use some other API?
> 
> Hey Arnaud,
> 
> Why do we want to use the generic LTTng tracepoint instead of the concrete
> ones? See also the existing work that went into
> https://codereview.qt-project.org/#/c/185287/ and the dependent changesets.
> We hope to be able to revive this patchset soon.
> 
> Cheers
> 
> --
> Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer KDAB
> (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470
> KDAB - The Qt Experts


-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts


_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Using LTTng for existing Qt-based tracepoints?

[Arnaud Clère]

Hi,
So, back to the original question...

I think I can assemble event data from a qDebug() tracepoint in *binary* form using LTTng variants and sequences, thus avoiding numeric formatting and byte array encoding (base64 encoding).
Python uses a generic tracepoint with string data and I assume LTTng uses the '\0' terminated convention to know the string length.
Is there a way to pass a buffer of CTF-encoded binary data to LTTng?

No doubt there are other performance killers in "generic" qDebug statements (Thiago mentioned some at QtCS2017) but since we are evaluating a different implementation from QDebug, I do not want to presume too much about the resulting performance. Also, if more performance is needed by some users for some tracepoints, they will still have other options, and the Qt changeset you mentioned would be one. 

My point is just to provide more choice for Qt users. In my domain, no one will take the time to describe every tracepoint in external files. But we are still interested by a little bit more performance and a lot more structured traces if it just requires to replace all qDebug by qTrace or whatever 😊

Thanks a lot for any help!

Cheers,
Arnaud

-----Original Message-----
From: milian On Behalf Of Milian Wolff
Sent: lundi 23 octobre 2017 20:06
To: Arnaud Clère <arnaud.clere@minmaxmedical.com>
Cc: lttng-dev@lists.lttng.org
Subject: Re: [lttng-dev] Using LTTng for existing Qt-based tracepoints?

On Montag, 23. Oktober 2017 17:19:47 CEST Arnaud Clère wrote:
> Hi,
> I am exploring the feasibility of getting part of LTTng performance 
> and structure for existing qDebug() tracepoints without having to 
> rework them at all. The idea would be to interpret QDebug::operator<< differently to:
> 1/ collect static information such as the tracepoint message "format"
> without having to formalize and "freeze" it in external description 
> files 2/ assemble event data in a binary but self-describing format 
> that could be decoded by generic tools without having to "sync" the 
> trace and tracepoint definitions.
> 
> Maybe it is not so useful to device/system developers but I think it 
> would be highly valuable to application developers to get decent 
> performance and trace analysis tools for free (almost).

Have you ever benchmarked qDebug? The main performance issue isn't so much writing the data out, but rather it is assembling the string representation in the first place. Replacing the output system with LTTng won't make this magically faster.

> So, what is this "generic" LTTng
> tracepoint you were talking about?

You mentioned that one, you said: "we could use a generic LTT tracepoint like in Python".

Bye

> 
> -----Original Message-----
> From: milian On Behalf Of Milian Wolff
> Sent: vendredi 20 octobre 2017 21:55
> To: lttng-dev@lists.lttng.org
> Cc: Arnaud Clère <arnaud.clere@minmaxmedical.com>
> Subject: Re: [lttng-dev] Using LTTng for existing Qt-based tracepoints?
> 
> On Donnerstag, 19. Oktober 2017 12:00:01 CEST Arnaud Clère wrote:
> > Hi all,
> > I am evaluating the feasibility of using LTTng as a backend for 
> > existing Qt tracepoints following the discussions at Qt Contributors 
> > Summit : https://wiki.qt.io/QtCS2017_Discuss_Qt_Logging_enhancements .
> > The idea would be to get less performance impact when tracing and 
> > more structured data to analyse without changing Qt users' code.
> > 
> > I see that instead of describing all tracepoints, we could use a 
> > generic LTT tracepoint like in Python:
> > http://git.lttng.org/?p=lttng-ust.git;a=blob;f=liblttng-ust-python-a
> > ge
> > nt/lt
> > tng_ust_python.h
> > 
> > Except, I would like to structure the tracepoint data at run-time 
> > following a JSON-based data model. I think I could encode that part 
> > of CTF dynamically by appending binary user data, CTF variants, 
> > dynamically-sized CTF structures and sequences in a local buffer.
> > 
> > But I am wondering how to pass this buffer along with its size to LTTng.
> > I am afraid a TP_ARGS(void*, buffer) and TP_FIELDS(buffer) would not 
> > work since it would require LTTng to read the CTF in "buffer" to 
> > discover where it stops...
> > 
> > Or should I use some other API?
> 
> Hey Arnaud,
> 
> Why do we want to use the generic LTTng tracepoint instead of the 
> concrete ones? See also the existing work that went into 
> https://codereview.qt-project.org/#/c/185287/ and the dependent changesets.
> We hope to be able to revive this patchset soon.
> 
> Cheers
> 
> --
> Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer KDAB
> (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470 
> KDAB - The Qt Experts


--
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts


_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Using LTTng for existing Qt-based tracepoints?

[Arnaud Clère]

[-- Attachment #1.1: Type: text/plain, Size: 1141 bytes --]

Hi all,
I am evaluating the feasibility of using LTTng as a backend for existing Qt tracepoints following the discussions at Qt Contributors Summit : https://wiki.qt.io/QtCS2017_Discuss_Qt_Logging_enhancements .
The idea would be to get less performance impact when tracing and more structured data to analyse without changing Qt users' code.

I see that instead of describing all tracepoints, we could use a generic LTT tracepoint like in Python:
http://git.lttng.org/?p=lttng-ust.git;a=blob;f=liblttng-ust-python-agent/lttng_ust_python.h

Except, I would like to structure the tracepoint data at run-time following a JSON-based data model.
I think I could encode that part of CTF dynamically by appending binary user data, CTF variants, dynamically-sized CTF structures and sequences in a local buffer.

But I am wondering how to pass this buffer along with its size to LTTng.
I am afraid a TP_ARGS(void*, buffer) and TP_FIELDS(buffer) would not work since it would require LTTng to read the CTF in "buffer" to discover where it stops...

Or should I use some other API?

Thanks for any advice on how to do that.
Arnaud


[-- Attachment #1.2: Type: text/html, Size: 4275 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev