[PATCH] systemtap: 3.3 -> 4.0

[PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky]

Upgrade systemtap from 3.3 to 4.0: Removed backported patch.

Very short summary of major changes from SystemTap 4.0
announcement by Frank Ch. Eigler <fche@redhat.com>:

> prometheus exporter network service; ebpf support extensions including
> strings and implementation of traditional log(), sprintf() functions;
> rebuilt rich tapset coverage for 4.17+ syscalls and for
> tracepoint-based syscalls; script language tweaks for supporting
> machine-generated scripts

Fixes [YOCTO #12950]

Signed-off-by: Victor Kamensky <kamensky@cisco.com>
---
Hi Guys,

I'll let you to decide whether you want to accept
it for 2.6 at this point or push it for the next release.
On one hand after move to 4.18 kernel SystemTap 3.3
fails to trace any system call on x86_64 - it is major
functionality breakage. Look at:

https://bugzilla.redhat.com/show_bug.cgi?id=1596456

On other hand it is quite late to do major version
updates.

I did go through limitted set of tests on all
supported QEMU targets. As far as these tests
concerned proposed 4.0 matches 3.3 results.

Thanks,
Victor

 ...entrypc-avoid-usage-of-uninitialized.patch | 46 -------------------
 .../systemtap/systemtap_git.inc               |  5 +-
 2 files changed, 2 insertions(+), 49 deletions(-)
 delete mode 100644 meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch

diff --git a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch b/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
deleted file mode 100644
index d0082a1094..0000000000
--- a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
+++ /dev/null
@@ -1,46 +0,0 @@
-From 8466fca2a074323a235ef38d425f994a2ff7e64f Mon Sep 17 00:00:00 2001
-From: Victor Kamensky <kamensky@cisco.com>
-Date: Mon, 9 Jul 2018 09:31:19 -0700
-Subject: [PATCH] dwflpp::function_entrypc avoid usage of uninitialized memory
-
-Failure on 3.3 release was observed. Failure was elusive and
-disappeared after seemingly random configure option change, or when
-code was compiled with -O1 or -O0 (vs default -O2). Running failing
-test case under valgrind memcheck pointed to couple places where
-'Conditional jump or move depends on uninitialised value(s)' occured.
-
-After addressing these in two places in dwflpp::function_entrypc,
-valgrind memcheck run is clean and original issue got fixed.
-
-Upstream-Status: Backport
-Signed-off-by: Victor Kamensky <kamensky@cisco.com>
----
- dwflpp.cxx | 6 +++++-
- 1 file changed, 5 insertions(+), 1 deletion(-)
-
-diff --git a/dwflpp.cxx b/dwflpp.cxx
-index bfbb6b096..2172e705a 100644
---- a/dwflpp.cxx
-+++ b/dwflpp.cxx
-@@ -2465,13 +2465,17 @@ bool
- dwflpp::function_entrypc (Dwarf_Addr * addr)
- {
-   assert (function);
-+
-+  // assign default value
-+  *addr = 0;
-+
-   // PR10574: reject 0, which tends to be eliminated COMDAT
-   if (dwarf_entrypc (function, addr) == 0 && *addr != 0)
-     return true;
- 
-   /* Assume the entry pc is the base address, or (if zero)
-      the first address of the ranges covering this DIE.  */
--  Dwarf_Addr start, end;
-+  Dwarf_Addr start = 0, end;
-   if (dwarf_ranges (function, 0, addr, &start, &end) >= 0)
-     {
-       if (*addr == 0)
--- 
-2.17.1
-
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc b/meta/recipes-kernel/systemtap/systemtap_git.inc
index 06924fc240..274fcde5c1 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -1,7 +1,7 @@
 LICENSE = "GPLv2"
 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
-SRCREV = "48867d1cface9445d58e3c6e14497770b7eb77e1"
-PV = "3.3"
+SRCREV = "428f84e9e656bce71018e8902e4edb8aacafcc0e"
+PV = "4.0"
 
 SRC_URI = "git://sourceware.org/git/systemtap.git \
            file://configure-allow-to-disable-libvirt.patch \
@@ -11,7 +11,6 @@ SRC_URI = "git://sourceware.org/git/systemtap.git \
            file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
            file://0001-Install-python-modules-to-correct-library-dir.patch \
            file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
-           file://0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch \
            "
 
 COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips).*-linux'
-- 
2.17.1

Re: [PATCH] systemtap: 3.3 -> 4.0

[Randy MacLeod]

On 10/15/2018 02:21 PM, Victor Kamensky wrote:
> Upgrade systemtap from 3.3 to 4.0: Removed backported patch.
> 
> Very short summary of major changes from SystemTap 4.0
> announcement by Frank Ch. Eigler <fche@redhat.com>:
> 
>> prometheus exporter network service; ebpf support extensions including
>> strings and implementation of traditional log(), sprintf() functions;
>> rebuilt rich tapset coverage for 4.17+ syscalls and for
>> tracepoint-based syscalls; script language tweaks for supporting
>> machine-generated scripts
> 
> Fixes [YOCTO #12950]
> 
> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> ---
> Hi Guys,
> 
> I'll let you to decide whether you want to accept
> it for 2.6 at this point or push it for the next release.
> On one hand after move to 4.18 kernel SystemTap 3.3
> fails to trace any system call on x86_64 - it is major
> functionality breakage. Look at:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1596456

Ouch!

> 
> On other hand it is quite late to do major version
> updates.

It is but nothing depends on systemtap so I think we
don't have much choice and we should do it unless there
are autobuilder tests that regress.


A quick read through the YP mega-manual suggests that
we don't have to change any of the text.
Victor,
Could you take a look and confirm that please?

I noticed that the link in section:
    20.3.3. Documentation
for:
    http://sourceware.org/systemtap/langref/
is dead. There is a pdf:
    http://sourceware.org/systemtap/langref.pdf
Perhaps this is temporary due to the recent release.

> 
> I did go through limited set of tests on all
> supported QEMU targets. As far as these tests
> concerned proposed 4.0 matches 3.3 results.
> 
> Thanks,
> Victor

Thanks Victor.

Talk about hot off the presses, 4.0 was released at:
    13-Oct-2018 18:54

$ git log --oneline release-3.3..release-4.0 | wc -l
252

$ git diff release-3.3..release-4.0 | diffstat | tail -1
  1574 files changed, 50756 insertions(+), 23604 deletions(-)

Lots of changes to docs, testsuite but also quite a bit to core code
and tapsets:

$ git diff release-3.3..release-4.0 | diffstat  | cut -d"/" -f-2 | uniq 
-c |grep -v "   [1-5] "
     770  b/doc
      16  b/httpd
       8  b/man
       9  b/po
      18  b/runtime
      15  b/stap-exporter
       6  b/staprun
     440  b/tapset
     223  b/testsuite


../Randy

> 
>   ...entrypc-avoid-usage-of-uninitialized.patch | 46 -------------------
>   .../systemtap/systemtap_git.inc               |  5 +-
>   2 files changed, 2 insertions(+), 49 deletions(-)
>   delete mode 100644 meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
> 
> diff --git a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch b/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
> deleted file mode 100644
> index d0082a1094..0000000000
> --- a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
> +++ /dev/null
> @@ -1,46 +0,0 @@
> -From 8466fca2a074323a235ef38d425f994a2ff7e64f Mon Sep 17 00:00:00 2001
> -From: Victor Kamensky <kamensky@cisco.com>
> -Date: Mon, 9 Jul 2018 09:31:19 -0700
> -Subject: [PATCH] dwflpp::function_entrypc avoid usage of uninitialized memory
> -
> -Failure on 3.3 release was observed. Failure was elusive and
> -disappeared after seemingly random configure option change, or when
> -code was compiled with -O1 or -O0 (vs default -O2). Running failing
> -test case under valgrind memcheck pointed to couple places where
> -'Conditional jump or move depends on uninitialised value(s)' occured.
> -
> -After addressing these in two places in dwflpp::function_entrypc,
> -valgrind memcheck run is clean and original issue got fixed.
> -
> -Upstream-Status: Backport
> -Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> ----
> - dwflpp.cxx | 6 +++++-
> - 1 file changed, 5 insertions(+), 1 deletion(-)
> -
> -diff --git a/dwflpp.cxx b/dwflpp.cxx
> -index bfbb6b096..2172e705a 100644
> ---- a/dwflpp.cxx
> -+++ b/dwflpp.cxx
> -@@ -2465,13 +2465,17 @@ bool
> - dwflpp::function_entrypc (Dwarf_Addr * addr)
> - {
> -   assert (function);
> -+
> -+  // assign default value
> -+  *addr = 0;
> -+
> -   // PR10574: reject 0, which tends to be eliminated COMDAT
> -   if (dwarf_entrypc (function, addr) == 0 && *addr != 0)
> -     return true;
> -
> -   /* Assume the entry pc is the base address, or (if zero)
> -      the first address of the ranges covering this DIE.  */
> --  Dwarf_Addr start, end;
> -+  Dwarf_Addr start = 0, end;
> -   if (dwarf_ranges (function, 0, addr, &start, &end) >= 0)
> -     {
> -       if (*addr == 0)
> ---
> -2.17.1
> -
> diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc b/meta/recipes-kernel/systemtap/systemtap_git.inc
> index 06924fc240..274fcde5c1 100644
> --- a/meta/recipes-kernel/systemtap/systemtap_git.inc
> +++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
> @@ -1,7 +1,7 @@
>   LICENSE = "GPLv2"
>   LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
> -SRCREV = "48867d1cface9445d58e3c6e14497770b7eb77e1"
> -PV = "3.3"
> +SRCREV = "428f84e9e656bce71018e8902e4edb8aacafcc0e"
> +PV = "4.0"
>   
>   SRC_URI = "git://sourceware.org/git/systemtap.git \
>              file://configure-allow-to-disable-libvirt.patch \
> @@ -11,7 +11,6 @@ SRC_URI = "git://sourceware.org/git/systemtap.git \
>              file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
>              file://0001-Install-python-modules-to-correct-library-dir.patch \
>              file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch \
> -           file://0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch \
>              "
>   
>   COMPATIBLE_HOST = '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips).*-linux'
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky]

Hi Randy,

On Mon, 15 Oct 2018, Randy MacLeod wrote:

> On 10/15/2018 02:21 PM, Victor Kamensky wrote:
>> Upgrade systemtap from 3.3 to 4.0: Removed backported patch.
>> 
>> Very short summary of major changes from SystemTap 4.0
>> announcement by Frank Ch. Eigler <fche@redhat.com>:
>> 
>>> prometheus exporter network service; ebpf support extensions including
>>> strings and implementation of traditional log(), sprintf() functions;
>>> rebuilt rich tapset coverage for 4.17+ syscalls and for
>>> tracepoint-based syscalls; script language tweaks for supporting
>>> machine-generated scripts
>> 
>> Fixes [YOCTO #12950]
>> 
>> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
>> ---
>> Hi Guys,
>> 
>> I'll let you to decide whether you want to accept
>> it for 2.6 at this point or push it for the next release.
>> On one hand after move to 4.18 kernel SystemTap 3.3
>> fails to trace any system call on x86_64 - it is major
>> functionality breakage. Look at:
>> 
>> https://bugzilla.redhat.com/show_bug.cgi?id=1596456
>
> Ouch!
>
>> 
>> On other hand it is quite late to do major version
>> updates.
>
> It is but nothing depends on systemtap so I think we
> don't have much choice and we should do it unless there
> are autobuilder tests that regress.

Personally, I agree. I had similar thoughts, since it
is leaf project risk is quite contained.

>
> A quick read through the YP mega-manual suggests that
> we don't have to change any of the text.
> Victor,
> Could you take a look and confirm that please?
>
> I noticed that the link in section:
>   20.3.3. Documentation
> for:
>   http://sourceware.org/systemtap/langref/
> is dead. There is a pdf:
>   http://sourceware.org/systemtap/langref.pdf
> Perhaps this is temporary due to the recent release.

Yes, I agree. It is better to change link to langref.pdf
as you suggested. But I don't know how to do that. Is there
somewhere git that holds YP mega-manual sources?

>> 
>> I did go through limited set of tests on all
>> supported QEMU targets. As far as these tests
>> concerned proposed 4.0 matches 3.3 results.
>> 
>> Thanks,
>> Victor
>
> Thanks Victor.
>
> Talk about hot off the presses, 4.0 was released at:
>   13-Oct-2018 18:54

Yes :), I was looking how to fix [YOCTO #12950] and was
about to write email to SystemTap guys asking when the
next release is planned and got email from Frank on
SystemTap mailing list announcing 4.0 release. It was
very good timing.

My understanding is that FC28 and FC27 will get SystemTap
4.0 very soon since they moved to 4.18 kernel too.

Thanks,
Victor

> $ git log --oneline release-3.3..release-4.0 | wc -l
> 252
>
> $ git diff release-3.3..release-4.0 | diffstat | tail -1
> 1574 files changed, 50756 insertions(+), 23604 deletions(-)
>
> Lots of changes to docs, testsuite but also quite a bit to core code
> and tapsets:
>
> $ git diff release-3.3..release-4.0 | diffstat  | cut -d"/" -f-2 | uniq -c 
> |grep -v "   [1-5] "
>    770  b/doc
>     16  b/httpd
>      8  b/man
>      9  b/po
>     18  b/runtime
>     15  b/stap-exporter
>      6  b/staprun
>    440  b/tapset
>    223  b/testsuite
>
>
> ../Randy
>
>>
>>   ...entrypc-avoid-usage-of-uninitialized.patch | 46 -------------------
>>   .../systemtap/systemtap_git.inc               |  5 +-
>>   2 files changed, 2 insertions(+), 49 deletions(-)
>>   delete mode 100644 
>> meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
>> 
>> diff --git 
>> a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch 
>> b/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
>> deleted file mode 100644
>> index d0082a1094..0000000000
>> --- 
>> a/meta/recipes-kernel/systemtap/systemtap/0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch
>> +++ /dev/null
>> @@ -1,46 +0,0 @@
>> -From 8466fca2a074323a235ef38d425f994a2ff7e64f Mon Sep 17 00:00:00 2001
>> -From: Victor Kamensky <kamensky@cisco.com>
>> -Date: Mon, 9 Jul 2018 09:31:19 -0700
>> -Subject: [PATCH] dwflpp::function_entrypc avoid usage of uninitialized 
>> memory
>> -
>> -Failure on 3.3 release was observed. Failure was elusive and
>> -disappeared after seemingly random configure option change, or when
>> -code was compiled with -O1 or -O0 (vs default -O2). Running failing
>> -test case under valgrind memcheck pointed to couple places where
>> -'Conditional jump or move depends on uninitialised value(s)' occured.
>> -
>> -After addressing these in two places in dwflpp::function_entrypc,
>> -valgrind memcheck run is clean and original issue got fixed.
>> -
>> -Upstream-Status: Backport
>> -Signed-off-by: Victor Kamensky <kamensky@cisco.com>
>> ----
>> - dwflpp.cxx | 6 +++++-
>> - 1 file changed, 5 insertions(+), 1 deletion(-)
>> -
>> -diff --git a/dwflpp.cxx b/dwflpp.cxx
>> -index bfbb6b096..2172e705a 100644
>> ---- a/dwflpp.cxx
>> -+++ b/dwflpp.cxx
>> -@@ -2465,13 +2465,17 @@ bool
>> - dwflpp::function_entrypc (Dwarf_Addr * addr)
>> - {
>> -   assert (function);
>> -+
>> -+  // assign default value
>> -+  *addr = 0;
>> -+
>> -   // PR10574: reject 0, which tends to be eliminated COMDAT
>> -   if (dwarf_entrypc (function, addr) == 0 && *addr != 0)
>> -     return true;
>> -
>> -   /* Assume the entry pc is the base address, or (if zero)
>> -      the first address of the ranges covering this DIE.  */
>> --  Dwarf_Addr start, end;
>> -+  Dwarf_Addr start = 0, end;
>> -   if (dwarf_ranges (function, 0, addr, &start, &end) >= 0)
>> -     {
>> -       if (*addr == 0)
>> ---
>> -2.17.1
>> -
>> diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
>> b/meta/recipes-kernel/systemtap/systemtap_git.inc
>> index 06924fc240..274fcde5c1 100644
>> --- a/meta/recipes-kernel/systemtap/systemtap_git.inc
>> +++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
>> @@ -1,7 +1,7 @@
>>   LICENSE = "GPLv2"
>>   LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
>> -SRCREV = "48867d1cface9445d58e3c6e14497770b7eb77e1"
>> -PV = "3.3"
>> +SRCREV = "428f84e9e656bce71018e8902e4edb8aacafcc0e"
>> +PV = "4.0"
>>     SRC_URI = "git://sourceware.org/git/systemtap.git \
>>              file://configure-allow-to-disable-libvirt.patch \
>> @@ -11,7 +11,6 @@ SRC_URI = "git://sourceware.org/git/systemtap.git \
>>              file://0001-Do-not-let-configure-write-a-python-location-into-th.patch 
>> \
>>              file://0001-Install-python-modules-to-correct-library-dir.patch 
>> \
>>              file://0001-staprun-stapbpf-don-t-support-installing-a-non-root.patch 
>> \
>> - 
>> file://0001-dwflpp-function_entrypc-avoid-usage-of-uninitialized.patch \
>>              "
>>     COMPATIBLE_HOST = 
>> '(x86_64|i.86|powerpc|arm|aarch64|microblazeel|mips).*-linux'
>> 
>
>
> -- 
> # Randy MacLeod
> # Wind River Linux
>

Re: [PATCH] systemtap: 3.3 -> 4.0

[Burton, Ross]

On Mon, 15 Oct 2018 at 19:22, Victor Kamensky <kamensky@cisco.com> wrote:
> Upgrade systemtap from 3.3 to 4.0: Removed backported patch.

WARNING: systemtap-4.0-r0 do_package_qa: QA Issue: systemtap:
/systemtap/etc/stap-exporter/EXAMPLE is owned by uid 1000, which is
the same as the user running bitbake. This may be due to host
contamination
systemtap: /systemtap/etc/stap-exporter/EXAMPLE_NOERROR is owned by
uid 1000, which is the same as the user running bitbake. This may be
due to host contamination
systemtap: /systemtap/etc/stap-exporter/errno.stp is owned by uid
1000, which is the same as the user running bitbake. This may be due
to host contamination
systemtap: /systemtap/etc/stap-exporter/syscalllatency.stp is owned by
uid 1000, which is the same as the user running bitbake. This may be
due to host contamination
systemtap: /systemtap/etc/stap-exporter/timeout.stp is owned by uid
1000, which is the same as the user running bitbake. This may be due
to host contamination
systemtap: /systemtap/etc/stap-exporter/topsys.stp is owned by uid
1000, which is the same as the user running bitbake. This may be due
to host contamination [host-user-contaminated]

Re: [PATCH] systemtap: 3.3 -> 4.0

[Richard Purdie]

On Tue, 2018-10-16 at 10:27 +0100, Burton, Ross wrote:
> On Mon, 15 Oct 2018 at 19:22, Victor Kamensky <kamensky@cisco.com>
> wrote:
> > Upgrade systemtap from 3.3 to 4.0: Removed backported patch.
> 
> WARNING: systemtap-4.0-r0 do_package_qa: QA Issue: systemtap:
> /systemtap/etc/stap-exporter/EXAMPLE is owned by uid 1000, which is
> the same as the user running bitbake. This may be due to host
> contamination
> systemtap: /systemtap/etc/stap-exporter/EXAMPLE_NOERROR is owned by
> uid 1000, which is the same as the user running bitbake. This may be
> due to host contamination
> systemtap: /systemtap/etc/stap-exporter/errno.stp is owned by uid
> 1000, which is the same as the user running bitbake. This may be due
> to host contamination
> systemtap: /systemtap/etc/stap-exporter/syscalllatency.stp is owned
> by
> uid 1000, which is the same as the user running bitbake. This may be
> due to host contamination
> systemtap: /systemtap/etc/stap-exporter/timeout.stp is owned by uid
> 1000, which is the same as the user running bitbake. This may be due
> to host contamination
> systemtap: /systemtap/etc/stap-exporter/topsys.stp is owned by uid
> 1000, which is the same as the user running bitbake. This may be due
> to host contamination [host-user-contaminated]

There are also other issues from the addition of systemd unit files in
4.0. I've a patch in -next which tweaks both issues which I'll put in
for testing.

Cheers,

Richard

Re: [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky]

On Tue, 16 Oct 2018, Richard Purdie wrote:

> On Tue, 2018-10-16 at 10:27 +0100, Burton, Ross wrote:
>> On Mon, 15 Oct 2018 at 19:22, Victor Kamensky <kamensky@cisco.com>
>> wrote:
>>> Upgrade systemtap from 3.3 to 4.0: Removed backported patch.
>>
>> WARNING: systemtap-4.0-r0 do_package_qa: QA Issue: systemtap:
>> /systemtap/etc/stap-exporter/EXAMPLE is owned by uid 1000, which is
>> the same as the user running bitbake. This may be due to host
>> contamination
>> systemtap: /systemtap/etc/stap-exporter/EXAMPLE_NOERROR is owned by
>> uid 1000, which is the same as the user running bitbake. This may be
>> due to host contamination
>> systemtap: /systemtap/etc/stap-exporter/errno.stp is owned by uid
>> 1000, which is the same as the user running bitbake. This may be due
>> to host contamination
>> systemtap: /systemtap/etc/stap-exporter/syscalllatency.stp is owned
>> by
>> uid 1000, which is the same as the user running bitbake. This may be
>> due to host contamination
>> systemtap: /systemtap/etc/stap-exporter/timeout.stp is owned by uid
>> 1000, which is the same as the user running bitbake. This may be due
>> to host contamination
>> systemtap: /systemtap/etc/stap-exporter/topsys.stp is owned by uid
>> 1000, which is the same as the user running bitbake. This may be due
>> to host contamination [host-user-contaminated]
>
> There are also other issues from the addition of systemd unit files in
> 4.0. I've a patch in -next which tweaks both issues which I'll put in
> for testing.

Richard, thank you very prompt help.

In the long term it would be beneficial that those issues be
fixed in SystemTap itself, correct? If so, I can look at creating
proper patches and pushing them into SystemTap.

Also on my todo list is to attempt to push few existing
OE systemtap patches into SystemTap upstream source. I believe
they look like quite legit issues and I hope they could be
upstreamed.

Thanks,
Victor

> Cheers,
>
> Richard
>
>

Re: [PATCH] systemtap: 3.3 -> 4.0

[richard.purdie]

On Tue, 2018-10-16 at 22:56 -0700, Victor Kamensky wrote:
> In the long term it would be beneficial that those issues be
> fixed in SystemTap itself, correct? If so, I can look at creating
> proper patches and pushing them into SystemTap.

Yes, fixing those upstream would be very helpful, probably through
adding the ability to configure the systemd unit path and fixing the
file installation method.

> Also on my todo list is to attempt to push few existing
> OE systemtap patches into SystemTap upstream source. I believe
> they look like quite legit issues and I hope they could be
> upstreamed.

That kind of work is very much appreciated and makes everyone's life
easier in the future!

Cheers,

Richard

Re: [PATCH] systemtap: 3.3 -> 4.0

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1604 bytes --]

There seems to be one more issue in current master (for which I haven't
seen a fix in master-next or ML):

ERROR: systemtap-4.0-r0 do_package: QA Issue: systemtap:
Files/directories were installed but not shipped in any package:
  /lib
  /lib/systemd
  /lib/systemd/system
  /lib/systemd/system/stap-exporter.service
Please set FILES such that these items are packaged. Alternatively if
they are unneeded, avoid installing them or delete them within
do_install.
systemtap: 4 installed and not shipped files. [installed-vs-shipped]



On Wed, Oct 17, 2018 at 1:59 PM <richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2018-10-16 at 22:56 -0700, Victor Kamensky wrote:
> > In the long term it would be beneficial that those issues be
> > fixed in SystemTap itself, correct? If so, I can look at creating
> > proper patches and pushing them into SystemTap.
>
> Yes, fixing those upstream would be very helpful, probably through
> adding the ability to configure the systemd unit path and fixing the
> file installation method.
>
> > Also on my todo list is to attempt to push few existing
> > OE systemtap patches into SystemTap upstream source. I believe
> > they look like quite legit issues and I hope they could be
> > upstreamed.
>
> That kind of work is very much appreciated and makes everyone's life
> easier in the future!
>
> Cheers,
>
> Richard
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 2409 bytes --]

Re: [PATCH] systemtap: 3.3 -> 4.0

[richard.purdie]

On Wed, 2018-10-17 at 16:50 +0200, Martin Jansa wrote:
> There seems to be one more issue in current master (for which I
> haven't seen a fix in master-next or ML):
> 
> ERROR: systemtap-4.0-r0 do_package: QA Issue: systemtap:
> Files/directories were installed but not shipped in any package:
>   /lib
>   /lib/systemd
>   /lib/systemd/system
>   /lib/systemd/system/stap-exporter.service
> Please set FILES such that these items are packaged. Alternatively if
> they are unneeded, avoid installing them or delete them within
> do_install.
> systemtap: 4 installed and not shipped files. [installed-vs-shipped]

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3b77e7b7852549dcfbc426d4ce258e6e857c0acd 
was supposed to fix that :(

Cheers,

Richard

Re: [PATCH] systemtap: 3.3 -> 4.0

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

systemd.bbclass won't include that .service file in FILES_<pkg> without
setting SYSTEMD_SERVICE_<pkg>.

I'll send fix for that in few minutes.

On Wed, Oct 17, 2018 at 5:07 PM <richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2018-10-17 at 16:50 +0200, Martin Jansa wrote:
> > There seems to be one more issue in current master (for which I
> > haven't seen a fix in master-next or ML):
> >
> > ERROR: systemtap-4.0-r0 do_package: QA Issue: systemtap:
> > Files/directories were installed but not shipped in any package:
> >   /lib
> >   /lib/systemd
> >   /lib/systemd/system
> >   /lib/systemd/system/stap-exporter.service
> > Please set FILES such that these items are packaged. Alternatively if
> > they are unneeded, avoid installing them or delete them within
> > do_install.
> > systemtap: 4 installed and not shipped files. [installed-vs-shipped]
>
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3b77e7b7852549dcfbc426d4ce258e6e857c0acd
> was supposed to fix that :(
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 1578 bytes --]

[PATCH] systemtap: fix QA issue

[Martin Jansa]

ERROR: systemtap-4.0-r0 do_package: QA Issue: systemtap:
Files/directories were installed but not shipped in any package:
  /lib
  /lib/systemd
  /lib/systemd/system
  /lib/systemd/system/stap-exporter.service
Please set FILES such that these items are packaged. Alternatively if
they are unneeded, avoid installing them or delete them within
do_install.
systemtap: 4 installed and not shipped files. [installed-vs-shipped]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 3c45bfa937..88942a2362 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -27,6 +27,8 @@ PACKAGECONFIG[python3-probes] = "--with-python3-probes,--without-python3-probes,
 
 inherit autotools gettext pkgconfig distutils3-base systemd
 
+SYSTEMD_SERVICE_${PN} = "stap-exporter.service"
+
 do_configure_prepend () {
     # Improve reproducibility for c++ object files
     reltivepath="${@os.path.relpath(d.getVar('STAGING_INCDIR'), d.getVar('S'))}"
-- 
2.17.1

Re: [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1676 bytes --]



On Wed, 17 Oct 2018, Martin Jansa wrote:

> systemd.bbclass won't include that .service file in FILES_<pkg> without setting SYSTEMD_SERVICE_<pkg>.
> I'll send fix for that in few minutes.

Martin, Richard, thank you for the help. I've reproduced
the issue reported by Martin in systemd enabled image, and
verified that his fix solves the issue.

The lesson I learnt:

o Look for WARNINGs

o Test systemd enabled image, along with default sysvinit

stap-exporter service is a new feature in SystemTap 4.0, I've never used
it before. I'll dig in and will try to make sure that it is functional
in resulting OE image.

Thanks,
Victor

> On Wed, Oct 17, 2018 at 5:07 PM <richard.purdie@linuxfoundation.org> wrote:
>       On Wed, 2018-10-17 at 16:50 +0200, Martin Jansa wrote:
>       > There seems to be one more issue in current master (for which I
>       > haven't seen a fix in master-next or ML):
>       >
>       > ERROR: systemtap-4.0-r0 do_package: QA Issue: systemtap:
>       > Files/directories were installed but not shipped in any package:
>       >   /lib
>       >   /lib/systemd
>       >   /lib/systemd/system
>       >   /lib/systemd/system/stap-exporter.service
>       > Please set FILES such that these items are packaged. Alternatively if
>       > they are unneeded, avoid installing them or delete them within
>       > do_install.
>       > systemtap: 4 installed and not shipped files. [installed-vs-shipped]
>
>       http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3b77e7b7852549dcfbc426d4ce258e6e857c0acd
>       was supposed to fix that :(
>
>       Cheers,
>
>       Richard
> 
> 
>

Re: [PATCH] systemtap: 3.3 -> 4.0

[Victor Kamensky]

Hi Guys,

On Wed, 17 Oct 2018, Victor Kamensky wrote:

> stap-exporter service is a new feature in SystemTap 4.0, I've never used
> it before. I'll dig in and will try to make sure that it is functional
> in resulting OE image.

I've had a chance to look at stap-exporter and its service more closely.

It looks like it is simple HTTP server that allows to run stap scripts
on target and relay its information to Prometheus monitoring system.
Please check.

https://www.mankier.com/8/stap-exporter

https://prometheus.io

Since it is HTTP server, and its man page says that it is not supposed
to run in untrusted environment I am quite hesitant to have it enabled
by default. Moreover, it comes with some default example scripts that
actually fail to compile in OE core-image-lsb-sdk because it does not
include kernel-vmlinux package by default. stap-exporter is not
prepared to handle script compilation failure and tries to compile it
in the loop, so in core-image-lsb-sdk systemd image where 
stap-exporter.service would be enabled by default it would trash the
system.

Considering all that I suggest stap-exporter to be moved in separate
systemtap-exporter package, so it can be installed explcitely. I
posted patch for it at

http://lists.openembedded.org/pipermail/openembedded-core/2018-October/157073.html

BTW in FC it is separate systemtap-exporter package.

I've tested systemtap-exporter and stap-exporter, if installed
along with proper prerequisites (kernel-vmlinux) is functioning
OK with given default example as far as my manual http testing
with curl shows. I did not test it with prometheus.

Thanks,
Victor