ROOTFS_RPM_DEBUG undocumented

ROOTFS_RPM_DEBUG undocumented

[Paulo Neves]

Hello all,

I recently had the problem where the do_rootfs task seemed to hang.
The output where it hang looked like [1] and stayed that way for the
better part of an hour.

When I ran bitbake with -vD arguments not more relevant output was
given [2], also hanging for a lot of time.

As noted in another mailing list post about rootfs hanging[3], the
culprit was the enormous amount of kernel modules which were being
built. I only discovered this because I was able to strace and see
that actually bitbake was not hanging but actually running dnf
install.

When I searched for the code responsible for the invocation of dnf I
notice there is an undocumented variable called ROOTFS_RPM_DEBUG that
enables verbosity of the dnf and prints it out. This would have been
very helpful output for my 'bitbake -vD' invocation, and truly
debugging information would have helped.

What I ask is a decision on which option would better suite this issue:

# Add ROOTFS_RPM_DEBUG to the documentation;
# Detect if we are running with debug output and enable the debugging
output. This is the most elegant solution but I do not know how to
detect if debug log level is turned on;
# Have dnf always print in verbose mode and print the output to
bb.debug instead of bb.note.

I am happy to provide a patch upon decision or suggestions.

Paulo Neves

[1] NOTE: recipe drotag-cloud-image-debug-1.0-r0: task do_rootfs: Started

[2] DEBUG: drotag-cloud-image-debug-1.0-r0 do_rootfs: Executing python
function do_rootfs

[3] https://lists.yoctoproject.org/pipermail/yocto/2017-May/036293.html

Re: ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/03/2018 01:42 PM, Paulo Neves wrote:
> # Add ROOTFS_RPM_DEBUG to the documentation;

I'd rather get rid of it, the less variables the better :)

> # Detect if we are running with debug output and enable the debugging
> output. This is the most elegant solution but I do not know how to
> detect if debug log level is turned on;

I don't think you are meant to detect that; bitbake's debug level is not 
meant to affect what is being printed, but merely whether it's printed 
or discarded.

> # Have dnf always print in verbose mode and print the output to
> bb.debug instead of bb.note.

I think this is the best solution actually.

> I am happy to provide a patch upon decision or suggestions.

Thank you :) Do measure the performance before and after as we don't 
want to introduce a significant slowdown here.


Alex

Re: ROOTFS_RPM_DEBUG undocumented

[Peter Kjellerstedt]

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> bounces@yoctoproject.org] On Behalf Of Alexander Kanavin
> Sent: den 3 maj 2018 15:04
> To: Paulo Neves <ptsneves@gmail.com>; Yocto list discussion
> <yocto@yoctoproject.org>
> Subject: Re: [yocto] ROOTFS_RPM_DEBUG undocumented
> 
> On 05/03/2018 01:42 PM, Paulo Neves wrote:
> > # Add ROOTFS_RPM_DEBUG to the documentation;
> 
> I'd rather get rid of it, the less variables the better :)
> 
> > # Detect if we are running with debug output and enable the debugging
> > output. This is the most elegant solution but I do not know how to
> > detect if debug log level is turned on;
> 
> I don't think you are meant to detect that; bitbake's debug level is
> not
> meant to affect what is being printed, but merely whether it's printed
> or discarded.
> 
> > # Have dnf always print in verbose mode and print the output to
> > bb.debug instead of bb.note.
> 
> I think this is the best solution actually.

I disagree with changing bb.note to bb.debug for this. It is very good 
to be able to look in log.do_rootfs to see what dnf has done (we have 
ROOTFS_RPM_DEBUG set by default). Changing the output to debug level 
would mean having to run bitbake with -D, which is anything but 
desirable.

> > I am happy to provide a patch upon decision or suggestions.
> 
> Thank you :) Do measure the performance before and after as we don't
> want to introduce a significant slowdown here.
> 
> Alex

//Peter

Re: ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/03/2018 04:37 PM, Peter Kjellerstedt wrote:
>>> # Have dnf always print in verbose mode and print the output to
>>> bb.debug instead of bb.note.
>>
>> I think this is the best solution actually.
> 
> I disagree with changing bb.note to bb.debug for this. It is very good
> to be able to look in log.do_rootfs to see what dnf has done (we have
> ROOTFS_RPM_DEBUG set by default). Changing the output to debug level
> would mean having to run bitbake with -D, which is anything but
> desirable.

If I'm reading the bitbake --help right, bb.debug (and bb.note) are 
always written to the log; setting -D level or -v only affects what is 
printed on stdout.


Alex

Re: ROOTFS_RPM_DEBUG undocumented

[Paulo Neves]

I will propose a patch with a default bb.debug, and always have the
verbose on dnf. Let's see how it affects the performance. I cannot
test this in a docker container because of the problems described
below:

In the mean time I found what was happening with my slow do_rootfs. I
was running bitbake with RPM packaging in a docker instance. It seems
there is a problem with RPM/dnf in bitbake where it is extremely slow
inside a docker container (1 file per second). After changing to IPK
packages everything went smoothly.

Others have had the same problem[1]. I will later report this to
openembedded mailing list as this effectively makes RPM unusable in a
container. I will also provide a Dockerfile to reproduce this problem.

[1] https://github.com/moby/moby/issues/23137#issuecomment-359097008

Paulo Neves

On Thu, May 3, 2018 at 3:03 PM, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 05/03/2018 01:42 PM, Paulo Neves wrote:
>>
>> # Add ROOTFS_RPM_DEBUG to the documentation;
>
>
> I'd rather get rid of it, the less variables the better :)
>
>> # Detect if we are running with debug output and enable the debugging
>> output. This is the most elegant solution but I do not know how to
>> detect if debug log level is turned on;
>
>
> I don't think you are meant to detect that; bitbake's debug level is not
> meant to affect what is being printed, but merely whether it's printed or
> discarded.
>
>> # Have dnf always print in verbose mode and print the output to
>> bb.debug instead of bb.note.
>
>
> I think this is the best solution actually.
>
>> I am happy to provide a patch upon decision or suggestions.
>
>
> Thank you :) Do measure the performance before and after as we don't want to
> introduce a significant slowdown here.
>
>
> Alex

Re: ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/04/2018 03:42 PM, Paulo Neves wrote:
> I will propose a patch with a default bb.debug, and always have the
> verbose on dnf. Let's see how it affects the performance. I cannot
> test this in a docker container because of the problems described
> below:
> 
> In the mean time I found what was happening with my slow do_rootfs. I
> was running bitbake with RPM packaging in a docker instance. It seems
> there is a problem with RPM/dnf in bitbake where it is extremely slow
> inside a docker container (1 file per second). After changing to IPK
> packages everything went smoothly.
> 
> Others have had the same problem[1]. I will later report this to
> openembedded mailing list as this effectively makes RPM unusable in a
> container. I will also provide a Dockerfile to reproduce this problem.
> 
> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008

No need to report. We have already fixed this and reported upstream:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch

https://bugzilla.redhat.com/show_bug.cgi?id=1537564


Alex

Re: ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/04/2018 03:41 PM, Alexander Kanavin wrote:

> https://bugzilla.redhat.com/show_bug.cgi?id=1537564

Obviously you and anyone else affected need to make noise in the redhat 
bugzilla, otherwise they're likely to keep this low priority.

Alex

Re: ROOTFS_RPM_DEBUG undocumented

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 1695 bytes --]

If ROOTFS_RPM_DEBUG should be a documented variable in the Yocto Project
Reference Manual, could someone please provide me with some base
explanation of the variable and any usage specifics?

Thanks,
Scott

On Fri, May 4, 2018 at 5:41 AM, Alexander Kanavin <
alexander.kanavin@linux.intel.com> wrote:

> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/rec
> ipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opene
> d-files-before-exec-fr.patch
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1537564
>
>
>
> Alex
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 2859 bytes --]

Re: ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/04/2018 03:50 PM, Scott Rifenbark wrote:
> If ROOTFS_RPM_DEBUG should be a documented variable in the Yocto Project 
> Reference Manual, could someone please provide me with some base 
> explanation of the variable and any usage specifics?

No need, we're going to remove it.

Alex

Re: ROOTFS_RPM_DEBUG undocumented

[Paulo Neves]

Ouch you are right. Damn me for using pyro still?
After applying this patch I will report on the remaining issues.

On Fri, May 4, 2018 at 2:41 PM, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1537564
>
>
> Alex

Re: ROOTFS_RPM_DEBUG undocumented

[Mirza Krak]

On 4 May 2018 at 14:41, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch

I have also been bitten by rpm performance issues in docker and glad
that I found this. I am in the process of verifying above (on pyro
branch).

Is this patch on its way to pyro and rocko branches?

-- 
Med Vänliga Hälsningar / Best Regards

Mirza Krak

Re: ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/07/2018 12:03 PM, Mirza Krak wrote:
> I have also been bitten by rpm performance issues in docker and glad
> that I found this. I am in the process of verifying above (on pyro
> branch).
> 
> Is this patch on its way to pyro and rocko branches?

As far as I know, no. You are welcome to backport and submit.

Alex

Re: ROOTFS_RPM_DEBUG undocumented

[Peter Kjellerstedt]

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> bounces@yoctoproject.org] On Behalf Of Alexander Kanavin
> Sent: den 7 maj 2018 12:12
> To: Mirza Krak <mirza.krak@gmail.com>
> Cc: Yocto list discussion <yocto@yoctoproject.org>
> Subject: Re: [yocto] ROOTFS_RPM_DEBUG undocumented
> 
> On 05/07/2018 12:03 PM, Mirza Krak wrote:
> > I have also been bitten by rpm performance issues in docker and glad
> > that I found this. I am in the process of verifying above (on pyro
> > branch).
> >
> > Is this patch on its way to pyro and rocko branches?
> 
> As far as I know, no. You are welcome to backport and submit.
> 
> Alex

FYI, the already applied patch only solves half the problem. I have an 
updated version that solves all of it that I am preparing for publication. 
I only need to make some verifications first...

//Peter

Re: ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/08/2018 12:33 PM, Peter Kjellerstedt wrote:
> FYI, the already applied patch only solves half the problem. I have an
> updated version that solves all of it that I am preparing for publication.
> I only need to make some verifications first...

I just found that out as well. Lua is not relevant, we don't even enable 
or build it. Anyway, I've been fighting*scratch*convincing upstream all 
morning that fcntl bombs are a real problem under Docker, you are 
welcome to join :)

https://bugzilla.redhat.com/show_bug.cgi?id=1537564

Alex