All of lore.kernel.org
 help / color / mirror / Atom feed
* ROOTFS_RPM_DEBUG undocumented
@ 2018-05-03 10:42 Paulo Neves
  2018-05-03 13:03 ` Alexander Kanavin
  0 siblings, 1 reply; 14+ messages in thread
From: Paulo Neves @ 2018-05-03 10:42 UTC (permalink / raw)
  To: Yocto list discussion

Hello all,

I recently had the problem where the do_rootfs task seemed to hang.
The output where it hang looked like [1] and stayed that way for the
better part of an hour.

When I ran bitbake with -vD arguments not more relevant output was
given [2], also hanging for a lot of time.

As noted in another mailing list post about rootfs hanging[3], the
culprit was the enormous amount of kernel modules which were being
built. I only discovered this because I was able to strace and see
that actually bitbake was not hanging but actually running dnf
install.

When I searched for the code responsible for the invocation of dnf I
notice there is an undocumented variable called ROOTFS_RPM_DEBUG that
enables verbosity of the dnf and prints it out. This would have been
very helpful output for my 'bitbake -vD' invocation, and truly
debugging information would have helped.

What I ask is a decision on which option would better suite this issue:

# Add ROOTFS_RPM_DEBUG to the documentation;
# Detect if we are running with debug output and enable the debugging
output. This is the most elegant solution but I do not know how to
detect if debug log level is turned on;
# Have dnf always print in verbose mode and print the output to
bb.debug instead of bb.note.

I am happy to provide a patch upon decision or suggestions.

Paulo Neves

[1] NOTE: recipe drotag-cloud-image-debug-1.0-r0: task do_rootfs: Started

[2] DEBUG: drotag-cloud-image-debug-1.0-r0 do_rootfs: Executing python
function do_rootfs

[3] https://lists.yoctoproject.org/pipermail/yocto/2017-May/036293.html


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-03 10:42 ROOTFS_RPM_DEBUG undocumented Paulo Neves
@ 2018-05-03 13:03 ` Alexander Kanavin
  2018-05-03 13:37   ` Peter Kjellerstedt
  2018-05-04 12:42   ` Paulo Neves
  0 siblings, 2 replies; 14+ messages in thread
From: Alexander Kanavin @ 2018-05-03 13:03 UTC (permalink / raw)
  To: Paulo Neves, Yocto list discussion

On 05/03/2018 01:42 PM, Paulo Neves wrote:
> # Add ROOTFS_RPM_DEBUG to the documentation;

I'd rather get rid of it, the less variables the better :)

> # Detect if we are running with debug output and enable the debugging
> output. This is the most elegant solution but I do not know how to
> detect if debug log level is turned on;

I don't think you are meant to detect that; bitbake's debug level is not 
meant to affect what is being printed, but merely whether it's printed 
or discarded.

> # Have dnf always print in verbose mode and print the output to
> bb.debug instead of bb.note.

I think this is the best solution actually.

> I am happy to provide a patch upon decision or suggestions.

Thank you :) Do measure the performance before and after as we don't 
want to introduce a significant slowdown here.


Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-03 13:03 ` Alexander Kanavin
@ 2018-05-03 13:37   ` Peter Kjellerstedt
  2018-05-03 14:21     ` Alexander Kanavin
  2018-05-04 12:42   ` Paulo Neves
  1 sibling, 1 reply; 14+ messages in thread
From: Peter Kjellerstedt @ 2018-05-03 13:37 UTC (permalink / raw)
  To: Alexander Kanavin, Paulo Neves, Yocto list discussion

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> bounces@yoctoproject.org] On Behalf Of Alexander Kanavin
> Sent: den 3 maj 2018 15:04
> To: Paulo Neves <ptsneves@gmail.com>; Yocto list discussion
> <yocto@yoctoproject.org>
> Subject: Re: [yocto] ROOTFS_RPM_DEBUG undocumented
> 
> On 05/03/2018 01:42 PM, Paulo Neves wrote:
> > # Add ROOTFS_RPM_DEBUG to the documentation;
> 
> I'd rather get rid of it, the less variables the better :)
> 
> > # Detect if we are running with debug output and enable the debugging
> > output. This is the most elegant solution but I do not know how to
> > detect if debug log level is turned on;
> 
> I don't think you are meant to detect that; bitbake's debug level is
> not
> meant to affect what is being printed, but merely whether it's printed
> or discarded.
> 
> > # Have dnf always print in verbose mode and print the output to
> > bb.debug instead of bb.note.
> 
> I think this is the best solution actually.

I disagree with changing bb.note to bb.debug for this. It is very good 
to be able to look in log.do_rootfs to see what dnf has done (we have 
ROOTFS_RPM_DEBUG set by default). Changing the output to debug level 
would mean having to run bitbake with -D, which is anything but 
desirable.

> > I am happy to provide a patch upon decision or suggestions.
> 
> Thank you :) Do measure the performance before and after as we don't
> want to introduce a significant slowdown here.
> 
> Alex

//Peter



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-03 13:37   ` Peter Kjellerstedt
@ 2018-05-03 14:21     ` Alexander Kanavin
  0 siblings, 0 replies; 14+ messages in thread
From: Alexander Kanavin @ 2018-05-03 14:21 UTC (permalink / raw)
  To: Peter Kjellerstedt, Paulo Neves, Yocto list discussion

On 05/03/2018 04:37 PM, Peter Kjellerstedt wrote:
>>> # Have dnf always print in verbose mode and print the output to
>>> bb.debug instead of bb.note.
>>
>> I think this is the best solution actually.
> 
> I disagree with changing bb.note to bb.debug for this. It is very good
> to be able to look in log.do_rootfs to see what dnf has done (we have
> ROOTFS_RPM_DEBUG set by default). Changing the output to debug level
> would mean having to run bitbake with -D, which is anything but
> desirable.

If I'm reading the bitbake --help right, bb.debug (and bb.note) are 
always written to the log; setting -D level or -v only affects what is 
printed on stdout.


Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-04 12:42   ` Paulo Neves
@ 2018-05-04 12:41     ` Alexander Kanavin
  2018-05-04 12:43       ` Alexander Kanavin
                         ` (3 more replies)
  0 siblings, 4 replies; 14+ messages in thread
From: Alexander Kanavin @ 2018-05-04 12:41 UTC (permalink / raw)
  To: Paulo Neves; +Cc: Yocto list discussion

On 05/04/2018 03:42 PM, Paulo Neves wrote:
> I will propose a patch with a default bb.debug, and always have the
> verbose on dnf. Let's see how it affects the performance. I cannot
> test this in a docker container because of the problems described
> below:
> 
> In the mean time I found what was happening with my slow do_rootfs. I
> was running bitbake with RPM packaging in a docker instance. It seems
> there is a problem with RPM/dnf in bitbake where it is extremely slow
> inside a docker container (1 file per second). After changing to IPK
> packages everything went smoothly.
> 
> Others have had the same problem[1]. I will later report this to
> openembedded mailing list as this effectively makes RPM unusable in a
> container. I will also provide a Dockerfile to reproduce this problem.
> 
> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008

No need to report. We have already fixed this and reported upstream:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch

https://bugzilla.redhat.com/show_bug.cgi?id=1537564


Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-03 13:03 ` Alexander Kanavin
  2018-05-03 13:37   ` Peter Kjellerstedt
@ 2018-05-04 12:42   ` Paulo Neves
  2018-05-04 12:41     ` Alexander Kanavin
  1 sibling, 1 reply; 14+ messages in thread
From: Paulo Neves @ 2018-05-04 12:42 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: Yocto list discussion

I will propose a patch with a default bb.debug, and always have the
verbose on dnf. Let's see how it affects the performance. I cannot
test this in a docker container because of the problems described
below:

In the mean time I found what was happening with my slow do_rootfs. I
was running bitbake with RPM packaging in a docker instance. It seems
there is a problem with RPM/dnf in bitbake where it is extremely slow
inside a docker container (1 file per second). After changing to IPK
packages everything went smoothly.

Others have had the same problem[1]. I will later report this to
openembedded mailing list as this effectively makes RPM unusable in a
container. I will also provide a Dockerfile to reproduce this problem.

[1] https://github.com/moby/moby/issues/23137#issuecomment-359097008

Paulo Neves

On Thu, May 3, 2018 at 3:03 PM, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 05/03/2018 01:42 PM, Paulo Neves wrote:
>>
>> # Add ROOTFS_RPM_DEBUG to the documentation;
>
>
> I'd rather get rid of it, the less variables the better :)
>
>> # Detect if we are running with debug output and enable the debugging
>> output. This is the most elegant solution but I do not know how to
>> detect if debug log level is turned on;
>
>
> I don't think you are meant to detect that; bitbake's debug level is not
> meant to affect what is being printed, but merely whether it's printed or
> discarded.
>
>> # Have dnf always print in verbose mode and print the output to
>> bb.debug instead of bb.note.
>
>
> I think this is the best solution actually.
>
>> I am happy to provide a patch upon decision or suggestions.
>
>
> Thank you :) Do measure the performance before and after as we don't want to
> introduce a significant slowdown here.
>
>
> Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-04 12:41     ` Alexander Kanavin
@ 2018-05-04 12:43       ` Alexander Kanavin
  2018-05-04 12:50       ` Scott Rifenbark
                         ` (2 subsequent siblings)
  3 siblings, 0 replies; 14+ messages in thread
From: Alexander Kanavin @ 2018-05-04 12:43 UTC (permalink / raw)
  To: Paulo Neves; +Cc: Yocto list discussion

On 05/04/2018 03:41 PM, Alexander Kanavin wrote:

> https://bugzilla.redhat.com/show_bug.cgi?id=1537564

Obviously you and anyone else affected need to make noise in the redhat 
bugzilla, otherwise they're likely to keep this low priority.

Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-04 12:50       ` Scott Rifenbark
@ 2018-05-04 12:45         ` Alexander Kanavin
  0 siblings, 0 replies; 14+ messages in thread
From: Alexander Kanavin @ 2018-05-04 12:45 UTC (permalink / raw)
  To: Scott Rifenbark; +Cc: Yocto list discussion

On 05/04/2018 03:50 PM, Scott Rifenbark wrote:
> If ROOTFS_RPM_DEBUG should be a documented variable in the Yocto Project 
> Reference Manual, could someone please provide me with some base 
> explanation of the variable and any usage specifics?

No need, we're going to remove it.

Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-04 12:41     ` Alexander Kanavin
  2018-05-04 12:43       ` Alexander Kanavin
@ 2018-05-04 12:50       ` Scott Rifenbark
  2018-05-04 12:45         ` Alexander Kanavin
  2018-05-04 12:55       ` Paulo Neves
  2018-05-07  9:03       ` Mirza Krak
  3 siblings, 1 reply; 14+ messages in thread
From: Scott Rifenbark @ 2018-05-04 12:50 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: Yocto list discussion

[-- Attachment #1: Type: text/plain, Size: 1695 bytes --]

If ROOTFS_RPM_DEBUG should be a documented variable in the Yocto Project
Reference Manual, could someone please provide me with some base
explanation of the variable and any usage specifics?

Thanks,
Scott

On Fri, May 4, 2018 at 5:41 AM, Alexander Kanavin <
alexander.kanavin@linux.intel.com> wrote:

> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/rec
> ipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opene
> d-files-before-exec-fr.patch
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1537564
>
>
>
> Alex
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 2859 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-04 12:41     ` Alexander Kanavin
  2018-05-04 12:43       ` Alexander Kanavin
  2018-05-04 12:50       ` Scott Rifenbark
@ 2018-05-04 12:55       ` Paulo Neves
  2018-05-07  9:03       ` Mirza Krak
  3 siblings, 0 replies; 14+ messages in thread
From: Paulo Neves @ 2018-05-04 12:55 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: Yocto list discussion

Ouch you are right. Damn me for using pyro still?
After applying this patch I will report on the remaining issues.

On Fri, May 4, 2018 at 2:41 PM, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1537564
>
>
> Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-04 12:41     ` Alexander Kanavin
                         ` (2 preceding siblings ...)
  2018-05-04 12:55       ` Paulo Neves
@ 2018-05-07  9:03       ` Mirza Krak
  2018-05-07 10:12         ` Alexander Kanavin
  3 siblings, 1 reply; 14+ messages in thread
From: Mirza Krak @ 2018-05-07  9:03 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: Yocto list discussion

On 4 May 2018 at 14:41, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch

I have also been bitten by rpm performance issues in docker and glad
that I found this. I am in the process of verifying above (on pyro
branch).

Is this patch on its way to pyro and rocko branches?

-- 
Med Vänliga Hälsningar / Best Regards

Mirza Krak


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-07  9:03       ` Mirza Krak
@ 2018-05-07 10:12         ` Alexander Kanavin
  2018-05-08  9:33           ` Peter Kjellerstedt
  0 siblings, 1 reply; 14+ messages in thread
From: Alexander Kanavin @ 2018-05-07 10:12 UTC (permalink / raw)
  To: Mirza Krak; +Cc: Yocto list discussion

On 05/07/2018 12:03 PM, Mirza Krak wrote:
> I have also been bitten by rpm performance issues in docker and glad
> that I found this. I am in the process of verifying above (on pyro
> branch).
> 
> Is this patch on its way to pyro and rocko branches?

As far as I know, no. You are welcome to backport and submit.

Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-07 10:12         ` Alexander Kanavin
@ 2018-05-08  9:33           ` Peter Kjellerstedt
  2018-05-08 10:29             ` Alexander Kanavin
  0 siblings, 1 reply; 14+ messages in thread
From: Peter Kjellerstedt @ 2018-05-08  9:33 UTC (permalink / raw)
  To: Alexander Kanavin, Mirza Krak; +Cc: Yocto list discussion

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> bounces@yoctoproject.org] On Behalf Of Alexander Kanavin
> Sent: den 7 maj 2018 12:12
> To: Mirza Krak <mirza.krak@gmail.com>
> Cc: Yocto list discussion <yocto@yoctoproject.org>
> Subject: Re: [yocto] ROOTFS_RPM_DEBUG undocumented
> 
> On 05/07/2018 12:03 PM, Mirza Krak wrote:
> > I have also been bitten by rpm performance issues in docker and glad
> > that I found this. I am in the process of verifying above (on pyro
> > branch).
> >
> > Is this patch on its way to pyro and rocko branches?
> 
> As far as I know, no. You are welcome to backport and submit.
> 
> Alex

FYI, the already applied patch only solves half the problem. I have an 
updated version that solves all of it that I am preparing for publication. 
I only need to make some verifications first...

//Peter



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: ROOTFS_RPM_DEBUG undocumented
  2018-05-08  9:33           ` Peter Kjellerstedt
@ 2018-05-08 10:29             ` Alexander Kanavin
  0 siblings, 0 replies; 14+ messages in thread
From: Alexander Kanavin @ 2018-05-08 10:29 UTC (permalink / raw)
  To: Peter Kjellerstedt, Mirza Krak; +Cc: Yocto list discussion

On 05/08/2018 12:33 PM, Peter Kjellerstedt wrote:
> FYI, the already applied patch only solves half the problem. I have an
> updated version that solves all of it that I am preparing for publication.
> I only need to make some verifications first...

I just found that out as well. Lua is not relevant, we don't even enable 
or build it. Anyway, I've been fighting*scratch*convincing upstream all 
morning that fcntl bombs are a real problem under Docker, you are 
welcome to join :)

https://bugzilla.redhat.com/show_bug.cgi?id=1537564

Alex


^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2018-05-08 10:35 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-03 10:42 ROOTFS_RPM_DEBUG undocumented Paulo Neves
2018-05-03 13:03 ` Alexander Kanavin
2018-05-03 13:37   ` Peter Kjellerstedt
2018-05-03 14:21     ` Alexander Kanavin
2018-05-04 12:42   ` Paulo Neves
2018-05-04 12:41     ` Alexander Kanavin
2018-05-04 12:43       ` Alexander Kanavin
2018-05-04 12:50       ` Scott Rifenbark
2018-05-04 12:45         ` Alexander Kanavin
2018-05-04 12:55       ` Paulo Neves
2018-05-07  9:03       ` Mirza Krak
2018-05-07 10:12         ` Alexander Kanavin
2018-05-08  9:33           ` Peter Kjellerstedt
2018-05-08 10:29             ` Alexander Kanavin

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.