Re: linux-next build error (8)

Re: linux-next build error (8)

[Dmitry Vyukov]

On Fri, May 22, 2020 at 6:29 AM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> Hello.
>
> This report is already reporting next problem. Since the location seems to be
> different, this might be caused by OOM due to too much parallel compilation.
> Maybe syzbot can detect "gcc: fatal error: Killed signal terminated program cc1"
> sequence and retry with reduced "make -j$NUM" settings.
>
>   gcc: fatal error: Killed signal terminated program cc1
>   compilation terminated.
>   scripts/Makefile.build:272: recipe for target 'fs/xfs/libxfs/xfs_btree.o' failed
>   make[2]: *** [fs/xfs/libxfs/xfs_btree.o] Error 1
>   make[2]: *** Waiting for unfinished jobs....

+linux-next and XFS maintainers

Interesting. This seems to repeat reliably and this machine is not
known for any misbehavior and it always happens on all XFS files.
Did XFS get something that crashes gcc's?

  CC      drivers/hid/hid-uclogic-rdesc.o
  CC      drivers/hid/hid-uclogic-params.o
  CC      drivers/hid/hid-led.o
  CC      drivers/hid/hid-zpff.o
  CC      drivers/hid/hid-zydacron.o
  CC      drivers/hid/wacom_wac.o
  CC      drivers/hid/wacom_sys.o
  CC      drivers/hid/hid-waltop.o
  CC      drivers/hid/hid-wiimote-core.o
  CC      drivers/hid/hid-wiimote-debug.o
  CC      drivers/hid/hid-wiimote-modules.o
  CC      drivers/hid/usbhid/hid-core.o
  CC      drivers/hid/usbhid/hid-pidff.o
  CC      drivers/hid/usbhid/hiddev.o
  AR      drivers/gpu/drm/i915/built-in.a
  AR      drivers/gpu/drm/built-in.a
  AR      drivers/gpu/built-in.a
gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
scripts/Makefile.build:272: recipe for target 'fs/xfs/xfs_file.o' failed
make[2]: *** [fs/xfs/xfs_file.o] Error 1
  AR      drivers/hid/usbhid/built-in.a
gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
scripts/Makefile.build:272: recipe for target 'fs/xfs/xfs_fsmap.o' failed
make[2]: *** [fs/xfs/xfs_fsmap.o] Error 1
  AR      drivers/hid/built-in.a
  AR      drivers/built-in.a
gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
scripts/Makefile.build:272: recipe for target 'fs/xfs/xfs_filestream.o' failed
make[2]: *** [fs/xfs/xfs_filestream.o] Error 1
gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
scripts/Makefile.build:272: recipe for target 'fs/xfs/xfs_icache.o' failed
make[2]: *** [fs/xfs/xfs_icache.o] Error 1
  AR      fs/ocfs2/built-in.a
gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
scripts/Makefile.build:272: recipe for target 'fs/xfs/xfs_iops.o' failed
make[2]: *** [fs/xfs/xfs_iops.o] Error 1
gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
scripts/Makefile.build:272: recipe for target 'fs/xfs/xfs_mount.o' failed
make[2]: *** [fs/xfs/xfs_mount.o] Error 1
gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
scripts/Makefile.build:272: recipe for target 'fs/xfs/xfs_inode.o' failed
make[2]: *** [fs/xfs/xfs_inode.o] Error 1
scripts/Makefile.build:494: recipe for target 'fs/xfs' failed
make[1]: *** [fs/xfs] Error 2
Makefile:1736: recipe for target 'fs' failed
make: *** [fs] Error 2

Re: linux-next build error (8)

[Qian Cai]

> On May 26, 2020, at 8:09 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> 
> +linux-next and XFS maintainers
> 
> Interesting. This seems to repeat reliably and this machine is not
> known for any misbehavior and it always happens on all XFS files.
> Did XFS get something that crashes gcc's?

Are you still seeing this in today’s linux-next? There was an issue had already been sorted out.

Re: linux-next build error (8)

[Dmitry Vyukov]

On Tue, May 26, 2020 at 2:19 PM Qian Cai <cai@lca.pw> wrote:

> > On May 26, 2020, at 8:09 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > +linux-next and XFS maintainers
> >
> > Interesting. This seems to repeat reliably and this machine is not
> > known for any misbehavior and it always happens on all XFS files.
> > Did XFS get something that crashes gcc's?
>
> Are you still seeing this in today’s linux-next? There was an issue had already been sorted out.

syzbot seen this on these commits/dates:

https://syzkaller.appspot.com/bug?extid=792dec47d693ccdc05a0
Crashes (4):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro
ci-upstream-linux-next-kasan-gce-root 2020/05/22 01:23 linux-next
e8f32747 5afa2ddd .config log report
ci-upstream-linux-next-kasan-gce-root 2020/05/21 15:01 linux-next
e8f32747 1f30020f .config log report
ci-upstream-linux-next-kasan-gce-root 2020/05/19 18:24 linux-next
fb57b1fa 6d882fd2 .config log report
ci-upstream-linux-next-kasan-gce-root 2020/03/18 16:19 linux-next
47780d78 0a96a13c .config log report

Re: linux-next build error (8)

[Qian Cai]

> On May 26, 2020, at 8:28 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> 
> Crashes (4):
> Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro
> ci-upstream-linux-next-kasan-gce-root 2020/05/22 01:23 linux-next
> e8f32747 5afa2ddd .config log report
> ci-upstream-linux-next-kasan-gce-root 2020/05/21 15:01 linux-next
> e8f32747 1f30020f .config log report
> ci-upstream-linux-next-kasan-gce-root 2020/05/19 18:24 linux-next
> fb57b1fa 6d882fd2 .config log report
> ci-upstream-linux-next-kasan-gce-root 2020/03/18 16:19 linux-next
> 47780d78 0a96a13c .config log report

You’ll probably need to use an known good kernel version. For example, a stock kernel or any of a mainline -rc / GA kernel to compile next-20200526 and then test from there.

Re: linux-next build error (8)

[Tetsuo Handa]

On 2020/05/26 21:41, Qian Cai wrote:
> 
> 
>> On May 26, 2020, at 8:28 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
>>
>> Crashes (4):
>> Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro
>> ci-upstream-linux-next-kasan-gce-root 2020/05/22 01:23 linux-next
>> e8f32747 5afa2ddd .config log report
>> ci-upstream-linux-next-kasan-gce-root 2020/05/21 15:01 linux-next
>> e8f32747 1f30020f .config log report
>> ci-upstream-linux-next-kasan-gce-root 2020/05/19 18:24 linux-next
>> fb57b1fa 6d882fd2 .config log report
>> ci-upstream-linux-next-kasan-gce-root 2020/03/18 16:19 linux-next
>> 47780d78 0a96a13c .config log report
> 
> You’ll probably need to use an known good kernel version. For example, a stock kernel or any of a mainline -rc / GA kernel to compile next-20200526 and then test from there.
> 

The last occurrence was next-20200521. Do you know the commit which fixed
this problem (so that we can confirm the problem was already fixed) ?

Re: linux-next build error (8)

[Qian Cai]

> On May 26, 2020, at 8:45 AM, Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote:
> 
> The last occurrence was next-20200521. Do you know the commit which fixed
> this problem (so that we can confirm the problem was already fixed) ?

Not on top of my head, but I did confirm the issue was fixed since next-20200525.

Re: linux-next build error (8)

[Tetsuo Handa]

On 2020/05/26 21:50, Qian Cai wrote:
> 
> 
>> On May 26, 2020, at 8:45 AM, Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote:
>>
>> The last occurrence was next-20200521. Do you know the commit which fixed
>> this problem (so that we can confirm the problem was already fixed) ?
> 
> Not on top of my head, but I did confirm the issue was fixed since next-20200525.
> 

Then, we can ignore this problem. Thank you.

Re: linux-next build error (8)

[Dmitry Vyukov]

On Tue, May 26, 2020 at 2:41 PM Qian Cai <cai@lca.pw> wrote:
>
>
>
> > On May 26, 2020, at 8:28 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > Crashes (4):
> > Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro
> > ci-upstream-linux-next-kasan-gce-root 2020/05/22 01:23 linux-next
> > e8f32747 5afa2ddd .config log report
> > ci-upstream-linux-next-kasan-gce-root 2020/05/21 15:01 linux-next
> > e8f32747 1f30020f .config log report
> > ci-upstream-linux-next-kasan-gce-root 2020/05/19 18:24 linux-next
> > fb57b1fa 6d882fd2 .config log report
> > ci-upstream-linux-next-kasan-gce-root 2020/03/18 16:19 linux-next
> > 47780d78 0a96a13c .config log report
>
> You’ll probably need to use an known good kernel version. For example, a stock kernel or any of a mainline -rc / GA kernel to compile next-20200526 and then test from there.

People also argued for the opposite -- finding bugs only on rc's is
too late. I think Linus also did not want bugs from entering the
mainline tree.

Ideally, all kernel patches tested on CI for simpler bugs before
entering any tree. And then fuzzing finds only harder to hit bugs.
syzbot is a really poor CI and it wasn't built to be a one.

Re: linux-next build error (8)

[Qian Cai]

On Tue, May 26, 2020 at 03:26:40PM +0200, Dmitry Vyukov wrote:
> On Tue, May 26, 2020 at 2:41 PM Qian Cai <cai@lca.pw> wrote:
> >
> >
> >
> > > On May 26, 2020, at 8:28 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> > >
> > > Crashes (4):
> > > Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro
> > > ci-upstream-linux-next-kasan-gce-root 2020/05/22 01:23 linux-next
> > > e8f32747 5afa2ddd .config log report
> > > ci-upstream-linux-next-kasan-gce-root 2020/05/21 15:01 linux-next
> > > e8f32747 1f30020f .config log report
> > > ci-upstream-linux-next-kasan-gce-root 2020/05/19 18:24 linux-next
> > > fb57b1fa 6d882fd2 .config log report
> > > ci-upstream-linux-next-kasan-gce-root 2020/03/18 16:19 linux-next
> > > 47780d78 0a96a13c .config log report
> >
> > You’ll probably need to use an known good kernel version. For
> > example, a stock kernel or any of a mainline -rc / GA kernel to
> > compile next-20200526 and then test from there.
> 
> People also argued for the opposite -- finding bugs only on rc's is
> too late. I think Linus also did not want bugs from entering the
> mainline tree.
> 
> Ideally, all kernel patches tested on CI for simpler bugs before
> entering any tree. And then fuzzing finds only harder to hit bugs.
> syzbot is a really poor CI and it wasn't built to be a one.

I had only suggested it as a way to workaround/confirm this bug which
will cause abormal memory usage for compilation workloads. Once you get
a working next-0526 kernel, you should be able to use that to compile
future -next trees.

I would also agree that our maintainers should make the quality bar
higher for linux-next commits, so I could get some vacations.

Re: linux-next build error (8)

[Dmitry Vyukov]

On Tue, May 26, 2020 at 3:50 PM Qian Cai <cai@lca.pw> wrote:
> > > > On May 26, 2020, at 8:28 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> > > >
> > > > Crashes (4):
> > > > Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro
> > > > ci-upstream-linux-next-kasan-gce-root 2020/05/22 01:23 linux-next
> > > > e8f32747 5afa2ddd .config log report
> > > > ci-upstream-linux-next-kasan-gce-root 2020/05/21 15:01 linux-next
> > > > e8f32747 1f30020f .config log report
> > > > ci-upstream-linux-next-kasan-gce-root 2020/05/19 18:24 linux-next
> > > > fb57b1fa 6d882fd2 .config log report
> > > > ci-upstream-linux-next-kasan-gce-root 2020/03/18 16:19 linux-next
> > > > 47780d78 0a96a13c .config log report
> > >
> > > You’ll probably need to use an known good kernel version. For
> > > example, a stock kernel or any of a mainline -rc / GA kernel to
> > > compile next-20200526 and then test from there.
> >
> > People also argued for the opposite -- finding bugs only on rc's is
> > too late. I think Linus also did not want bugs from entering the
> > mainline tree.
> >
> > Ideally, all kernel patches tested on CI for simpler bugs before
> > entering any tree. And then fuzzing finds only harder to hit bugs.
> > syzbot is a really poor CI and it wasn't built to be a one.
>
> I had only suggested it as a way to workaround/confirm this bug which
> will cause abormal memory usage for compilation workloads. Once you get
> a working next-0526 kernel, you should be able to use that to compile
> future -next trees.
>
> I would also agree that our maintainers should make the quality bar
> higher for linux-next commits, so I could get some vacations.

:)
true

Re: linux-next build error (8)

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

Hi Dmitry,

On Tue, 26 May 2020 14:09:28 +0200 Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Fri, May 22, 2020 at 6:29 AM Tetsuo Handa
> <penguin-kernel@i-love.sakura.ne.jp> wrote:
> >
> > Hello.
> >
> > This report is already reporting next problem. Since the location seems to be
> > different, this might be caused by OOM due to too much parallel compilation.
> > Maybe syzbot can detect "gcc: fatal error: Killed signal terminated program cc1"
> > sequence and retry with reduced "make -j$NUM" settings.
> >
> >   gcc: fatal error: Killed signal terminated program cc1
> >   compilation terminated.
> >   scripts/Makefile.build:272: recipe for target 'fs/xfs/libxfs/xfs_btree.o' failed
> >   make[2]: *** [fs/xfs/libxfs/xfs_btree.o] Error 1
> >   make[2]: *** Waiting for unfinished jobs....  
> 
> +linux-next and XFS maintainers

What version of linux-next is this?  There was a problem last week with
some changes in the tip tree that caused large memory usage.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linux-next build error (8)

[Dmitry Vyukov]

On Wed, May 27, 2020 at 1:33 AM Stephen Rothwell <sfr@canb.auug.org.au> wrote:
>
> Hi Dmitry,
>
> On Tue, 26 May 2020 14:09:28 +0200 Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > On Fri, May 22, 2020 at 6:29 AM Tetsuo Handa
> > <penguin-kernel@i-love.sakura.ne.jp> wrote:
> > >
> > > Hello.
> > >
> > > This report is already reporting next problem. Since the location seems to be
> > > different, this might be caused by OOM due to too much parallel compilation.
> > > Maybe syzbot can detect "gcc: fatal error: Killed signal terminated program cc1"
> > > sequence and retry with reduced "make -j$NUM" settings.
> > >
> > >   gcc: fatal error: Killed signal terminated program cc1
> > >   compilation terminated.
> > >   scripts/Makefile.build:272: recipe for target 'fs/xfs/libxfs/xfs_btree.o' failed
> > >   make[2]: *** [fs/xfs/libxfs/xfs_btree.o] Error 1
> > >   make[2]: *** Waiting for unfinished jobs....
> >
> > +linux-next and XFS maintainers
>
> What version of linux-next is this?  There was a problem last week with
> some changes in the tip tree that caused large memory usage.

Hi Stephen,

Detailed info about each syzbot crash is always available over the
dashboard link:

https://syzkaller.appspot.com/bug?extid=792dec47d693ccdc05a0
Crashes (4):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro
ci-upstream-linux-next-kasan-gce-root 2020/05/22 01:23 linux-next
e8f32747 5afa2ddd .config log report
ci-upstream-linux-next-kasan-gce-root 2020/05/21 15:01 linux-next
e8f32747 1f30020f .config log report
ci-upstream-linux-next-kasan-gce-root 2020/05/19 18:24 linux-next
fb57b1fa 6d882fd2 .config log report
ci-upstream-linux-next-kasan-gce-root 2020/03/18 16:19 linux-next
47780d78 0a96a13c .config log report

Re: linux-next build error (8)

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Hi Dmitry,

On Wed, 27 May 2020 07:41:15 +0200 Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Wed, May 27, 2020 at 1:33 AM Stephen Rothwell <sfr@canb.auug.org.au> wrote:
> >
> > What version of linux-next is this?  There was a problem last week with
> > some changes in the tip tree that caused large memory usage.  
> 
> Hi Stephen,
> 
> Detailed info about each syzbot crash is always available over the
> dashboard link:

Thanks.  As others have said, this has been taken care of - the
problematic commits were dropped from next-2020522 and the fixes
appeared in next-20200525.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linux-next build error (8)

[Dmitry Vyukov]

On Wed, May 27, 2020 at 8:25 AM Stephen Rothwell <sfr@canb.auug.org.au> wrote:
>
> Hi Dmitry,
>
> On Wed, 27 May 2020 07:41:15 +0200 Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > On Wed, May 27, 2020 at 1:33 AM Stephen Rothwell <sfr@canb.auug.org.au> wrote:
> > >
> > > What version of linux-next is this?  There was a problem last week with
> > > some changes in the tip tree that caused large memory usage.
> >
> > Hi Stephen,
> >
> > Detailed info about each syzbot crash is always available over the
> > dashboard link:
>
> Thanks.  As others have said, this has been taken care of - the
> problematic commits were dropped from next-2020522 and the fixes
> appeared in next-20200525.

OK, let's close this then, otherwise this is open since March, new
failures are piling up, we are not getting notifications about new
ones, and it turns into an unuseful mess.

#syz invalid