Alsa-Devel Archive on lore.kernel.org
 help / color / Atom feed
* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
       [not found] <cki.1210A7ECB0.BD9Q3APV4K@redhat.com>
@ 2019-11-04 13:35 ` Jan Stancek
  2019-11-04 13:51   ` Greg KH
  2019-11-05  8:15   ` [alsa-devel] " Takashi Iwai
  0 siblings, 2 replies; 10+ messages in thread
From: Jan Stancek @ 2019-11-04 13:35 UTC (permalink / raw)
  To: CKI Project, Jaroslav Kysela, alsa-devel, LTP Mailing List
  Cc: Memory Management, Linux Stable maillist



----- Original Message -----
> 
> Hello,
> 
> We ran automated tests on a recent commit from this kernel tree:
> 
>        Kernel repo:
>        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
>             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> 
> The results of these automated tests are provided below.
> 
>     Overall result: FAILED (see details below)
>              Merge: OK
>            Compile: OK
>              Tests: FAILED
> 
> All kernel binaries, config files, and logs are available for download here:
> 
>   https://artifacts.cki-project.org/pipelines/262380
> 
> One or more kernel tests failed:
> 
>     x86_64:
>      ❌ LTP lite
>

Not a 5.3 -stable regression.

Failure comes from test that sanity checks all /proc files by doing
1k read from each. There are couple issues it hits wrt. snd_hda_*.

Example reproducer:
  dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock

It's slow and triggers soft lockups [1]. And it also requires lot
of memory, triggering OOMs on smaller VMs:
0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144

I'm leaning towards skipping all regmap entries in this test.
Comments are welcomed.

Regards,
Jan

[1]
[15341.137116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [dd:636573]
[15341.144141] Modules linked in: nls_utf8 isofs dummy minix binfmt_misc nfsv3 nfs_acl nfs lockd grace fscache sctp rds brd vfat fat btrfs xor zstd_compress raid6_pq zstd_decompress loop tu
n ip6table_nat ip6_tables xt_conntrack iptable_filter xt_MASQUERADE xt_comment iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 veth bridge stp llc overlay fuse snd_hda_codec_r
ealtek snd_hda_codec_generic sunrpc ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_hda_codec edac_mce_amd kvm_amd snd_hda_core ccp snd_hwdep snd_pcm kvm snd_timer irqbypass joydev snd c
rct10dif_pclmul crc32_pclmul soundcore broadcom bcm_phy_lib ghash_clmulni_intel sp5100_tco fam15h_power k10temp tg3 wmi_bmof pcspkr i2c_piix4 acpi_cpufreq ip_tables xfs libcrc32c radeon ata
_generic i2c_algo_bit pata_acpi drm_kms_helper firewire_ohci ttm crc32c_intel serio_raw pata_atiixp firewire_core drm crc_itu_t wmi [last unloaded: can]
[15341.223635] CPU: 2 PID: 636573 Comm: dd Tainted: G             L    5.3.9-rc1-dfe283e.cki #1
[15341.232073] Hardware name: AMD Pike/Pike, BIOS RPK1506A 09/03/2014
[15341.238467] RIP: 0010:regmap_readable+0x15/0x60
[15341.242996] Code: 25 28 00 00 00 75 07 48 83 c4 10 5b 5d c3 e8 92 08 a6 ff 66 90 0f 1f 44 00 00 48 83 bf b0 01 00 00 00 74 31 8b 97 48 01 00 00 <39> f2 0f 92 c0 85 d2 0f 95 c2 20 d0 75 1
d 48 83 7f 70 00 74 01 c3
[15341.261765] RSP: 0018:ffffb3b100697dc8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[15341.269330] RAX: 0000000000000000 RBX: ffff8d1a63773400 RCX: 0000000000000b41
[15341.276723] RDX: 000000000fffffff RSI: 000000000f3b4139 RDI: ffff8d1a63773400
[15341.283858] RBP: 000000000f3b4139 R08: 0000000000000000 R09: 0000000000000000
[15341.290989] R10: 000000000000000f R11: ffff8d19901fffff R12: 0000000000000079
[15341.298114] R13: 000000000f3b4139 R14: 00000000ffffffff R15: 000000000000006e
[15341.305501] FS:  00007f4e067a0580(0000) GS:ffff8d1a6aa80000(0000) knlGS:0000000000000000
[15341.313583] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15341.319322] CR2: 00007f5083682dc0 CR3: 00000001287dc000 CR4: 00000000000406e0
[15341.326462] Call Trace:
[15341.328912]  regmap_volatile+0x4f/0xa0
[15341.332667]  regmap_access_show+0x70/0x100
[15341.336997]  seq_read+0xcd/0x400
[15341.340247]  full_proxy_read+0x57/0x70
[15341.343997]  vfs_read+0x9d/0x150
[15341.347231]  ksys_read+0x5f/0xe0
[15341.350473]  do_syscall_64+0x5f/0x1a0
[15341.354166]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[15341.359216] RIP: 0033:0x7f4e066823c2
[15341.362795] Code: c0 e9 c2 fe ff ff 50 48 8d 3d c2 0d 0a 00 e8 b5 f1 01 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 0
0 48 83 ec 28 48 89 54 24
[15341.381852] RSP: 002b:00007ffc6ad30e88 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[15341.389427] RAX: ffffffffffffffda RBX: 0000000000000400 RCX: 00007f4e066823c2
[15341.396568] RDX: 0000000000000400 RSI: 0000561bc69f0000 RDI: 0000000000000000
[15341.403979] RBP: 0000561bc69f0000 R08: 0000561bc69efb60 R09: 00000000000000c0
[15341.411111] R10: 0000561bc69f0000 R11: 0000000000000246 R12: ffffffffffffffff
[15341.418244] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffffffffff

_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 13:35 ` [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable) Jan Stancek
@ 2019-11-04 13:51   ` Greg KH
  2019-11-04 14:28     ` Jan Stancek
  2019-11-05  8:15   ` [alsa-devel] " Takashi Iwai
  1 sibling, 1 reply; 10+ messages in thread
From: Greg KH @ 2019-11-04 13:51 UTC (permalink / raw)
  To: Jan Stancek
  Cc: alsa-devel, Memory Management, Linux Stable maillist,
	Jaroslav Kysela, CKI Project, LTP Mailing List

On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
> 
> 
> ----- Original Message -----
> > 
> > Hello,
> > 
> > We ran automated tests on a recent commit from this kernel tree:
> > 
> >        Kernel repo:
> >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> > 
> > The results of these automated tests are provided below.
> > 
> >     Overall result: FAILED (see details below)
> >              Merge: OK
> >            Compile: OK
> >              Tests: FAILED
> > 
> > All kernel binaries, config files, and logs are available for download here:
> > 
> >   https://artifacts.cki-project.org/pipelines/262380
> > 
> > One or more kernel tests failed:
> > 
> >     x86_64:
> >      ❌ LTP lite
> >
> 
> Not a 5.3 -stable regression.
> 
> Failure comes from test that sanity checks all /proc files by doing
> 1k read from each. There are couple issues it hits wrt. snd_hda_*.
> 
> Example reproducer:
>   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock

That's not a proc file :)

> It's slow and triggers soft lockups [1]. And it also requires lot
> of memory, triggering OOMs on smaller VMs:
> 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
> 
> I'm leaning towards skipping all regmap entries in this test.
> Comments are welcomed.

Randomly poking around in debugfs is a sure way to cause crashes and
major problems.  Also, debugfs files are NOT stable and only for
debugging and should never be enabled on "real" systems.

So what exactly is the test trying to do here?

thanks,

greg k-h
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 13:51   ` Greg KH
@ 2019-11-04 14:28     ` Jan Stancek
  2019-11-04 14:59       ` Greg KH
  0 siblings, 1 reply; 10+ messages in thread
From: Jan Stancek @ 2019-11-04 14:28 UTC (permalink / raw)
  To: Greg KH
  Cc: alsa-devel, Memory Management, Linux Stable maillist,
	Jaroslav Kysela, CKI Project, LTP Mailing List



----- Original Message -----
> On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
> > 
> > 
> > ----- Original Message -----
> > > 
> > > Hello,
> > > 
> > > We ran automated tests on a recent commit from this kernel tree:
> > > 
> > >        Kernel repo:
> > >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> > >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> > > 
> > > The results of these automated tests are provided below.
> > > 
> > >     Overall result: FAILED (see details below)
> > >              Merge: OK
> > >            Compile: OK
> > >              Tests: FAILED
> > > 
> > > All kernel binaries, config files, and logs are available for download
> > > here:
> > > 
> > >   https://artifacts.cki-project.org/pipelines/262380
> > > 
> > > One or more kernel tests failed:
> > > 
> > >     x86_64:
> > >      ❌ LTP lite
> > >
> > 
> > Not a 5.3 -stable regression.
> > 
> > Failure comes from test that sanity checks all /proc files by doing
> > 1k read from each. There are couple issues it hits wrt. snd_hda_*.
> > 
> > Example reproducer:
> >   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt
> >   count=1 bs=1024 iflag=nonblock
> 
> That's not a proc file :)

Right. It's same test that's used for /proc too.

> 
> > It's slow and triggers soft lockups [1]. And it also requires lot
> > of memory, triggering OOMs on smaller VMs:
> > 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400
> > pages=262144 vmalloc vpages N0=262144
> > 
> > I'm leaning towards skipping all regmap entries in this test.
> > Comments are welcomed.
> 
> Randomly poking around in debugfs is a sure way to cause crashes and
> major problems.  Also, debugfs files are NOT stable and only for
> debugging and should never be enabled on "real" systems.
> 
> So what exactly is the test trying to do here?

It's (unprivileged) user trying to open/read anything it can (/proc, /sys)
to see if that triggers anything bad.

It can run as privileged user too, which was the case above.

[1] https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/fs/read_all/read_all.c

_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 14:28     ` Jan Stancek
@ 2019-11-04 14:59       ` Greg KH
  2019-11-04 15:25         ` Jan Stancek
  0 siblings, 1 reply; 10+ messages in thread
From: Greg KH @ 2019-11-04 14:59 UTC (permalink / raw)
  To: Jan Stancek
  Cc: alsa-devel, Memory Management, Linux Stable maillist,
	Jaroslav Kysela, CKI Project, LTP Mailing List

On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
> 
> 
> ----- Original Message -----
> > On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
> > > 
> > > 
> > > ----- Original Message -----
> > > > 
> > > > Hello,
> > > > 
> > > > We ran automated tests on a recent commit from this kernel tree:
> > > > 
> > > >        Kernel repo:
> > > >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> > > >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> > > > 
> > > > The results of these automated tests are provided below.
> > > > 
> > > >     Overall result: FAILED (see details below)
> > > >              Merge: OK
> > > >            Compile: OK
> > > >              Tests: FAILED
> > > > 
> > > > All kernel binaries, config files, and logs are available for download
> > > > here:
> > > > 
> > > >   https://artifacts.cki-project.org/pipelines/262380
> > > > 
> > > > One or more kernel tests failed:
> > > > 
> > > >     x86_64:
> > > >      ❌ LTP lite
> > > >
> > > 
> > > Not a 5.3 -stable regression.
> > > 
> > > Failure comes from test that sanity checks all /proc files by doing
> > > 1k read from each. There are couple issues it hits wrt. snd_hda_*.
> > > 
> > > Example reproducer:
> > >   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt
> > >   count=1 bs=1024 iflag=nonblock
> > 
> > That's not a proc file :)
> 
> Right. It's same test that's used for /proc too.
> 
> > 
> > > It's slow and triggers soft lockups [1]. And it also requires lot
> > > of memory, triggering OOMs on smaller VMs:
> > > 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400
> > > pages=262144 vmalloc vpages N0=262144
> > > 
> > > I'm leaning towards skipping all regmap entries in this test.
> > > Comments are welcomed.
> > 
> > Randomly poking around in debugfs is a sure way to cause crashes and
> > major problems.  Also, debugfs files are NOT stable and only for
> > debugging and should never be enabled on "real" systems.
> > 
> > So what exactly is the test trying to do here?
> 
> It's (unprivileged) user trying to open/read anything it can (/proc, /sys)
> to see if that triggers anything bad.
> 
> It can run as privileged user too, which was the case above.

Sure, you can do tons of bad things as root poking around in sysfs,
debugfs, and procfs.  What exactly are you trying to do, break the
system?

That sounds like a horrible test that is just setting itself up to lock
the system up, you should revisit it...

thanks,

greg k-h
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 14:59       ` Greg KH
@ 2019-11-04 15:25         ` Jan Stancek
  2019-11-04 16:30           ` Greg KH
  0 siblings, 1 reply; 10+ messages in thread
From: Jan Stancek @ 2019-11-04 15:25 UTC (permalink / raw)
  To: Greg KH
  Cc: alsa-devel, Memory Management, Linux Stable maillist,
	Jaroslav Kysela, CKI Project, LTP Mailing List


----- Original Message -----
> On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
> > 
> > 
> > ----- Original Message -----
> > > On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > 
> > > > > Hello,
> > > > > 
> > > > > We ran automated tests on a recent commit from this kernel tree:
> > > > > 
> > > > >        Kernel repo:
> > > > >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> > > > >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> > > > > 
> > > > > The results of these automated tests are provided below.
> > > > > 
> > > > >     Overall result: FAILED (see details below)
> > > > >              Merge: OK
> > > > >            Compile: OK
> > > > >              Tests: FAILED
> > > > > 
> > > > > All kernel binaries, config files, and logs are available for
> > > > > download
> > > > > here:
> > > > > 
> > > > >   https://artifacts.cki-project.org/pipelines/262380
> > > > > 
> > > > > One or more kernel tests failed:
> > > > > 
> > > > >     x86_64:
> > > > >      ❌ LTP lite
> > > > >
> > > > 
> > > > Not a 5.3 -stable regression.
> > > > 
> > > > Failure comes from test that sanity checks all /proc files by doing
> > > > 1k read from each. There are couple issues it hits wrt. snd_hda_*.
> > > > 
> > > > Example reproducer:
> > > >   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt
> > > >   count=1 bs=1024 iflag=nonblock
> > > 
> > > That's not a proc file :)
> > 
> > Right. It's same test that's used for /proc too.
> > 
> > > 
> > > > It's slow and triggers soft lockups [1]. And it also requires lot
> > > > of memory, triggering OOMs on smaller VMs:
> > > > 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400
> > > > pages=262144 vmalloc vpages N0=262144
> > > > 
> > > > I'm leaning towards skipping all regmap entries in this test.
> > > > Comments are welcomed.
> > > 
> > > Randomly poking around in debugfs is a sure way to cause crashes and
> > > major problems.  Also, debugfs files are NOT stable and only for
> > > debugging and should never be enabled on "real" systems.
> > > 
> > > So what exactly is the test trying to do here?
> > 
> > It's (unprivileged) user trying to open/read anything it can (/proc, /sys)
> > to see if that triggers anything bad.
> > 
> > It can run as privileged user too, which was the case above.
> 
> Sure, you can do tons of bad things as root poking around in sysfs,
> debugfs, and procfs.  What exactly are you trying to do, break the
> system?

We are talking about read-only here. Is it unreasonable to expect
that root can read all /proc entries without crashing the system?

Some entries are readable only by root. So an unprivileged user
will skip considerable number of entries:
-r--------. 1 root root 0 Nov  4 16:13 /proc/slabinfo

> That sounds like a horrible test that is just setting itself up to lock
> the system up, you should revisit it...

_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 15:25         ` Jan Stancek
@ 2019-11-04 16:30           ` Greg KH
  2019-11-04 17:02             ` Jan Stancek
  0 siblings, 1 reply; 10+ messages in thread
From: Greg KH @ 2019-11-04 16:30 UTC (permalink / raw)
  To: Jan Stancek
  Cc: alsa-devel, Memory Management, Linux Stable maillist,
	Jaroslav Kysela, CKI Project, LTP Mailing List

On Mon, Nov 04, 2019 at 10:25:21AM -0500, Jan Stancek wrote:
> 
> ----- Original Message -----
> > On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
> > > 
> > > 
> > > ----- Original Message -----
> > > > On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
> > > > > 
> > > > > 
> > > > > ----- Original Message -----
> > > > > > 
> > > > > > Hello,
> > > > > > 
> > > > > > We ran automated tests on a recent commit from this kernel tree:
> > > > > > 
> > > > > >        Kernel repo:
> > > > > >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> > > > > >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> > > > > > 
> > > > > > The results of these automated tests are provided below.
> > > > > > 
> > > > > >     Overall result: FAILED (see details below)
> > > > > >              Merge: OK
> > > > > >            Compile: OK
> > > > > >              Tests: FAILED
> > > > > > 
> > > > > > All kernel binaries, config files, and logs are available for
> > > > > > download
> > > > > > here:
> > > > > > 
> > > > > >   https://artifacts.cki-project.org/pipelines/262380
> > > > > > 
> > > > > > One or more kernel tests failed:
> > > > > > 
> > > > > >     x86_64:
> > > > > >      ❌ LTP lite
> > > > > >
> > > > > 
> > > > > Not a 5.3 -stable regression.
> > > > > 
> > > > > Failure comes from test that sanity checks all /proc files by doing
> > > > > 1k read from each. There are couple issues it hits wrt. snd_hda_*.
> > > > > 
> > > > > Example reproducer:
> > > > >   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt
> > > > >   count=1 bs=1024 iflag=nonblock
> > > > 
> > > > That's not a proc file :)
> > > 
> > > Right. It's same test that's used for /proc too.
> > > 
> > > > 
> > > > > It's slow and triggers soft lockups [1]. And it also requires lot
> > > > > of memory, triggering OOMs on smaller VMs:
> > > > > 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400
> > > > > pages=262144 vmalloc vpages N0=262144
> > > > > 
> > > > > I'm leaning towards skipping all regmap entries in this test.
> > > > > Comments are welcomed.
> > > > 
> > > > Randomly poking around in debugfs is a sure way to cause crashes and
> > > > major problems.  Also, debugfs files are NOT stable and only for
> > > > debugging and should never be enabled on "real" systems.
> > > > 
> > > > So what exactly is the test trying to do here?
> > > 
> > > It's (unprivileged) user trying to open/read anything it can (/proc, /sys)
> > > to see if that triggers anything bad.
> > > 
> > > It can run as privileged user too, which was the case above.
> > 
> > Sure, you can do tons of bad things as root poking around in sysfs,
> > debugfs, and procfs.  What exactly are you trying to do, break the
> > system?
> 
> We are talking about read-only here. Is it unreasonable to expect
> that root can read all /proc entries without crashing the system?

You are NOT reading /proc/ here.  You are reading debugfs which you
really have NOT idea what is in there.  As you saw, you are reading from
hardware that is slow and doing odd things when you read from it.

And yes, there are some /proc/ files that you should not read from as
root and expect things to always work.  PCI devices are notorious for
this, and if you are reading those files as root, you "know" you know
what you are doing and can accept the risk for when things go wrong.

It is fine to write tests to read specific /proc/ files that you know
what is happening in them.  To pick random ones is never a good idea.

thanks,

greg k-h
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 16:30           ` Greg KH
@ 2019-11-04 17:02             ` Jan Stancek
  2019-11-04 20:32               ` Greg KH
  0 siblings, 1 reply; 10+ messages in thread
From: Jan Stancek @ 2019-11-04 17:02 UTC (permalink / raw)
  To: Greg KH
  Cc: alsa-devel, Memory Management, Linux Stable maillist,
	Jaroslav Kysela, CKI Project, LTP Mailing List


----- Original Message -----
> On Mon, Nov 04, 2019 at 10:25:21AM -0500, Jan Stancek wrote:
> > 
> > ----- Original Message -----
> > > On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
> > > > > > 
> > > > > > 
> > > > > > ----- Original Message -----
> > > > > > > 
> > > > > > > Hello,
> > > > > > > 
> > > > > > > We ran automated tests on a recent commit from this kernel tree:
> > > > > > > 
> > > > > > >        Kernel repo:
> > > > > > >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> > > > > > >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> > > > > > > 
> > > > > > > The results of these automated tests are provided below.
> > > > > > > 
> > > > > > >     Overall result: FAILED (see details below)
> > > > > > >              Merge: OK
> > > > > > >            Compile: OK
> > > > > > >              Tests: FAILED
> > > > > > > 
> > > > > > > All kernel binaries, config files, and logs are available for
> > > > > > > download
> > > > > > > here:
> > > > > > > 
> > > > > > >   https://artifacts.cki-project.org/pipelines/262380
> > > > > > > 
> > > > > > > One or more kernel tests failed:
> > > > > > > 
> > > > > > >     x86_64:
> > > > > > >      ❌ LTP lite
> > > > > > >
> > > > > > 
> > > > > > Not a 5.3 -stable regression.
> > > > > > 
> > > > > > Failure comes from test that sanity checks all /proc files by doing
> > > > > > 1k read from each. There are couple issues it hits wrt. snd_hda_*.
> > > > > > 
> > > > > > Example reproducer:
> > > > > >   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access
> > > > > >   of=out.txt
> > > > > >   count=1 bs=1024 iflag=nonblock
> > > > > 
> > > > > That's not a proc file :)
> > > > 
> > > > Right. It's same test that's used for /proc too.
> > > > 
> > > > > 
> > > > > > It's slow and triggers soft lockups [1]. And it also requires lot
> > > > > > of memory, triggering OOMs on smaller VMs:
> > > > > > 0x0000000024f0437b-0x000000001a32b1c8 1073745920
> > > > > > seq_read+0x131/0x400
> > > > > > pages=262144 vmalloc vpages N0=262144
> > > > > > 
> > > > > > I'm leaning towards skipping all regmap entries in this test.
> > > > > > Comments are welcomed.
> > > > > 
> > > > > Randomly poking around in debugfs is a sure way to cause crashes and
> > > > > major problems.  Also, debugfs files are NOT stable and only for
> > > > > debugging and should never be enabled on "real" systems.
> > > > > 
> > > > > So what exactly is the test trying to do here?
> > > > 
> > > > It's (unprivileged) user trying to open/read anything it can (/proc,
> > > > /sys)
> > > > to see if that triggers anything bad.
> > > > 
> > > > It can run as privileged user too, which was the case above.
> > > 
> > > Sure, you can do tons of bad things as root poking around in sysfs,
> > > debugfs, and procfs.  What exactly are you trying to do, break the
> > > system?
> > 
> > We are talking about read-only here. Is it unreasonable to expect
> > that root can read all /proc entries without crashing the system?
> 
> You are NOT reading /proc/ here.

No. That was a general question to usefulness of privileged read,
using /proc as example where it commonly happens.

> You are reading debugfs which you
> really have NOT idea what is in there.  As you saw, you are reading from
> hardware that is slow and doing odd things when you read from it.

Agreed, I already sent a patch to LTP to blacklist it.

> And yes, there are some /proc/ files that you should not read from as
> root and expect things to always work.  PCI devices are notorious for
> this, and if you are reading those files as root, you "know" you know
> what you are doing and can accept the risk for when things go wrong.
> 
> It is fine to write tests to read specific /proc/ files that you know
> what is happening in them.  To pick random ones is never a good idea.

Thanks for example. 

_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 17:02             ` Jan Stancek
@ 2019-11-04 20:32               ` Greg KH
  2019-11-07 11:05                 ` [alsa-devel] [LTP] " Richard Palethorpe
  0 siblings, 1 reply; 10+ messages in thread
From: Greg KH @ 2019-11-04 20:32 UTC (permalink / raw)
  To: Jan Stancek
  Cc: alsa-devel, Memory Management, Linux Stable maillist,
	Jaroslav Kysela, CKI Project, LTP Mailing List

On Mon, Nov 04, 2019 at 12:02:53PM -0500, Jan Stancek wrote:
> 
> ----- Original Message -----
> > On Mon, Nov 04, 2019 at 10:25:21AM -0500, Jan Stancek wrote:
> > > 
> > > ----- Original Message -----
> > > > On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
> > > > > 
> > > > > 
> > > > > ----- Original Message -----
> > > > > > On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > ----- Original Message -----
> > > > > > > > 
> > > > > > > > Hello,
> > > > > > > > 
> > > > > > > > We ran automated tests on a recent commit from this kernel tree:
> > > > > > > > 
> > > > > > > >        Kernel repo:
> > > > > > > >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> > > > > > > >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> > > > > > > > 
> > > > > > > > The results of these automated tests are provided below.
> > > > > > > > 
> > > > > > > >     Overall result: FAILED (see details below)
> > > > > > > >              Merge: OK
> > > > > > > >            Compile: OK
> > > > > > > >              Tests: FAILED
> > > > > > > > 
> > > > > > > > All kernel binaries, config files, and logs are available for
> > > > > > > > download
> > > > > > > > here:
> > > > > > > > 
> > > > > > > >   https://artifacts.cki-project.org/pipelines/262380
> > > > > > > > 
> > > > > > > > One or more kernel tests failed:
> > > > > > > > 
> > > > > > > >     x86_64:
> > > > > > > >      ❌ LTP lite
> > > > > > > >
> > > > > > > 
> > > > > > > Not a 5.3 -stable regression.
> > > > > > > 
> > > > > > > Failure comes from test that sanity checks all /proc files by doing
> > > > > > > 1k read from each. There are couple issues it hits wrt. snd_hda_*.
> > > > > > > 
> > > > > > > Example reproducer:
> > > > > > >   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access
> > > > > > >   of=out.txt
> > > > > > >   count=1 bs=1024 iflag=nonblock
> > > > > > 
> > > > > > That's not a proc file :)
> > > > > 
> > > > > Right. It's same test that's used for /proc too.
> > > > > 
> > > > > > 
> > > > > > > It's slow and triggers soft lockups [1]. And it also requires lot
> > > > > > > of memory, triggering OOMs on smaller VMs:
> > > > > > > 0x0000000024f0437b-0x000000001a32b1c8 1073745920
> > > > > > > seq_read+0x131/0x400
> > > > > > > pages=262144 vmalloc vpages N0=262144
> > > > > > > 
> > > > > > > I'm leaning towards skipping all regmap entries in this test.
> > > > > > > Comments are welcomed.
> > > > > > 
> > > > > > Randomly poking around in debugfs is a sure way to cause crashes and
> > > > > > major problems.  Also, debugfs files are NOT stable and only for
> > > > > > debugging and should never be enabled on "real" systems.
> > > > > > 
> > > > > > So what exactly is the test trying to do here?
> > > > > 
> > > > > It's (unprivileged) user trying to open/read anything it can (/proc,
> > > > > /sys)
> > > > > to see if that triggers anything bad.
> > > > > 
> > > > > It can run as privileged user too, which was the case above.
> > > > 
> > > > Sure, you can do tons of bad things as root poking around in sysfs,
> > > > debugfs, and procfs.  What exactly are you trying to do, break the
> > > > system?
> > > 
> > > We are talking about read-only here. Is it unreasonable to expect
> > > that root can read all /proc entries without crashing the system?
> > 
> > You are NOT reading /proc/ here.
> 
> No. That was a general question to usefulness of privileged read,
> using /proc as example where it commonly happens.
> 
> > You are reading debugfs which you
> > really have NOT idea what is in there.  As you saw, you are reading from
> > hardware that is slow and doing odd things when you read from it.
> 
> Agreed, I already sent a patch to LTP to blacklist it.

Hopefully you are blacklisting all of debugfs.  and sysfs.  And procfs.
Adding specific files back is fine, as long as you "know" they are ok
and you are actually testing something valid there.

thanks,

greg k-h
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 13:35 ` [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable) Jan Stancek
  2019-11-04 13:51   ` Greg KH
@ 2019-11-05  8:15   ` " Takashi Iwai
  1 sibling, 0 replies; 10+ messages in thread
From: Takashi Iwai @ 2019-11-05  8:15 UTC (permalink / raw)
  To: Jan Stancek
  Cc: alsa-devel, Memory Management, Linux Stable maillist,
	Jaroslav Kysela, CKI Project, LTP Mailing List

On Mon, 04 Nov 2019 14:35:51 +0100,
Jan Stancek wrote:
> 
> 
> 
> ----- Original Message -----
> > 
> > Hello,
> > 
> > We ran automated tests on a recent commit from this kernel tree:
> > 
> >        Kernel repo:
> >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
> > 
> > The results of these automated tests are provided below.
> > 
> >     Overall result: FAILED (see details below)
> >              Merge: OK
> >            Compile: OK
> >              Tests: FAILED
> > 
> > All kernel binaries, config files, and logs are available for download here:
> > 
> >   https://artifacts.cki-project.org/pipelines/262380
> > 
> > One or more kernel tests failed:
> > 
> >     x86_64:
> >      ❌ LTP lite
> >
> 
> Not a 5.3 -stable regression.
> 
> Failure comes from test that sanity checks all /proc files by doing
> 1k read from each. There are couple issues it hits wrt. snd_hda_*.

Yes, the debugfs access for HD-audio may easily lead to the OOM or
such because HD-audio provides a pretty sparse register maps and the
regmap debugfs iterates all possible register values.

I'm going to push a patch to disable the regmap internal locking,
which eventually disables the whole regmap debugfs exposure of the
device.

But, as Greg already suggested, the whole debugfs access can't be
treated as safe at all, no matter which device is handled, so it
should be avoided.  It's the interface allowing you to touch the
hardware internals directly, so you can screw up the system very
easily.


thanks,

Takashi

> 
> Example reproducer:
>   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access of=out.txt count=1 bs=1024 iflag=nonblock
> 
> It's slow and triggers soft lockups [1]. And it also requires lot
> of memory, triggering OOMs on smaller VMs:
> 0x0000000024f0437b-0x000000001a32b1c8 1073745920 seq_read+0x131/0x400 pages=262144 vmalloc vpages N0=262144
> 
> I'm leaning towards skipping all regmap entries in this test.
> Comments are welcomed.
> 
> Regards,
> Jan
> 
> [1]
> [15341.137116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [dd:636573]
> [15341.144141] Modules linked in: nls_utf8 isofs dummy minix binfmt_misc nfsv3 nfs_acl nfs lockd grace fscache sctp rds brd vfat fat btrfs xor zstd_compress raid6_pq zstd_decompress loop tu
> n ip6table_nat ip6_tables xt_conntrack iptable_filter xt_MASQUERADE xt_comment iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 veth bridge stp llc overlay fuse snd_hda_codec_r
> ealtek snd_hda_codec_generic sunrpc ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_hda_codec edac_mce_amd kvm_amd snd_hda_core ccp snd_hwdep snd_pcm kvm snd_timer irqbypass joydev snd c
> rct10dif_pclmul crc32_pclmul soundcore broadcom bcm_phy_lib ghash_clmulni_intel sp5100_tco fam15h_power k10temp tg3 wmi_bmof pcspkr i2c_piix4 acpi_cpufreq ip_tables xfs libcrc32c radeon ata
> _generic i2c_algo_bit pata_acpi drm_kms_helper firewire_ohci ttm crc32c_intel serio_raw pata_atiixp firewire_core drm crc_itu_t wmi [last unloaded: can]
> [15341.223635] CPU: 2 PID: 636573 Comm: dd Tainted: G             L    5.3.9-rc1-dfe283e.cki #1
> [15341.232073] Hardware name: AMD Pike/Pike, BIOS RPK1506A 09/03/2014
> [15341.238467] RIP: 0010:regmap_readable+0x15/0x60
> [15341.242996] Code: 25 28 00 00 00 75 07 48 83 c4 10 5b 5d c3 e8 92 08 a6 ff 66 90 0f 1f 44 00 00 48 83 bf b0 01 00 00 00 74 31 8b 97 48 01 00 00 <39> f2 0f 92 c0 85 d2 0f 95 c2 20 d0 75 1
> d 48 83 7f 70 00 74 01 c3
> [15341.261765] RSP: 0018:ffffb3b100697dc8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
> [15341.269330] RAX: 0000000000000000 RBX: ffff8d1a63773400 RCX: 0000000000000b41
> [15341.276723] RDX: 000000000fffffff RSI: 000000000f3b4139 RDI: ffff8d1a63773400
> [15341.283858] RBP: 000000000f3b4139 R08: 0000000000000000 R09: 0000000000000000
> [15341.290989] R10: 000000000000000f R11: ffff8d19901fffff R12: 0000000000000079
> [15341.298114] R13: 000000000f3b4139 R14: 00000000ffffffff R15: 000000000000006e
> [15341.305501] FS:  00007f4e067a0580(0000) GS:ffff8d1a6aa80000(0000) knlGS:0000000000000000
> [15341.313583] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [15341.319322] CR2: 00007f5083682dc0 CR3: 00000001287dc000 CR4: 00000000000406e0
> [15341.326462] Call Trace:
> [15341.328912]  regmap_volatile+0x4f/0xa0
> [15341.332667]  regmap_access_show+0x70/0x100
> [15341.336997]  seq_read+0xcd/0x400
> [15341.340247]  full_proxy_read+0x57/0x70
> [15341.343997]  vfs_read+0x9d/0x150
> [15341.347231]  ksys_read+0x5f/0xe0
> [15341.350473]  do_syscall_64+0x5f/0x1a0
> [15341.354166]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [15341.359216] RIP: 0033:0x7f4e066823c2
> [15341.362795] Code: c0 e9 c2 fe ff ff 50 48 8d 3d c2 0d 0a 00 e8 b5 f1 01 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 0
> 0 48 83 ec 28 48 89 54 24
> [15341.381852] RSP: 002b:00007ffc6ad30e88 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> [15341.389427] RAX: ffffffffffffffda RBX: 0000000000000400 RCX: 00007f4e066823c2
> [15341.396568] RDX: 0000000000000400 RSI: 0000561bc69f0000 RDI: 0000000000000000
> [15341.403979] RBP: 0000561bc69f0000 R08: 0000561bc69efb60 R09: 00000000000000c0
> [15341.411111] R10: 0000561bc69f0000 R11: 0000000000000246 R12: ffffffffffffffff
> [15341.418244] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffffffffff
> 
> _______________________________________________
> Alsa-devel mailing list
> Alsa-devel@alsa-project.org
> https://mailman.alsa-project.org/mailman/listinfo/alsa-devel
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [alsa-devel] [LTP] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable)
  2019-11-04 20:32               ` Greg KH
@ 2019-11-07 11:05                 ` " Richard Palethorpe
  0 siblings, 0 replies; 10+ messages in thread
From: Richard Palethorpe @ 2019-11-07 11:05 UTC (permalink / raw)
  To: ltp
  Cc: alsa-devel, Memory Management, Jaroslav Kysela,
	Linux Stable maillist, CKI Project, Jan Stancek


Greg KH <gregkh@linuxfoundation.org> writes:

> On Mon, Nov 04, 2019 at 12:02:53PM -0500, Jan Stancek wrote:
>>
>> ----- Original Message -----
>> > On Mon, Nov 04, 2019 at 10:25:21AM -0500, Jan Stancek wrote:
>> > >
>> > > ----- Original Message -----
>> > > > On Mon, Nov 04, 2019 at 09:28:10AM -0500, Jan Stancek wrote:
>> > > > >
>> > > > >
>> > > > > ----- Original Message -----
>> > > > > > On Mon, Nov 04, 2019 at 08:35:51AM -0500, Jan Stancek wrote:
>> > > > > > >
>> > > > > > >
>> > > > > > > ----- Original Message -----
>> > > > > > > >
>> > > > > > > > Hello,
>> > > > > > > >
>> > > > > > > > We ran automated tests on a recent commit from this kernel tree:
>> > > > > > > >
>> > > > > > > >        Kernel repo:
>> > > > > > > >        git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
>> > > > > > > >             Commit: dfe283e9fdac - Linux 5.3.9-rc1
>> > > > > > > >
>> > > > > > > > The results of these automated tests are provided below.
>> > > > > > > >
>> > > > > > > >     Overall result: FAILED (see details below)
>> > > > > > > >              Merge: OK
>> > > > > > > >            Compile: OK
>> > > > > > > >              Tests: FAILED
>> > > > > > > >
>> > > > > > > > All kernel binaries, config files, and logs are available for
>> > > > > > > > download
>> > > > > > > > here:
>> > > > > > > >
>> > > > > > > >   https://artifacts.cki-project.org/pipelines/262380
>> > > > > > > >
>> > > > > > > > One or more kernel tests failed:
>> > > > > > > >
>> > > > > > > >     x86_64:
>> > > > > > > >      ❌ LTP lite
>> > > > > > > >
>> > > > > > >
>> > > > > > > Not a 5.3 -stable regression.
>> > > > > > >
>> > > > > > > Failure comes from test that sanity checks all /proc files by doing
>> > > > > > > 1k read from each. There are couple issues it hits wrt. snd_hda_*.
>> > > > > > >
>> > > > > > > Example reproducer:
>> > > > > > >   dd if=/sys/kernel/debug/regmap/hdaudioC0D3-hdaudio/access
>> > > > > > >   of=out.txt
>> > > > > > >   count=1 bs=1024 iflag=nonblock
>> > > > > >
>> > > > > > That's not a proc file :)
>> > > > >
>> > > > > Right. It's same test that's used for /proc too.
>> > > > >
>> > > > > >
>> > > > > > > It's slow and triggers soft lockups [1]. And it also requires lot
>> > > > > > > of memory, triggering OOMs on smaller VMs:
>> > > > > > > 0x0000000024f0437b-0x000000001a32b1c8 1073745920
>> > > > > > > seq_read+0x131/0x400
>> > > > > > > pages=262144 vmalloc vpages N0=262144
>> > > > > > >
>> > > > > > > I'm leaning towards skipping all regmap entries in this test.
>> > > > > > > Comments are welcomed.
>> > > > > >
>> > > > > > Randomly poking around in debugfs is a sure way to cause crashes and
>> > > > > > major problems.  Also, debugfs files are NOT stable and only for
>> > > > > > debugging and should never be enabled on "real" systems.
>> > > > > >
>> > > > > > So what exactly is the test trying to do here?
>> > > > >
>> > > > > It's (unprivileged) user trying to open/read anything it can (/proc,
>> > > > > /sys)
>> > > > > to see if that triggers anything bad.
>> > > > >
>> > > > > It can run as privileged user too, which was the case above.
>> > > >
>> > > > Sure, you can do tons of bad things as root poking around in sysfs,
>> > > > debugfs, and procfs.  What exactly are you trying to do, break the
>> > > > system?
>> > >
>> > > We are talking about read-only here. Is it unreasonable to expect
>> > > that root can read all /proc entries without crashing the system?
>> >
>> > You are NOT reading /proc/ here.
>>
>> No. That was a general question to usefulness of privileged read,
>> using /proc as example where it commonly happens.
>>
>> > You are reading debugfs which you
>> > really have NOT idea what is in there.  As you saw, you are reading from
>> > hardware that is slow and doing odd things when you read from it.
>>
>> Agreed, I already sent a patch to LTP to blacklist it.
>
> Hopefully you are blacklisting all of debugfs.  and sysfs.  And procfs.
> Adding specific files back is fine, as long as you "know" they are ok
> and you are actually testing something valid there.
>
> thanks,
>
> greg k-h

TBH, most of the value of this test comes from being dumb and doing
things nobody thought to do because it would be stupid. It's an accident
that the test exists in the first place because I found some bugs in
older kernels naively trying to dump the contents of /sys.

It is useful to occasionally run this test as root in a VM where it
can't do any harm, even on debugfs and dev. We have detected missing
patches doing things like that which would probably be stopped by a sane
whitelist.

However I would agree that this creates too much noise for a CI under
normal circumstances. We could maybe add some meta-data to the test
saying it will create noise and freeze the SUT and in the future users
can specify what level of noise they are prepared to accept when
executing the LTP.

In the meantime, IMO, we can just force the test to switch to nobody by
default which it already does when reading /dev.

--
Thank you,
Richard.
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
https://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, back to index

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <cki.1210A7ECB0.BD9Q3APV4K@redhat.com>
2019-11-04 13:35 ` [alsa-devel] ❌ FAIL: Test report for kernel 5.3.9-rc1-dfe283e.cki (stable) Jan Stancek
2019-11-04 13:51   ` Greg KH
2019-11-04 14:28     ` Jan Stancek
2019-11-04 14:59       ` Greg KH
2019-11-04 15:25         ` Jan Stancek
2019-11-04 16:30           ` Greg KH
2019-11-04 17:02             ` Jan Stancek
2019-11-04 20:32               ` Greg KH
2019-11-07 11:05                 ` [alsa-devel] [LTP] " Richard Palethorpe
2019-11-05  8:15   ` [alsa-devel] " Takashi Iwai

Alsa-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/alsa-devel/0 alsa-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 alsa-devel alsa-devel/ https://lore.kernel.org/alsa-devel \
		alsa-devel@alsa-project.org
	public-inbox-index alsa-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.alsa-project.alsa-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git