All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: A bug about cache inconsistency report
@ 2018-08-01  2:05 Hans Hu(SH-RD)
  2018-08-01  4:45 ` Takashi Iwai
  0 siblings, 1 reply; 10+ messages in thread
From: Hans Hu(SH-RD) @ 2018-08-01  2:05 UTC (permalink / raw)
  To: 'Takashi Iwai'; +Cc: 'alsa-devel@alsa-project.org'

I did more test base on different Intel platform, and result is the same.
Maybe I made some mistake? Can you do the test in yours  platform?


Thank you!
Hans

-----邮件原件-----
发件人: Takashi Iwai [mailto:tiwai@suse.de]
发送时间: 31 July 2018 21:55
收件人: Hans Hu(SH-RD) <HansHu@zhaoxin.com>
抄送: 'alsa-devel@alsa-project.org' <alsa-devel@alsa-project.org>
主题: Re: 答复: [alsa-devel] A bug about cache inconsistency report

On Tue, 31 Jul 2018 15:51:06 +0200,
Hans Hu(SH-RD) wrote:
>
> I did the test and Oops report when install driver( both snoop and
> non-snoop mode)

I don't understand.  Did my one-liner change the behavior or not?


thanks,

Takashi

>
> -----邮件原件-----
> 发件人: Takashi Iwai [mailto:tiwai@suse.de]
> 发送时间: 31 July 2018 19:59
> 收件人: Hans Hu(SH-RD) <HansHu@zhaoxin.com>
> 抄送: 'alsa-devel@alsa-project.org' <alsa-devel@alsa-project.org>
> 主题: Re: [alsa-devel] A bug about cache inconsistency report
>
> On Tue, 31 Jul 2018 12:52:04 +0200,
> Hans Hu(SH-RD) wrote:
> >
> > Hi All,
> >
> >                Try send an email.
> >
> >                Appearance:
> > when use non-snoop + ioctl path combination, an issue occurs:
> > usage: aplay -D plughw:0,0 44100_S16_LE_2c.wav Ubuntu, it
> > performance intermittently noise;
> >
> > Root case:
> >                Function set_pages_array_wc marked the PTE of the original address; Function set_memory_wc marked the PTE both original and after vmaped address.
> > [cid:image002.png@01D428FF.92621520]
> > However, cache or not in transmission process is based on the PTE of the later, and the set_pages_array_wc did't mark it, so a cache inconsistency issue occurs.
> >
> >                Related bug:
> >                SHA1 ID: 9ddf1aeb2134e72275c97a2c6ff2e3eb04f2f27a
>
> OK, so did you confirm that dropping the SG buffer support "fixes" the noise problem on your system?  The easiest test would be like:
>
> --- a/sound/pci/hda/hda_controller.c
> +++ b/sound/pci/hda/hda_controller.c
> @@ -770,7 +770,7 @@ int snd_hda_attach_pcm_stream(struct hda_bus
> *_bus, struct hda_codec *codec,  size = CONFIG_SND_HDA_PREALLOC_SIZE *
> 1024;  if (size > MAX_PREALLOC_SIZE)  size = MAX_PREALLOC_SIZE;
> -snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_DEV_SG,
> +snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_DEV,
>        chip->card->dev,
>        size, MAX_PREALLOC_SIZE);
>  return 0;
>
>
> thanks,
>
> Takashi
>
>
> 保密声明:
> 本邮件含有保密或专有信息,仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
> CONFIDENTIAL NOTE:
> This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.


保密声明:
本邮件含有保密或专有信息,仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
http://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread
* Re: A bug about cache inconsistency report
@ 2018-08-08 11:07 Hans Hu(SH-RD)
  2018-08-10 15:15 ` Takashi Iwai
  0 siblings, 1 reply; 10+ messages in thread
From: Hans Hu(SH-RD) @ 2018-08-08 11:07 UTC (permalink / raw)
  To: 'Takashi Iwai'
  Cc: Tony W. Wang(BJ-RD), 'alsa-devel@alsa-project.org',
	Tim Guo(BJ-RD), Annie Liu(BJ-RD)

> > > > >Then the next step would be to fake sg-buffer from this
> > > > >straight buffer.  Revert the above, and modify sgbuf.c to the following:
> > > > >- Allocate a large continuous buffer
> > > > >- Assign each page in this large buffer
> > > >
> > > > >If this still works, it's not about vmap, but it just means
> > > > >that the physically ordered pages do matter -- implicitly
> > > > >showing that the snooping behavior isn't properly turned on / off on the controller.
> > > >
> > > > To fake SG-buffer, I did this test: restore all the codes to the original, then added some codes in snd_malloc_sgbuf_pages() like below, the result is badly niose.
> > >
> > > >Not really what I meant.  Rather something like below (totally untested).
> > >
> > > [Hans :] I know what you mean now and complete code like below, but the result is still noise.
> >
> > >OK, so indeed the vmapped address does seem matter.  Interesting.
> > >What kind of user access does produce the noise?  Does it via aplay mmap mode, too?
> >
> > >In anyway, if the vmap is a problem, it might be worked around a patch like below (again totally untested and not sure whether it's correct).
> >
> > [Hans :] As I know, In the hardware layer HDAC’s stream have two data transport path : non-snoop & snoop; In the software layer ALSA-Driver have two data transport path : mmap & not mmap(test shows, it is dependent on wav's format or mmap_flag in aplay.c). When hardware at non-snoop mode, without hardware module's help, software must mark the mem to uncacheable type: when mmap used, the mark action happened at pcm_mmap_prepare(), the not-mmap mode's mark action happened at __mark_pages_wc(). And when at not-mmap mode, the vmapped address directly used in snd_pcm_lib_write_transfer() -> copy_from_user().
>
>
> >Well, that's usually no problem regarding that cache coherency.
> >At least it hasn't been any issue with Intel and AMD CPUs.
> >Does the problem happen with Intel CPU instead of VIA?
>
> [Hans :] I don't have enough Intel/AMD machine here, up to now, only find one Intel's : HW: mother board is Dell 042P49, HDA controller is 8086:1c20, codec is cx20641.

>And the same issue is seen on that machine?

[Hans :] Yes

Thanks,

Hans

> > Yes, of course, this path combinations(non-snoop + not-mmap) unlikely be used by most music players.
> >
> > [Hans :] I complete the patch in your last mail like below, not used the vunmap & re-vmap way(it has a little problem). Result is OK now(no noise anymore).
>
> >What problem did you get with vunmap / vmap?
>
> [Hans :] if patch like below, even the new vmapped address new_area be assigned to dmab->area, it will not use this time, but next. This time used dmab->area still the last one, but the old one had been vunmap, so Oops reported.

>That's bad.  But the problem isn't as you guessed, I guess.
>The allocator re-uses the pre-allocated buffer if available, and this makes things inconsistent.  I'll cook later.


保密声明:
本邮件含有保密或专有信息,仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
http://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread
* Re: A bug about cache inconsistency report
@ 2018-08-07  9:00 Hans Hu(SH-RD)
  2018-08-07  9:25 ` Takashi Iwai
  0 siblings, 1 reply; 10+ messages in thread
From: Hans Hu(SH-RD) @ 2018-08-07  9:00 UTC (permalink / raw)
  To: 'Takashi Iwai'
  Cc: Tony W. Wang(BJ-RD), 'alsa-devel@alsa-project.org',
	Tim Guo(BJ-RD), Annie Liu(BJ-RD)

> > > 1. noise relate test:
> > > When I annotate the return after set_pages_array_wc, then
> > > set_memory_wc works, noise disappear static void
> > > __mark_pages_wc(struct azx *chip, struct snd_dma_buffer *dmab,
> > > bool
> > > on) { ...
> > > set_pages_array_wb(sgbuf->page_table, sgbuf->pages); //return; }
> > > #endif
> >
> > > set_memory_uc() performs __pa() for the given address, and in this case, it's vmapped address.  Something bogus in your test, I'm afraid.
> >
> > [Hans:] I know __pa() handle the vmapped address is wrong, that why I don't know how to fix the problem, I try this because I see kernel older than 3.7.6. [bogus? pretending to be real or genuine?] no no no, but the noise problem machine(8086:1c20) in another city, I will make sure it again.
>
> >I meant about the reliability of the test result.  Or whether we're heading to a wrong fish.
> >In anyway, which kernel version are you testing?  And exactly what is your problem?  Which machine, which hardware setup (codec, etc) and which output, and how is the sound backend (PulseAudio or direct ALSA write)?
>
> [Hans:] OS: Ubuntu 17.10 + 4.13.0-21-generic (or any one after 3.7.6).
> HW: mother board is Dell 042P49, HDA controller is 8086:1c20, codec is cx20641.
> Output and sound backend: Except the command below, anything other is default setting.
>
> Problem(what I have done and found): When system setup, I add "modprobe snd-had-intel.snoop=0" in grub, after setup, execute command "aplay -D plughw:0,0 44.1k_16b_2c.wav", then I hear music with noise. The reason cause this is the buffer address should been mark as WC type and the __mark_pages_wc was patched for this, but if define CONFIG_SND_DMA_SGBUF, the actual marked address is the address before vmapped(0xffff88*...*), however the address used in snd_pcm_lib_write_transfer() is the address after vmapped(0xffffc9*...*), so the mark didn't work and cache inconsistency occur(noise).

>Well, basically the vmapped address isn't used *at all* on the HD-audio controller hardware.  The SG buffer addresses are passed on BDL of the chip, but the vmapped address is used only in PCM core for some data manipulation.  And this is purely software stuff, hence it shouldn't matter there whether it's in snoop mode or not.  The snoop mode difference appears mainly when accessed via mmap from user-space directly to kernel-space.
>That said, I guess your suspect heading to a wrong direction.

[Hans:] So, am I misunderstand? Or snoop has two different meanings? As I know, In the hardware layer HDAC’s stream have two data transport path : non-snoop & snoop; In the software layer ALSA-Driver have two data transport path : mmap & not mmap(test shows, it is dependent on wav's format or mmap_flag in aplay.c). When hardware at non-snoop mode, without hardware module's help, software must mark the mem to WC type: when mmap used, the mark action happened at pcm_mmap_prepare(), the not-mmap mode's mark action happened at __mark_pages_wc(). And when at not-mmap mode, the vmapped address directly used in snd_pcm_lib_write_transfer() -> copy_from_user().
Yes, of course, this path combinations(non-snoop + not-mmap) unlikely be used in most music players.


> > > And moreover, all Intel chipsets do work with snooping well, AFAIK.
> > [Hans:] yes, all Intel chipsets do work with snooping well here too. And the problem case here is "non-snoop"+"aplay -D plughw:*.*"+"44.1k 16bit"+"8086:1c20", I will see if I can find more machine do the test.
> >Ah, so you explicitly test with non-snoop mode for this chipset?
> >For what purpose...?
>
> [Hans:] Yes, I've been testing a HDA controller recently(predecessor is VIA), it has the same problem. I want to figure out if it is hardware's problem or not.
>
> [Hans:] I did the same test on another platform(8086:A170), it do work with non-snoop well.  In addition, I did such a change, oneliner, the purpose is to mark nothing, and it still do work with non-snoop well, this  supposed to be right?
>
> --- a/sound/pci/hda/hda_intel.c
> +++ b/sound/pci/hda/hda_intel.c
> #ifdef CONFIG_X86
> static void __mark_pages_wc(struct azx *chip, struct snd_dma_buffer
> *dmab, bool on) { int pages;
> +return;

>This is essentially other way round; it skips the non-cache setup.


> > > 2. change *_SG relate test:
> > > Just followed your proposal
> > > -snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_DEV_SG,
> > > +snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_DEV,
> >
> > >OK, you need one more oneliner:
> >
> > >diff --git a/sound/pci/hda/hda_controller.c
> > >b/sound/pci/hda/hda_controller.c index a12e594d4e3b..2eaed4956e7f
> > >100644
> > >--- a/sound/pci/hda/hda_controller.c
> > >+++ b/sound/pci/hda/hda_controller.c
> > >@@ -707,7 +707,7 @@ static const struct snd_pcm_ops azx_pcm_ops = {
> >  >.pointer = azx_pcm_pointer,
> >  >.get_time_info =  azx_get_time_info,  >.mmap = azx_pcm_mmap,
> > >-.page = snd_pcm_sgbuf_ops_page,
> > >+// .page = snd_pcm_sgbuf_ops_page,
> > > };
> >
> > > static void azx_pcm_free(struct snd_pcm *pcm)
> > [Hans :] I did this, and the Oops report as the same.
>
> >My bad, there are other SG-buffer codes in sound/hda/*, too.
>
> >So scratch these two oneliners.  The only thing you'd need is the big-hammer change (again oneliner :)
>
> >--- a/sound/core/Kconfig
> >+++ b/sound/core/Kconfig
> >@@ -181,7 +181,7 @@ config SND_VMASTER
>  >bool
> >
> > config SND_DMA_SGBUF
> >-def_bool y
> >+def_bool n
> > depends on X86
>
> > source "sound/core/seq/Kconfig"
>
> [Hans :] If this, All the platform are all right.

>Then the next step would be to fake sg-buffer from this straight
>buffer.  Revert the above, and modify sgbuf.c to the following:
>- Allocate a large continuous buffer
>- Assign each page in this large buffer

>If this still works, it's not about vmap, but it just means that the
>physically ordered pages do matter -- implicitly showing that the
>snooping behavior isn't properly turned on / off on the controller.

To fake SG-buffer, I did this test: restore all the codes to the original, then added some codes in snd_malloc_sgbuf_pages() like below, the result is badly niose.

---a/sound/core/sgbuf.c
+++b/sound/core/sgbuf.c
void *snd_malloc_sgbuf_pages(struct device *device,
     size_t size, struct snd_dma_buffer *dmab,
     size_t *res_size)
{
struct snd_sg_buf *sgbuf;
unsigned int i, pages, chunk, maxpages;
struct snd_dma_buffer tmpb;
struct snd_sg_page *table;
struct page **pgtable;

+size *= 2;  /* expand the buffer two times */

dmab->area = NULL;
dmab->addr = 0;

... ...

if (chunk < maxpages)
maxpages = chunk;
}

+for(i = 0; i < sgbuf->pages / 2; i ++)        /* restore size to the original it wanted */
+sgbuf->page_table[i] = sgbuf->page_table[i  * 2]; /* take a page every other page, just for test, ignore page release */
+size /= 2;
+sgbuf->pages /= 2;

sgbuf->size = size;
dmab->area = vmap(sgbuf->page_table, sgbuf->pages, VM_MAP, PAGE_KERNEL);


Thanks,

Hans



保密声明:
本邮件含有保密或专有信息,仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
http://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread
* Re: A bug about cache inconsistency report
@ 2018-08-02  8:22 Hans Hu(SH-RD)
  2018-08-02  8:50 ` Takashi Iwai
  0 siblings, 1 reply; 10+ messages in thread
From: Hans Hu(SH-RD) @ 2018-08-02  8:22 UTC (permalink / raw)
  To: 'Takashi Iwai'; +Cc: 'alsa-devel@alsa-project.org'

1. noise relate test:
8086:1C20   noise ;
8086:A170   ok;
8086:8D20   ok;
Seems this issue related to the HDA Card version

2. change *_SG relate test:
Both above platform report Oops

Thanks
Hans

-----邮件原件-----
发件人: Takashi Iwai [mailto:tiwai@suse.de]
发送时间: 01 August 2018 12:45
收件人: Hans Hu(SH-RD) <HansHu@zhaoxin.com>
抄送: 'alsa-devel@alsa-project.org' <alsa-devel@alsa-project.org>
主题: Re: [alsa-devel] A bug about cache inconsistency report

On Wed, 01 Aug 2018 04:05:18 +0200,
Hans Hu(SH-RD) wrote:
>
> I did more test base on different Intel platform, and result is the same.

Which platform did you test exactly?

> Maybe I made some mistake? Can you do the test in yours  platform?

The negative test result implies that the analysis isn't right.
Actually I haven't seen that problem on my test machines.

BTW, please avoid top-posting.


thanks,

Takashi

>
> Thank you!
> Hans
>
> -----邮件原件-----
> 发件人: Takashi Iwai [mailto:tiwai@suse.de]
> 发送时间: 31 July 2018 21:55
> 收件人: Hans Hu(SH-RD) <HansHu@zhaoxin.com>
> 抄送: 'alsa-devel@alsa-project.org' <alsa-devel@alsa-project.org>
> 主题: Re: 答复: [alsa-devel] A bug about cache inconsistency report
>
> On Tue, 31 Jul 2018 15:51:06 +0200,
> Hans Hu(SH-RD) wrote:
> >
> > I did the test and Oops report when install driver( both snoop and
> > non-snoop mode)
>
> I don't understand.  Did my one-liner change the behavior or not?
>
>
> thanks,
>
> Takashi
>
> >
> > -----邮件原件-----
> > 发件人: Takashi Iwai [mailto:tiwai@suse.de]
> > 发送时间: 31 July 2018 19:59
> > 收件人: Hans Hu(SH-RD) <HansHu@zhaoxin.com>
> > 抄送: 'alsa-devel@alsa-project.org' <alsa-devel@alsa-project.org>
> > 主题: Re: [alsa-devel] A bug about cache inconsistency report
> >
> > On Tue, 31 Jul 2018 12:52:04 +0200,
> > Hans Hu(SH-RD) wrote:
> > >
> > > Hi All,
> > >
> > >                Try send an email.
> > >
> > >                Appearance:
> > > when use non-snoop + ioctl path combination, an issue occurs:
> > > usage: aplay -D plughw:0,0 44100_S16_LE_2c.wav Ubuntu, it
> > > performance intermittently noise;
> > >
> > > Root case:
> > >                Function set_pages_array_wc marked the PTE of the original address; Function set_memory_wc marked the PTE both original and after vmaped address.
> > > [cid:image002.png@01D428FF.92621520]
> > > However, cache or not in transmission process is based on the PTE of the later, and the set_pages_array_wc did't mark it, so a cache inconsistency issue occurs.
> > >
> > >                Related bug:
> > >                SHA1 ID: 9ddf1aeb2134e72275c97a2c6ff2e3eb04f2f27a
> >
> > OK, so did you confirm that dropping the SG buffer support "fixes" the noise problem on your system?  The easiest test would be like:
> >
> > --- a/sound/pci/hda/hda_controller.c
> > +++ b/sound/pci/hda/hda_controller.c
> > @@ -770,7 +770,7 @@ int snd_hda_attach_pcm_stream(struct hda_bus
> > *_bus, struct hda_codec *codec,  size = CONFIG_SND_HDA_PREALLOC_SIZE
> > * 1024;  if (size > MAX_PREALLOC_SIZE)  size = MAX_PREALLOC_SIZE;
> > -snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_DEV_SG,
> > +snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_DEV,
> >        chip->card->dev,
> >        size, MAX_PREALLOC_SIZE);
> >  return 0;
> >
> >
> > thanks,
> >
> > Takashi
> >
> >
> > 保密声明:
> > 本邮件含有保密或专有信息,仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
> > CONFIDENTIAL NOTE:
> > This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
>
>
> 保密声明:
> 本邮件含有保密或专有信息,仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
> CONFIDENTIAL NOTE:
> This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.


保密声明:
本邮件含有保密或专有信息,仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
_______________________________________________
Alsa-devel mailing list
Alsa-devel@alsa-project.org
http://mailman.alsa-project.org/mailman/listinfo/alsa-devel

^ permalink raw reply	[flat|nested] 10+ messages in thread
* A bug about cache inconsistency report
@ 2018-07-31 10:52 Hans Hu(SH-RD)
  2018-07-31 11:59 ` Takashi Iwai
  0 siblings, 1 reply; 10+ messages in thread
From: Hans Hu(SH-RD) @ 2018-07-31 10:52 UTC (permalink / raw)
  To: 'alsa-devel@alsa-project.org'


[-- Attachment #1.1: Type: text/plain, Size: 1058 bytes --]

Hi All,

               Try send an email.

               Appearance:
when use non-snoop + ioctl path combination, an issue occurs:
usage: aplay -D plughw:0,0 44100_S16_LE_2c.wav
Ubuntu, it performance intermittently noise;

Root case:
               Function set_pages_array_wc marked the PTE of the original address; Function set_memory_wc marked the PTE both original and after vmaped address.
[cid:image002.png@01D428FF.92621520]
However, cache or not in transmission process is based on the PTE of the later, and the set_pages_array_wc did't mark it, so a cache inconsistency issue occurs.

               Related bug:
               SHA1 ID: 9ddf1aeb2134e72275c97a2c6ff2e3eb04f2f27a

Best Regards!
Hans Hu



?????
?????????????????????????????????????????????????????
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.

[-- Attachment #1.2: image002.png --]
[-- Type: image/png, Size: 11070 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2018-08-10 15:15 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-01  2:05 A bug about cache inconsistency report Hans Hu(SH-RD)
2018-08-01  4:45 ` Takashi Iwai
  -- strict thread matches above, loose matches on Subject: below --
2018-08-08 11:07 Hans Hu(SH-RD)
2018-08-10 15:15 ` Takashi Iwai
2018-08-07  9:00 Hans Hu(SH-RD)
2018-08-07  9:25 ` Takashi Iwai
2018-08-02  8:22 Hans Hu(SH-RD)
2018-08-02  8:50 ` Takashi Iwai
2018-07-31 10:52 Hans Hu(SH-RD)
2018-07-31 11:59 ` Takashi Iwai

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.