* [PATCH] arm64: mmu: no write cache for O_SYNC flag @ 2020-03-26 16:36 Li Wang 2020-03-26 16:55 ` Catalin Marinas 2020-03-27 14:29 ` Mark Rutland 0 siblings, 2 replies; 6+ messages in thread From: Li Wang @ 2020-03-26 16:36 UTC (permalink / raw) To: Catalin Marinas; +Cc: li.wang, Will Deacon, linux-arm-kernel, linux-kernel reproduce steps: 1. disable CONFIG_STRICT_DEVMEM in linux kernel 2. Process A gets a Physical Address of global variable by "/proc/self/pagemap". 3. Process B writes a value to the same Physical Address by mmap(): fd=open("/dev/mem",O_SYNC); Virtual Address=mmap(fd); problem symptom: after Process B write a value to the Physical Address, Process A of the value of global variable does not change. They both W/R the same Physical Address. technical reason: Process B writing the Physical Address is by the Virtual Address, and the Virtual Address comes from "/dev/mem" and mmap(). In arm64 arch, the Virtual Address has write cache. So, maybe the value is not written into Physical Address. fix reason: giving write cache flag in arm64 is in phys_mem_access_prot(): ===== arch/arm64/mm/mmu.c phys_mem_access_prot() { if (!pfn_valid(pfn)) return pgprot_noncached(vma_prot); else if (file->f_flags & O_SYNC) return pgprot_writecombine(vma_prot); return vma_prot; } ==== the other arch and the share function drivers/char/mem.c of phys_mem_access_prot() does not add write cache flag. So, removing the flag to fix the issue Signed-off-by: Li Wang <li.wang@windriver.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org --- arch/arm64/mm/mmu.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 128f70852bf3..d7083965ca17 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -81,8 +81,6 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, { if (!pfn_valid(pfn)) return pgprot_noncached(vma_prot); - else if (file->f_flags & O_SYNC) - return pgprot_writecombine(vma_prot); return vma_prot; } EXPORT_SYMBOL(phys_mem_access_prot); -- 2.24.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] arm64: mmu: no write cache for O_SYNC flag 2020-03-26 16:36 [PATCH] arm64: mmu: no write cache for O_SYNC flag Li Wang @ 2020-03-26 16:55 ` Catalin Marinas 2020-03-26 17:34 ` Wang, Li 2020-03-27 14:29 ` Mark Rutland 1 sibling, 1 reply; 6+ messages in thread From: Catalin Marinas @ 2020-03-26 16:55 UTC (permalink / raw) To: Li Wang; +Cc: Will Deacon, linux-arm-kernel, linux-kernel On Thu, Mar 26, 2020 at 09:36:25AM -0700, Li Wang wrote: > reproduce steps: > 1. > disable CONFIG_STRICT_DEVMEM in linux kernel > 2. > Process A gets a Physical Address of global variable by > "/proc/self/pagemap". > 3. > Process B writes a value to the same Physical Address by mmap(): > fd=open("/dev/mem",O_SYNC); > Virtual Address=mmap(fd); > > problem symptom: > after Process B write a value to the Physical Address, > Process A of the value of global variable does not change. > They both W/R the same Physical Address. > > technical reason: > Process B writing the Physical Address is by the Virtual Address, > and the Virtual Address comes from "/dev/mem" and mmap(). > In arm64 arch, the Virtual Address has write cache. > So, maybe the value is not written into Physical Address. > > fix reason: > giving write cache flag in arm64 is in phys_mem_access_prot(): > ===== > arch/arm64/mm/mmu.c > phys_mem_access_prot() > { > if (!pfn_valid(pfn)) > return pgprot_noncached(vma_prot); > else if (file->f_flags & O_SYNC) > return pgprot_writecombine(vma_prot); > return vma_prot; > } > ==== > the other arch and the share function drivers/char/mem.c of phys_mem_access_prot() > does not add write cache flag. > So, removing the flag to fix the issue Other architectures may have transparent caches and don't require different attributes. > Signed-off-by: Li Wang <li.wang@windriver.com> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Will Deacon <will@kernel.org> > Cc: linux-arm-kernel@lists.infradead.org > Cc: linux-kernel@vger.kernel.org > --- > arch/arm64/mm/mmu.c | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c > index 128f70852bf3..d7083965ca17 100644 > --- a/arch/arm64/mm/mmu.c > +++ b/arch/arm64/mm/mmu.c > @@ -81,8 +81,6 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, > { > if (!pfn_valid(pfn)) > return pgprot_noncached(vma_prot); > - else if (file->f_flags & O_SYNC) > - return pgprot_writecombine(vma_prot); > return vma_prot; > } > EXPORT_SYMBOL(phys_mem_access_prot); A better solution is for user space not to pass O_SYNC when opening /dev/mem. We've had this ABI for a long time (arch/arm/ and several other architectures do the same), why change it now? -- Catalin ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] arm64: mmu: no write cache for O_SYNC flag 2020-03-26 16:55 ` Catalin Marinas @ 2020-03-26 17:34 ` Wang, Li 0 siblings, 0 replies; 6+ messages in thread From: Wang, Li @ 2020-03-26 17:34 UTC (permalink / raw) To: Catalin Marinas; +Cc: Will Deacon, linux-arm-kernel, linux-kernel 在 2020/3/27 0:55, Catalin Marinas 写道: > On Thu, Mar 26, 2020 at 09:36:25AM -0700, Li Wang wrote: >> reproduce steps: >> 1. >> disable CONFIG_STRICT_DEVMEM in linux kernel >> 2. >> Process A gets a Physical Address of global variable by >> "/proc/self/pagemap". >> 3. >> Process B writes a value to the same Physical Address by mmap(): >> fd=open("/dev/mem",O_SYNC); >> Virtual Address=mmap(fd); >> >> problem symptom: >> after Process B write a value to the Physical Address, >> Process A of the value of global variable does not change. >> They both W/R the same Physical Address. >> >> technical reason: >> Process B writing the Physical Address is by the Virtual Address, >> and the Virtual Address comes from "/dev/mem" and mmap(). >> In arm64 arch, the Virtual Address has write cache. >> So, maybe the value is not written into Physical Address. >> >> fix reason: >> giving write cache flag in arm64 is in phys_mem_access_prot(): >> ===== >> arch/arm64/mm/mmu.c >> phys_mem_access_prot() >> { >> if (!pfn_valid(pfn)) >> return pgprot_noncached(vma_prot); >> else if (file->f_flags & O_SYNC) >> return pgprot_writecombine(vma_prot); >> return vma_prot; >> } >> ==== >> the other arch and the share function drivers/char/mem.c of phys_mem_access_prot() >> does not add write cache flag. >> So, removing the flag to fix the issue > Other architectures may have transparent caches and don't require > different attributes. > >> Signed-off-by: Li Wang <li.wang@windriver.com> >> Cc: Catalin Marinas <catalin.marinas@arm.com> >> Cc: Will Deacon <will@kernel.org> >> Cc: linux-arm-kernel@lists.infradead.org >> Cc: linux-kernel@vger.kernel.org >> --- >> arch/arm64/mm/mmu.c | 2 -- >> 1 file changed, 2 deletions(-) >> >> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c >> index 128f70852bf3..d7083965ca17 100644 >> --- a/arch/arm64/mm/mmu.c >> +++ b/arch/arm64/mm/mmu.c >> @@ -81,8 +81,6 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, >> { >> if (!pfn_valid(pfn)) >> return pgprot_noncached(vma_prot); >> - else if (file->f_flags & O_SYNC) >> - return pgprot_writecombine(vma_prot); >> return vma_prot; >> } >> EXPORT_SYMBOL(phys_mem_access_prot); > A better solution is for user space not to pass O_SYNC when opening > /dev/mem. We've had this ABI for a long time (arch/arm/ and several > other architectures do the same), why change it now? 1. no pass O_SYNC in user space is not a good idea. in fact, the codes come from 'devmem' command of busybox: ===== busybox-1.24.1/miscutils$ vim devmem.c fd = xopen("/dev/mem", O_SYNC); ===== the codes are used for a long time. 2. according to info of open man about "O_SYNC": ===== http://man7.org/linux/man-pages/man2/open.2.html the output data and associated file metadata have been transferred to the underlying hardware ===== I think "O_SYNC" means no cache. 3. /dev/mem of driver offers 2 ways to operate physical memory. one is mmap, the other is read/write. when use read/write way, it operates uncached memory: ===== kernel-source/drivers/char/mem.c write_mem(){ /* it must also be accessed uncached */ } ===== 4. arm64 arch is different with other arch about phys_mem_access_prot(). you can see no any other arch add cache flag in the function. only arm and arm64 add write cache for O_SYNC flag. x86/mm/pat.c phys_mem_access_prot(){ return vma_prot; } powerpc/mm/mem.c phys_mem_access_prot(){ if (ppc_md.phys_mem_access_prot) return ppc_md.phys_mem_access_prot(file, pfn, size, vma_prot); if (!page_is_ram(pfn)) vma_prot = pgprot_noncached(vma_prot); return vma_prot; } drivers/char/mem.c phys_mem_access_prot() { #ifdef pgprot_noncached phys_addr_t offset = pfn << PAGE_SHIFT; if (uncached_access(file, offset)) return pgprot_noncached(vma_prot); #endif return vma_prot; } Thanks, LiWang. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] arm64: mmu: no write cache for O_SYNC flag 2020-03-26 16:36 [PATCH] arm64: mmu: no write cache for O_SYNC flag Li Wang 2020-03-26 16:55 ` Catalin Marinas @ 2020-03-27 14:29 ` Mark Rutland 2020-03-27 16:47 ` Wang, Li 1 sibling, 1 reply; 6+ messages in thread From: Mark Rutland @ 2020-03-27 14:29 UTC (permalink / raw) To: Li Wang; +Cc: Catalin Marinas, Will Deacon, linux-arm-kernel, linux-kernel On Thu, Mar 26, 2020 at 09:36:25AM -0700, Li Wang wrote: > reproduce steps: > 1. > disable CONFIG_STRICT_DEVMEM in linux kernel > 2. > Process A gets a Physical Address of global variable by > "/proc/self/pagemap". > 3. > Process B writes a value to the same Physical Address by mmap(): > fd=open("/dev/mem",O_SYNC); > Virtual Address=mmap(fd); Is this just to demonstrate the behaviour, or is this meant to be indicative of a real use-case? I'm struggling to see the latter. > problem symptom: > after Process B write a value to the Physical Address, > Process A of the value of global variable does not change. > They both W/R the same Physical Address. If Process A is not using the same attributes as process B, there is no guarantee of coherency. How did process A map this memory? > technical reason: > Process B writing the Physical Address is by the Virtual Address, > and the Virtual Address comes from "/dev/mem" and mmap(). > In arm64 arch, the Virtual Address has write cache. > So, maybe the value is not written into Physical Address. I don't think that's true. I think what's happening here is: * Process A has a Normal WBWA Cacheable mapping. * Process B as a Normal Non-cacheable mapping. * Process B's write does not snoop any caches, and goes straight to memory. * Process A reads a value from cache, which does not include process B's write. That's a natural result of using mismatched attributes, and is consistent with the O_SYNC flag meaning that the write "is transferred to the underlying hardware". > > fix reason: > giving write cache flag in arm64 is in phys_mem_access_prot(): > ===== > arch/arm64/mm/mmu.c > phys_mem_access_prot() > { > if (!pfn_valid(pfn)) > return pgprot_noncached(vma_prot); > else if (file->f_flags & O_SYNC) > return pgprot_writecombine(vma_prot); > return vma_prot; > } > ==== > the other arch and the share function drivers/char/mem.c of phys_mem_access_prot() > does not add write cache flag. > So, removing the flag to fix the issue This will change behaviour that other software may be relying upon, and as above I do not believe this actually solves the problem you describe. Thanks, Mark. > > Signed-off-by: Li Wang <li.wang@windriver.com> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Will Deacon <will@kernel.org> > Cc: linux-arm-kernel@lists.infradead.org > Cc: linux-kernel@vger.kernel.org > --- > arch/arm64/mm/mmu.c | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c > index 128f70852bf3..d7083965ca17 100644 > --- a/arch/arm64/mm/mmu.c > +++ b/arch/arm64/mm/mmu.c > @@ -81,8 +81,6 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, > { > if (!pfn_valid(pfn)) > return pgprot_noncached(vma_prot); > - else if (file->f_flags & O_SYNC) > - return pgprot_writecombine(vma_prot); > return vma_prot; > } > EXPORT_SYMBOL(phys_mem_access_prot); > -- > 2.24.1 > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] arm64: mmu: no write cache for O_SYNC flag 2020-03-27 14:29 ` Mark Rutland @ 2020-03-27 16:47 ` Wang, Li 2020-03-27 17:02 ` Mark Rutland 0 siblings, 1 reply; 6+ messages in thread From: Wang, Li @ 2020-03-27 16:47 UTC (permalink / raw) To: Mark Rutland; +Cc: Catalin Marinas, Will Deacon, linux-arm-kernel, linux-kernel [-- Attachment #1: Type: text/plain, Size: 5804 bytes --] 在 2020/3/27 22:29, Mark Rutland 写道: > On Thu, Mar 26, 2020 at 09:36:25AM -0700, Li Wang wrote: >> reproduce steps: >> 1. >> disable CONFIG_STRICT_DEVMEM in linux kernel >> 2. >> Process A gets a Physical Address of global variable by >> "/proc/self/pagemap". >> 3. >> Process B writes a value to the same Physical Address by mmap(): >> fd=open("/dev/mem",O_SYNC); >> Virtual Address=mmap(fd); > Is this just to demonstrate the behaviour, or is this meant to be > indicative of a real use-case? I'm struggling to see the latter. > >> problem symptom: >> after Process B write a value to the Physical Address, >> Process A of the value of global variable does not change. >> They both W/R the same Physical Address. > If Process A is not using the same attributes as process B, there is no > guarantee of coherency. How did process A map this memory? about 2 Process: Process A: the memory is not declared by map function, it is just a global variable. only by /proc/self/pagemap to get its Physical Address. I attached the codes(wrl-cache-coh-test.c) Process B: it is command of "devmem" in busybox, it writes a value to Physical Address. it uses open(O_SYNC) and mmap. >> technical reason: >> Process B writing the Physical Address is by the Virtual Address, >> and the Virtual Address comes from "/dev/mem" and mmap(). >> In arm64 arch, the Virtual Address has write cache. >> So, maybe the value is not written into Physical Address. > I don't think that's true. I think what's happening here is: > > * Process A has a Normal WBWA Cacheable mapping. > * Process B as a Normal Non-cacheable mapping. > * Process B's write does not snoop any caches, and goes straight to > memory. > * Process A reads a value from cache, which does not include process B's > write. > > That's a natural result of using mismatched attributes, and is > consistent with the O_SYNC flag meaning that the write "is transferred > to the underlying hardware". if you agree that O_SYNC flag means "is transferred to the underlying hardware", the arm64 does not do that: when use O_SYNC flag under arm64 arch, it adds write cache feature, so, it is no guarantee "transferred to hardware". ===== arch/arm64/mm/mmu.c phys_mem_access_prot(){ else if (file->f_flags & O_SYNC) return pgprot_writecombine(vma_prot);} ===== by my test without the write cache, even if Process A is not using the same attributes as process B, it has guarantee of coherency: when Process B change value, Process B can see the change, too. Thanks, LiWang. my email server seems to reject to send to linux-arm-kernel@lists.infradead.org, the info is in another email not showing in linux-arm-kernel@lists.infradead.org: 1. no pass O_SYNC in user space is not a good idea. in fact, the codes come from 'devmem' command of busybox: ===== busybox-1.24.1/miscutils$ vim devmem.c fd = xopen("/dev/mem", O_SYNC); ===== the codes are used for a long time. 2. according to info of open man about "O_SYNC": ===== http://man7.org/linux/man-pages/man2/open.2.html the output data and associated file metadata have been transferred to the underlying hardware ===== I think "O_SYNC" means no cache. 3. /dev/mem of driver offers 2 ways to operate physical memory. one is mmap, the other is read/write. when use read/write way, it operates uncached memory: ===== kernel-source/drivers/char/mem.c write_mem(){ /* it must also be accessed uncached */ } ===== 4. arm64 arch is different with other arch about phys_mem_access_prot(). you can see no any other arch add cache flag in the function. only arm and arm64 add write cache for O_SYNC flag. x86/mm/pat.c phys_mem_access_prot(){ return vma_prot; } powerpc/mm/mem.c phys_mem_access_prot(){ if (ppc_md.phys_mem_access_prot) return ppc_md.phys_mem_access_prot(file, pfn, size, vma_prot); if (!page_is_ram(pfn)) vma_prot = pgprot_noncached(vma_prot); return vma_prot; } drivers/char/mem.c phys_mem_access_prot() { #ifdef pgprot_noncached phys_addr_t offset = pfn << PAGE_SHIFT; if (uncached_access(file, offset)) return pgprot_noncached(vma_prot); #endif return vma_prot; } >> fix reason: >> giving write cache flag in arm64 is in phys_mem_access_prot(): >> ===== >> arch/arm64/mm/mmu.c >> phys_mem_access_prot() >> { >> if (!pfn_valid(pfn)) >> return pgprot_noncached(vma_prot); >> else if (file->f_flags & O_SYNC) >> return pgprot_writecombine(vma_prot); >> return vma_prot; >> } >> ==== >> the other arch and the share function drivers/char/mem.c of phys_mem_access_prot() >> does not add write cache flag. >> So, removing the flag to fix the issue > This will change behaviour that other software may be relying upon, and > as above I do not believe this actually solves the problem you describe. > > Thanks, > Mark. > >> Signed-off-by: Li Wang <li.wang@windriver.com> >> Cc: Catalin Marinas <catalin.marinas@arm.com> >> Cc: Will Deacon <will@kernel.org> >> Cc: linux-arm-kernel@lists.infradead.org >> Cc: linux-kernel@vger.kernel.org >> --- >> arch/arm64/mm/mmu.c | 2 -- >> 1 file changed, 2 deletions(-) >> >> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c >> index 128f70852bf3..d7083965ca17 100644 >> --- a/arch/arm64/mm/mmu.c >> +++ b/arch/arm64/mm/mmu.c >> @@ -81,8 +81,6 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, >> { >> if (!pfn_valid(pfn)) >> return pgprot_noncached(vma_prot); >> - else if (file->f_flags & O_SYNC) >> - return pgprot_writecombine(vma_prot); >> return vma_prot; >> } >> EXPORT_SYMBOL(phys_mem_access_prot); >> -- >> 2.24.1 >> [-- Attachment #2: cache-test.c --] [-- Type: text/plain, Size: 1194 bytes --] #include <stdio.h> #include <stdint.h> #include <unistd.h> static uintptr_t virt_to_phys_address(uintptr_t vaddr) { FILE *pagemap; uintptr_t paddr = 0; off_t offset = (vaddr / sysconf(_SC_PAGESIZE)) * sizeof(uint64_t); uint64_t e; /* https://www.kernel.org/doc/Documentation/vm/pagemap.txt */ if ((pagemap = fopen("/proc/self/pagemap", "r"))) { if (lseek(fileno(pagemap), offset, SEEK_SET) == offset) { if (fread(&e, sizeof(uint64_t), 1, pagemap)) { if (e & (1ULL << 63)) { /* page present ? */ /* pfn mask */ paddr = e & ((1ULL << 54) - 1); paddr = paddr * sysconf(_SC_PAGESIZE); /* add offset within page */ paddr |= (vaddr & (sysconf(_SC_PAGESIZE) - 1)); } else printf("%s: No page present\n", __func__); } else printf("%s: fread failed\n", __func__); } else printf("%s: lseek did not find\n", __func__); fclose(pagemap); } else printf("%s: Pagemap open failed\n", __func__); return paddr; } volatile uint32_t var=0; int main() { void* phys_addr = virt_to_phys_address(&var); printf("%p %p\n", &var, phys_addr); while( var==0 ) sleep(1); printf("done\n"); fflush(stdout); return 0; } ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] arm64: mmu: no write cache for O_SYNC flag 2020-03-27 16:47 ` Wang, Li @ 2020-03-27 17:02 ` Mark Rutland 0 siblings, 0 replies; 6+ messages in thread From: Mark Rutland @ 2020-03-27 17:02 UTC (permalink / raw) To: Wang, Li; +Cc: Catalin Marinas, Will Deacon, linux-arm-kernel, linux-kernel On Sat, Mar 28, 2020 at 12:47:32AM +0800, Wang, Li wrote: > > 在 2020/3/27 22:29, Mark Rutland 写道: > > On Thu, Mar 26, 2020 at 09:36:25AM -0700, Li Wang wrote: > > > reproduce steps: > > > 1. > > > disable CONFIG_STRICT_DEVMEM in linux kernel > > > 2. > > > Process A gets a Physical Address of global variable by > > > "/proc/self/pagemap". > > > 3. > > > Process B writes a value to the same Physical Address by mmap(): > > > fd=open("/dev/mem",O_SYNC); > > > Virtual Address=mmap(fd); > > Is this just to demonstrate the behaviour, or is this meant to be > > indicative of a real use-case? I'm struggling to see the latter. > > > > > problem symptom: > > > after Process B write a value to the Physical Address, > > > Process A of the value of global variable does not change. > > > They both W/R the same Physical Address. > > If Process A is not using the same attributes as process B, there is no > > guarantee of coherency. How did process A map this memory? > > > about 2 Process: > > Process A: > > the memory is not declared by map function, it is just a global variable. Then it is exactly as I described previously, and Process A has it mapped with a Normal Write-Back Cacheable mappping. Process B requests a mapping of that memory via /dev/mem. It passes the O_SYNC flag, and to ensure that accesses go to "the underlying hardware" the kernel makes this mapping Normal Non-Cacheable (which means it should not look in a cache, or be allocated into one). The two mappings are not coherent because process A uses the cache, but process B does not. This is the expected behaviour, consistent with the semantic of O_SYNC. If you need the two to be coherent, they must both use the same attributes. Process B can be coherent with process A if it does *not* pass O_SYNC, which would give it a Normal Write-Back Cacheable mapping that was coherent with process A. > if you agree that O_SYNC flag means "is transferred to the underlying > hardware", > > the arm64 does not do that: > > when use O_SYNC flag under arm64 arch, it adds write cache feature, As above, this is not the case. O_SYNC causes the kernel to use a non-cacheable mapping, where it would normally create a cacheable mapping. i.e. O_SYNC *removes* cacheability. It just happens that process A is using a cacheable mapping, which is the case regardless of what process B does. Thanks, Mark. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-03-27 17:02 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-03-26 16:36 [PATCH] arm64: mmu: no write cache for O_SYNC flag Li Wang 2020-03-26 16:55 ` Catalin Marinas 2020-03-26 17:34 ` Wang, Li 2020-03-27 14:29 ` Mark Rutland 2020-03-27 16:47 ` Wang, Li 2020-03-27 17:02 ` Mark Rutland
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).