All of lore.kernel.org
 help / color / mirror / Atom feed
From: Walter Wu <walter-zh.wu@mediatek.com>
To: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>,
	Alexander Potapenko <glider@google.com>,
	Matthias Brugger <matthias.bgg@gmail.com>,
	LKML <linux-kernel@vger.kernel.org>,
	kasan-dev <kasan-dev@googlegroups.com>,
	Linux-MM <linux-mm@kvack.org>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>,
	<linux-mediatek@lists.infradead.org>,
	wsd_upstream <wsd_upstream@mediatek.com>
Subject: Re: [PATCH] kasan: fix the missing underflow in memmove and memcpy with CONFIG_KASAN_GENERIC=y
Date: Thu, 3 Oct 2019 17:38:45 +0800	[thread overview]
Message-ID: <1570095525.19702.59.camel@mtksdccf07> (raw)
In-Reply-To: <CACT4Y+ZwNv2-QBrvuR2JvemovmKPQ9Ggrr=ZkdTg6xy_Ki6UAg@mail.gmail.com>

On Thu, 2019-10-03 at 08:26 +0200, Dmitry Vyukov wrote:
> On Thu, Oct 3, 2019 at 4:18 AM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> >
> > On Wed, 2019-10-02 at 15:57 +0200, Dmitry Vyukov wrote:
> > > On Wed, Oct 2, 2019 at 2:15 PM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > >
> > > > On Mon, 2019-09-30 at 12:36 +0800, Walter Wu wrote:
> > > > > On Fri, 2019-09-27 at 21:41 +0200, Dmitry Vyukov wrote:
> > > > > > On Fri, Sep 27, 2019 at 4:22 PM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > > > > >
> > > > > > > On Fri, 2019-09-27 at 15:07 +0200, Dmitry Vyukov wrote:
> > > > > > > > On Fri, Sep 27, 2019 at 5:43 AM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > > > > > > >
> > > > > > > > > memmove() and memcpy() have missing underflow issues.
> > > > > > > > > When -7 <= size < 0, then KASAN will miss to catch the underflow issue.
> > > > > > > > > It looks like shadow start address and shadow end address is the same,
> > > > > > > > > so it does not actually check anything.
> > > > > > > > >
> > > > > > > > > The following test is indeed not caught by KASAN:
> > > > > > > > >
> > > > > > > > >         char *p = kmalloc(64, GFP_KERNEL);
> > > > > > > > >         memset((char *)p, 0, 64);
> > > > > > > > >         memmove((char *)p, (char *)p + 4, -2);
> > > > > > > > >         kfree((char*)p);
> > > > > > > > >
> > > > > > > > > It should be checked here:
> > > > > > > > >
> > > > > > > > > void *memmove(void *dest, const void *src, size_t len)
> > > > > > > > > {
> > > > > > > > >         check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > > > > > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > > > > > > >
> > > > > > > > >         return __memmove(dest, src, len);
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > We fix the shadow end address which is calculated, then generic KASAN
> > > > > > > > > get the right shadow end address and detect this underflow issue.
> > > > > > > > >
> > > > > > > > > [1] https://bugzilla.kernel.org/show_bug.cgi?id=199341
> > > > > > > > >
> > > > > > > > > Signed-off-by: Walter Wu <walter-zh.wu@mediatek.com>
> > > > > > > > > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > > > > > > > > ---
> > > > > > > > >  lib/test_kasan.c   | 36 ++++++++++++++++++++++++++++++++++++
> > > > > > > > >  mm/kasan/generic.c |  8 ++++++--
> > > > > > > > >  2 files changed, 42 insertions(+), 2 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/lib/test_kasan.c b/lib/test_kasan.c
> > > > > > > > > index b63b367a94e8..8bd014852556 100644
> > > > > > > > > --- a/lib/test_kasan.c
> > > > > > > > > +++ b/lib/test_kasan.c
> > > > > > > > > @@ -280,6 +280,40 @@ static noinline void __init kmalloc_oob_in_memset(void)
> > > > > > > > >         kfree(ptr);
> > > > > > > > >  }
> > > > > > > > >
> > > > > > > > > +static noinline void __init kmalloc_oob_in_memmove_underflow(void)
> > > > > > > > > +{
> > > > > > > > > +       char *ptr;
> > > > > > > > > +       size_t size = 64;
> > > > > > > > > +
> > > > > > > > > +       pr_info("underflow out-of-bounds in memmove\n");
> > > > > > > > > +       ptr = kmalloc(size, GFP_KERNEL);
> > > > > > > > > +       if (!ptr) {
> > > > > > > > > +               pr_err("Allocation failed\n");
> > > > > > > > > +               return;
> > > > > > > > > +       }
> > > > > > > > > +
> > > > > > > > > +       memset((char *)ptr, 0, 64);
> > > > > > > > > +       memmove((char *)ptr, (char *)ptr + 4, -2);
> > > > > > > > > +       kfree(ptr);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static noinline void __init kmalloc_oob_in_memmove_overflow(void)
> > > > > > > > > +{
> > > > > > > > > +       char *ptr;
> > > > > > > > > +       size_t size = 64;
> > > > > > > > > +
> > > > > > > > > +       pr_info("overflow out-of-bounds in memmove\n");
> > > > > > > > > +       ptr = kmalloc(size, GFP_KERNEL);
> > > > > > > > > +       if (!ptr) {
> > > > > > > > > +               pr_err("Allocation failed\n");
> > > > > > > > > +               return;
> > > > > > > > > +       }
> > > > > > > > > +
> > > > > > > > > +       memset((char *)ptr, 0, 64);
> > > > > > > > > +       memmove((char *)ptr + size, (char *)ptr, 2);
> > > > > > > > > +       kfree(ptr);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > >  static noinline void __init kmalloc_uaf(void)
> > > > > > > > >  {
> > > > > > > > >         char *ptr;
> > > > > > > > > @@ -734,6 +768,8 @@ static int __init kmalloc_tests_init(void)
> > > > > > > > >         kmalloc_oob_memset_4();
> > > > > > > > >         kmalloc_oob_memset_8();
> > > > > > > > >         kmalloc_oob_memset_16();
> > > > > > > > > +       kmalloc_oob_in_memmove_underflow();
> > > > > > > > > +       kmalloc_oob_in_memmove_overflow();
> > > > > > > > >         kmalloc_uaf();
> > > > > > > > >         kmalloc_uaf_memset();
> > > > > > > > >         kmalloc_uaf2();
> > > > > > > > > diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
> > > > > > > > > index 616f9dd82d12..34ca23d59e67 100644
> > > > > > > > > --- a/mm/kasan/generic.c
> > > > > > > > > +++ b/mm/kasan/generic.c
> > > > > > > > > @@ -131,9 +131,13 @@ static __always_inline bool memory_is_poisoned_n(unsigned long addr,
> > > > > > > > >                                                 size_t size)
> > > > > > > > >  {
> > > > > > > > >         unsigned long ret;
> > > > > > > > > +       void *shadow_start = kasan_mem_to_shadow((void *)addr);
> > > > > > > > > +       void *shadow_end = kasan_mem_to_shadow((void *)addr + size - 1) + 1;
> > > > > > > > >
> > > > > > > > > -       ret = memory_is_nonzero(kasan_mem_to_shadow((void *)addr),
> > > > > > > > > -                       kasan_mem_to_shadow((void *)addr + size - 1) + 1);
> > > > > > > > > +       if ((long)size < 0)
> > > > > > > > > +               shadow_end = kasan_mem_to_shadow((void *)addr + size);
> > > > > > > >
> > > > > > > > Hi Walter,
> > > > > > > >
> > > > > > > > Thanks for working on this.
> > > > > > > >
> > > > > > > > If size<0, does it make sense to continue at all? We will still check
> > > > > > > > 1PB of shadow memory? What happens when we pass such huge range to
> > > > > > > > memory_is_nonzero?
> > > > > > > > Perhaps it's better to produce an error and bail out immediately if size<0?
> > > > > > >
> > > > > > > I agree with what you said. when size<0, it is indeed an unreasonable
> > > > > > > behavior, it should be blocked from continuing to do.
> > > > > > >
> > > > > > >
> > > > > > > > Also, what's the failure mode of the tests? Didn't they badly corrupt
> > > > > > > > memory? We tried to keep tests such that they produce the KASAN
> > > > > > > > reports, but don't badly corrupt memory b/c/ we need to run all of
> > > > > > > > them.
> > > > > > >
> > > > > > > Maybe we should first produce KASAN reports and then go to execute
> > > > > > > memmove() or do nothing? It looks like it’s doing the following.or?
> > > > > > >
> > > > > > > void *memmove(void *dest, const void *src, size_t len)
> > > > > > >  {
> > > > > > > +       if (long(len) <= 0)
> > > > > >
> > > > > > /\/\/\/\/\/\
> > > > > >
> > > > > > This check needs to be inside of check_memory_region, otherwise we
> > > > > > will have similar problems in all other places that use
> > > > > > check_memory_region.
> > > > > Thanks for your reminder.
> > > > >
> > > > >  bool check_memory_region(unsigned long addr, size_t size, bool write,
> > > > >                                 unsigned long ret_ip)
> > > > >  {
> > > > > +       if (long(size) < 0) {
> > > > > +               kasan_report_invalid_size(src, dest, len, _RET_IP_);
> > > > > +               return false;
> > > > > +       }
> > > > > +
> > > > >         return check_memory_region_inline(addr, size, write, ret_ip);
> > > > >  }
> > > > >
> > > > > > But check_memory_region already returns a bool, so we could check that
> > > > > > bool and return early.
> > > > >
> > > > > When size<0, we should only show one KASAN report, and should we only
> > > > > limit to return when size<0 is true? If yse, then __memmove() will do
> > > > > nothing.
> > > > >
> > > > >
> > > > >  void *memmove(void *dest, const void *src, size_t len)
> > > > >  {
> > > > > -       check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > > > +       if(!check_memory_region((unsigned long)src, len, false,
> > > > > _RET_IP_)
> > > > > +               && long(size) < 0)
> > > > > +               return;
> > > > > +
> > > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > > >
> > > > >         return __memmove(dest, src, len);
> > > > >
> > > > > >
> > > > Hi Dmitry,
> > > >
> > > > What do you think the following code is better than the above one.
> > > > In memmmove/memset/memcpy, they need to determine whether size < 0 is
> > > > true. we directly determine whether size is negative in memmove and
> > > > return early. it avoid to generate repeated KASAN report. Is it better?
> > > >
> > > > void *memmove(void *dest, const void *src, size_t len)
> > > > {
> > > > +       if (long(size) < 0) {
> > > > +               kasan_report_invalid_size(src, dest, len, _RET_IP_);
> > > > +               return;
> > > > +       }
> > > > +
> > > >         check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > >
> > > >
> > > > check_memory_region() still has to check whether the size is negative.
> > > > but memmove/memset/memcpy generate invalid size KASAN report will not be
> > > > there.
> > >
> > >
> > > If check_memory_region() will do the check, why do we need to
> > > duplicate it inside of memmove and all other range functions?
> > >
> > Yes, I know it has duplication, but if we don't have to determine size<0
> > in memmove, then all check_memory_region return false will do nothing,
> 
> But they will produce a KASAN report, right? They are asked to check
> if 18446744073709551614 bytes are good. 18446744073709551614 bytes
> can't be good.
> 
> 
> > it includes other memory corruption behaviors, this is my original
> > concern.
> >
> > > I would do:
> > >
> > > void *memmove(void *dest, const void *src, size_t len)
> > > {
> > >         if (check_memory_region((unsigned long)src, len, false, _RET_IP_))
> > >                 return;
> > if check_memory_region return TRUE is to do nothing, but it is no memory
> > corruption? Should it return early when check_memory_region return a
> > FALSE?
> 
> Maybe. I just meant the overall idea: check_memory_region should
> detect that 18446744073709551614 bytes are bad, print an error, return
> an indication that bytes were bad, memmove should return early if the
> range is bad.
> 
ok, i will send new patch.
Thanks for your review.

> 
> > > This avoids duplicating the check, adds minimal amount of code to
> > > range functions and avoids adding kasan_report_invalid_size.
> > Thanks for your suggestion.
> > We originally want to show complete information(destination address,
> > source address, and its length), but add minimal amount of code into
> > kasan_report(), it should be good.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/1570069078.19702.57.camel%40mtksdccf07.



WARNING: multiple messages have this Message-ID (diff)
From: Walter Wu <walter-zh.wu@mediatek.com>
To: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>,
	Alexander Potapenko <glider@google.com>,
	Matthias Brugger <matthias.bgg@gmail.com>,
	LKML <linux-kernel@vger.kernel.org>,
	kasan-dev <kasan-dev@googlegroups.com>,
	Linux-MM <linux-mm@kvack.org>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>,
	linux-mediatek@lists.infradead.org,
	wsd_upstream <wsd_upstream@mediatek.com>
Subject: Re: [PATCH] kasan: fix the missing underflow in memmove and memcpy with CONFIG_KASAN_GENERIC=y
Date: Thu, 3 Oct 2019 17:38:45 +0800	[thread overview]
Message-ID: <1570095525.19702.59.camel@mtksdccf07> (raw)
In-Reply-To: <CACT4Y+ZwNv2-QBrvuR2JvemovmKPQ9Ggrr=ZkdTg6xy_Ki6UAg@mail.gmail.com>

On Thu, 2019-10-03 at 08:26 +0200, Dmitry Vyukov wrote:
> On Thu, Oct 3, 2019 at 4:18 AM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> >
> > On Wed, 2019-10-02 at 15:57 +0200, Dmitry Vyukov wrote:
> > > On Wed, Oct 2, 2019 at 2:15 PM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > >
> > > > On Mon, 2019-09-30 at 12:36 +0800, Walter Wu wrote:
> > > > > On Fri, 2019-09-27 at 21:41 +0200, Dmitry Vyukov wrote:
> > > > > > On Fri, Sep 27, 2019 at 4:22 PM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > > > > >
> > > > > > > On Fri, 2019-09-27 at 15:07 +0200, Dmitry Vyukov wrote:
> > > > > > > > On Fri, Sep 27, 2019 at 5:43 AM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > > > > > > >
> > > > > > > > > memmove() and memcpy() have missing underflow issues.
> > > > > > > > > When -7 <= size < 0, then KASAN will miss to catch the underflow issue.
> > > > > > > > > It looks like shadow start address and shadow end address is the same,
> > > > > > > > > so it does not actually check anything.
> > > > > > > > >
> > > > > > > > > The following test is indeed not caught by KASAN:
> > > > > > > > >
> > > > > > > > >         char *p = kmalloc(64, GFP_KERNEL);
> > > > > > > > >         memset((char *)p, 0, 64);
> > > > > > > > >         memmove((char *)p, (char *)p + 4, -2);
> > > > > > > > >         kfree((char*)p);
> > > > > > > > >
> > > > > > > > > It should be checked here:
> > > > > > > > >
> > > > > > > > > void *memmove(void *dest, const void *src, size_t len)
> > > > > > > > > {
> > > > > > > > >         check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > > > > > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > > > > > > >
> > > > > > > > >         return __memmove(dest, src, len);
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > We fix the shadow end address which is calculated, then generic KASAN
> > > > > > > > > get the right shadow end address and detect this underflow issue.
> > > > > > > > >
> > > > > > > > > [1] https://bugzilla.kernel.org/show_bug.cgi?id=199341
> > > > > > > > >
> > > > > > > > > Signed-off-by: Walter Wu <walter-zh.wu@mediatek.com>
> > > > > > > > > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > > > > > > > > ---
> > > > > > > > >  lib/test_kasan.c   | 36 ++++++++++++++++++++++++++++++++++++
> > > > > > > > >  mm/kasan/generic.c |  8 ++++++--
> > > > > > > > >  2 files changed, 42 insertions(+), 2 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/lib/test_kasan.c b/lib/test_kasan.c
> > > > > > > > > index b63b367a94e8..8bd014852556 100644
> > > > > > > > > --- a/lib/test_kasan.c
> > > > > > > > > +++ b/lib/test_kasan.c
> > > > > > > > > @@ -280,6 +280,40 @@ static noinline void __init kmalloc_oob_in_memset(void)
> > > > > > > > >         kfree(ptr);
> > > > > > > > >  }
> > > > > > > > >
> > > > > > > > > +static noinline void __init kmalloc_oob_in_memmove_underflow(void)
> > > > > > > > > +{
> > > > > > > > > +       char *ptr;
> > > > > > > > > +       size_t size = 64;
> > > > > > > > > +
> > > > > > > > > +       pr_info("underflow out-of-bounds in memmove\n");
> > > > > > > > > +       ptr = kmalloc(size, GFP_KERNEL);
> > > > > > > > > +       if (!ptr) {
> > > > > > > > > +               pr_err("Allocation failed\n");
> > > > > > > > > +               return;
> > > > > > > > > +       }
> > > > > > > > > +
> > > > > > > > > +       memset((char *)ptr, 0, 64);
> > > > > > > > > +       memmove((char *)ptr, (char *)ptr + 4, -2);
> > > > > > > > > +       kfree(ptr);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static noinline void __init kmalloc_oob_in_memmove_overflow(void)
> > > > > > > > > +{
> > > > > > > > > +       char *ptr;
> > > > > > > > > +       size_t size = 64;
> > > > > > > > > +
> > > > > > > > > +       pr_info("overflow out-of-bounds in memmove\n");
> > > > > > > > > +       ptr = kmalloc(size, GFP_KERNEL);
> > > > > > > > > +       if (!ptr) {
> > > > > > > > > +               pr_err("Allocation failed\n");
> > > > > > > > > +               return;
> > > > > > > > > +       }
> > > > > > > > > +
> > > > > > > > > +       memset((char *)ptr, 0, 64);
> > > > > > > > > +       memmove((char *)ptr + size, (char *)ptr, 2);
> > > > > > > > > +       kfree(ptr);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > >  static noinline void __init kmalloc_uaf(void)
> > > > > > > > >  {
> > > > > > > > >         char *ptr;
> > > > > > > > > @@ -734,6 +768,8 @@ static int __init kmalloc_tests_init(void)
> > > > > > > > >         kmalloc_oob_memset_4();
> > > > > > > > >         kmalloc_oob_memset_8();
> > > > > > > > >         kmalloc_oob_memset_16();
> > > > > > > > > +       kmalloc_oob_in_memmove_underflow();
> > > > > > > > > +       kmalloc_oob_in_memmove_overflow();
> > > > > > > > >         kmalloc_uaf();
> > > > > > > > >         kmalloc_uaf_memset();
> > > > > > > > >         kmalloc_uaf2();
> > > > > > > > > diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
> > > > > > > > > index 616f9dd82d12..34ca23d59e67 100644
> > > > > > > > > --- a/mm/kasan/generic.c
> > > > > > > > > +++ b/mm/kasan/generic.c
> > > > > > > > > @@ -131,9 +131,13 @@ static __always_inline bool memory_is_poisoned_n(unsigned long addr,
> > > > > > > > >                                                 size_t size)
> > > > > > > > >  {
> > > > > > > > >         unsigned long ret;
> > > > > > > > > +       void *shadow_start = kasan_mem_to_shadow((void *)addr);
> > > > > > > > > +       void *shadow_end = kasan_mem_to_shadow((void *)addr + size - 1) + 1;
> > > > > > > > >
> > > > > > > > > -       ret = memory_is_nonzero(kasan_mem_to_shadow((void *)addr),
> > > > > > > > > -                       kasan_mem_to_shadow((void *)addr + size - 1) + 1);
> > > > > > > > > +       if ((long)size < 0)
> > > > > > > > > +               shadow_end = kasan_mem_to_shadow((void *)addr + size);
> > > > > > > >
> > > > > > > > Hi Walter,
> > > > > > > >
> > > > > > > > Thanks for working on this.
> > > > > > > >
> > > > > > > > If size<0, does it make sense to continue at all? We will still check
> > > > > > > > 1PB of shadow memory? What happens when we pass such huge range to
> > > > > > > > memory_is_nonzero?
> > > > > > > > Perhaps it's better to produce an error and bail out immediately if size<0?
> > > > > > >
> > > > > > > I agree with what you said. when size<0, it is indeed an unreasonable
> > > > > > > behavior, it should be blocked from continuing to do.
> > > > > > >
> > > > > > >
> > > > > > > > Also, what's the failure mode of the tests? Didn't they badly corrupt
> > > > > > > > memory? We tried to keep tests such that they produce the KASAN
> > > > > > > > reports, but don't badly corrupt memory b/c/ we need to run all of
> > > > > > > > them.
> > > > > > >
> > > > > > > Maybe we should first produce KASAN reports and then go to execute
> > > > > > > memmove() or do nothing? It looks like it’s doing the following.or?
> > > > > > >
> > > > > > > void *memmove(void *dest, const void *src, size_t len)
> > > > > > >  {
> > > > > > > +       if (long(len) <= 0)
> > > > > >
> > > > > > /\/\/\/\/\/\
> > > > > >
> > > > > > This check needs to be inside of check_memory_region, otherwise we
> > > > > > will have similar problems in all other places that use
> > > > > > check_memory_region.
> > > > > Thanks for your reminder.
> > > > >
> > > > >  bool check_memory_region(unsigned long addr, size_t size, bool write,
> > > > >                                 unsigned long ret_ip)
> > > > >  {
> > > > > +       if (long(size) < 0) {
> > > > > +               kasan_report_invalid_size(src, dest, len, _RET_IP_);
> > > > > +               return false;
> > > > > +       }
> > > > > +
> > > > >         return check_memory_region_inline(addr, size, write, ret_ip);
> > > > >  }
> > > > >
> > > > > > But check_memory_region already returns a bool, so we could check that
> > > > > > bool and return early.
> > > > >
> > > > > When size<0, we should only show one KASAN report, and should we only
> > > > > limit to return when size<0 is true? If yse, then __memmove() will do
> > > > > nothing.
> > > > >
> > > > >
> > > > >  void *memmove(void *dest, const void *src, size_t len)
> > > > >  {
> > > > > -       check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > > > +       if(!check_memory_region((unsigned long)src, len, false,
> > > > > _RET_IP_)
> > > > > +               && long(size) < 0)
> > > > > +               return;
> > > > > +
> > > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > > >
> > > > >         return __memmove(dest, src, len);
> > > > >
> > > > > >
> > > > Hi Dmitry,
> > > >
> > > > What do you think the following code is better than the above one.
> > > > In memmmove/memset/memcpy, they need to determine whether size < 0 is
> > > > true. we directly determine whether size is negative in memmove and
> > > > return early. it avoid to generate repeated KASAN report. Is it better?
> > > >
> > > > void *memmove(void *dest, const void *src, size_t len)
> > > > {
> > > > +       if (long(size) < 0) {
> > > > +               kasan_report_invalid_size(src, dest, len, _RET_IP_);
> > > > +               return;
> > > > +       }
> > > > +
> > > >         check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > >
> > > >
> > > > check_memory_region() still has to check whether the size is negative.
> > > > but memmove/memset/memcpy generate invalid size KASAN report will not be
> > > > there.
> > >
> > >
> > > If check_memory_region() will do the check, why do we need to
> > > duplicate it inside of memmove and all other range functions?
> > >
> > Yes, I know it has duplication, but if we don't have to determine size<0
> > in memmove, then all check_memory_region return false will do nothing,
> 
> But they will produce a KASAN report, right? They are asked to check
> if 18446744073709551614 bytes are good. 18446744073709551614 bytes
> can't be good.
> 
> 
> > it includes other memory corruption behaviors, this is my original
> > concern.
> >
> > > I would do:
> > >
> > > void *memmove(void *dest, const void *src, size_t len)
> > > {
> > >         if (check_memory_region((unsigned long)src, len, false, _RET_IP_))
> > >                 return;
> > if check_memory_region return TRUE is to do nothing, but it is no memory
> > corruption? Should it return early when check_memory_region return a
> > FALSE?
> 
> Maybe. I just meant the overall idea: check_memory_region should
> detect that 18446744073709551614 bytes are bad, print an error, return
> an indication that bytes were bad, memmove should return early if the
> range is bad.
> 
ok, i will send new patch.
Thanks for your review.

> 
> > > This avoids duplicating the check, adds minimal amount of code to
> > > range functions and avoids adding kasan_report_invalid_size.
> > Thanks for your suggestion.
> > We originally want to show complete information(destination address,
> > source address, and its length), but add minimal amount of code into
> > kasan_report(), it should be good.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/1570069078.19702.57.camel%40mtksdccf07.

WARNING: multiple messages have this Message-ID (diff)
From: Walter Wu <walter-zh.wu@mediatek.com>
To: Dmitry Vyukov <dvyukov@google.com>
Cc: wsd_upstream <wsd_upstream@mediatek.com>,
	linux-mediatek@lists.infradead.org,
	LKML <linux-kernel@vger.kernel.org>,
	kasan-dev <kasan-dev@googlegroups.com>,
	Linux-MM <linux-mm@kvack.org>,
	Alexander Potapenko <glider@google.com>,
	Matthias Brugger <matthias.bgg@gmail.com>,
	Andrey Ryabinin <aryabinin@virtuozzo.com>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>
Subject: Re: [PATCH] kasan: fix the missing underflow in memmove and memcpy with CONFIG_KASAN_GENERIC=y
Date: Thu, 3 Oct 2019 17:38:45 +0800	[thread overview]
Message-ID: <1570095525.19702.59.camel@mtksdccf07> (raw)
In-Reply-To: <CACT4Y+ZwNv2-QBrvuR2JvemovmKPQ9Ggrr=ZkdTg6xy_Ki6UAg@mail.gmail.com>

On Thu, 2019-10-03 at 08:26 +0200, Dmitry Vyukov wrote:
> On Thu, Oct 3, 2019 at 4:18 AM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> >
> > On Wed, 2019-10-02 at 15:57 +0200, Dmitry Vyukov wrote:
> > > On Wed, Oct 2, 2019 at 2:15 PM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > >
> > > > On Mon, 2019-09-30 at 12:36 +0800, Walter Wu wrote:
> > > > > On Fri, 2019-09-27 at 21:41 +0200, Dmitry Vyukov wrote:
> > > > > > On Fri, Sep 27, 2019 at 4:22 PM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > > > > >
> > > > > > > On Fri, 2019-09-27 at 15:07 +0200, Dmitry Vyukov wrote:
> > > > > > > > On Fri, Sep 27, 2019 at 5:43 AM Walter Wu <walter-zh.wu@mediatek.com> wrote:
> > > > > > > > >
> > > > > > > > > memmove() and memcpy() have missing underflow issues.
> > > > > > > > > When -7 <= size < 0, then KASAN will miss to catch the underflow issue.
> > > > > > > > > It looks like shadow start address and shadow end address is the same,
> > > > > > > > > so it does not actually check anything.
> > > > > > > > >
> > > > > > > > > The following test is indeed not caught by KASAN:
> > > > > > > > >
> > > > > > > > >         char *p = kmalloc(64, GFP_KERNEL);
> > > > > > > > >         memset((char *)p, 0, 64);
> > > > > > > > >         memmove((char *)p, (char *)p + 4, -2);
> > > > > > > > >         kfree((char*)p);
> > > > > > > > >
> > > > > > > > > It should be checked here:
> > > > > > > > >
> > > > > > > > > void *memmove(void *dest, const void *src, size_t len)
> > > > > > > > > {
> > > > > > > > >         check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > > > > > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > > > > > > >
> > > > > > > > >         return __memmove(dest, src, len);
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > We fix the shadow end address which is calculated, then generic KASAN
> > > > > > > > > get the right shadow end address and detect this underflow issue.
> > > > > > > > >
> > > > > > > > > [1] https://bugzilla.kernel.org/show_bug.cgi?id=199341
> > > > > > > > >
> > > > > > > > > Signed-off-by: Walter Wu <walter-zh.wu@mediatek.com>
> > > > > > > > > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > > > > > > > > ---
> > > > > > > > >  lib/test_kasan.c   | 36 ++++++++++++++++++++++++++++++++++++
> > > > > > > > >  mm/kasan/generic.c |  8 ++++++--
> > > > > > > > >  2 files changed, 42 insertions(+), 2 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/lib/test_kasan.c b/lib/test_kasan.c
> > > > > > > > > index b63b367a94e8..8bd014852556 100644
> > > > > > > > > --- a/lib/test_kasan.c
> > > > > > > > > +++ b/lib/test_kasan.c
> > > > > > > > > @@ -280,6 +280,40 @@ static noinline void __init kmalloc_oob_in_memset(void)
> > > > > > > > >         kfree(ptr);
> > > > > > > > >  }
> > > > > > > > >
> > > > > > > > > +static noinline void __init kmalloc_oob_in_memmove_underflow(void)
> > > > > > > > > +{
> > > > > > > > > +       char *ptr;
> > > > > > > > > +       size_t size = 64;
> > > > > > > > > +
> > > > > > > > > +       pr_info("underflow out-of-bounds in memmove\n");
> > > > > > > > > +       ptr = kmalloc(size, GFP_KERNEL);
> > > > > > > > > +       if (!ptr) {
> > > > > > > > > +               pr_err("Allocation failed\n");
> > > > > > > > > +               return;
> > > > > > > > > +       }
> > > > > > > > > +
> > > > > > > > > +       memset((char *)ptr, 0, 64);
> > > > > > > > > +       memmove((char *)ptr, (char *)ptr + 4, -2);
> > > > > > > > > +       kfree(ptr);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static noinline void __init kmalloc_oob_in_memmove_overflow(void)
> > > > > > > > > +{
> > > > > > > > > +       char *ptr;
> > > > > > > > > +       size_t size = 64;
> > > > > > > > > +
> > > > > > > > > +       pr_info("overflow out-of-bounds in memmove\n");
> > > > > > > > > +       ptr = kmalloc(size, GFP_KERNEL);
> > > > > > > > > +       if (!ptr) {
> > > > > > > > > +               pr_err("Allocation failed\n");
> > > > > > > > > +               return;
> > > > > > > > > +       }
> > > > > > > > > +
> > > > > > > > > +       memset((char *)ptr, 0, 64);
> > > > > > > > > +       memmove((char *)ptr + size, (char *)ptr, 2);
> > > > > > > > > +       kfree(ptr);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > >  static noinline void __init kmalloc_uaf(void)
> > > > > > > > >  {
> > > > > > > > >         char *ptr;
> > > > > > > > > @@ -734,6 +768,8 @@ static int __init kmalloc_tests_init(void)
> > > > > > > > >         kmalloc_oob_memset_4();
> > > > > > > > >         kmalloc_oob_memset_8();
> > > > > > > > >         kmalloc_oob_memset_16();
> > > > > > > > > +       kmalloc_oob_in_memmove_underflow();
> > > > > > > > > +       kmalloc_oob_in_memmove_overflow();
> > > > > > > > >         kmalloc_uaf();
> > > > > > > > >         kmalloc_uaf_memset();
> > > > > > > > >         kmalloc_uaf2();
> > > > > > > > > diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
> > > > > > > > > index 616f9dd82d12..34ca23d59e67 100644
> > > > > > > > > --- a/mm/kasan/generic.c
> > > > > > > > > +++ b/mm/kasan/generic.c
> > > > > > > > > @@ -131,9 +131,13 @@ static __always_inline bool memory_is_poisoned_n(unsigned long addr,
> > > > > > > > >                                                 size_t size)
> > > > > > > > >  {
> > > > > > > > >         unsigned long ret;
> > > > > > > > > +       void *shadow_start = kasan_mem_to_shadow((void *)addr);
> > > > > > > > > +       void *shadow_end = kasan_mem_to_shadow((void *)addr + size - 1) + 1;
> > > > > > > > >
> > > > > > > > > -       ret = memory_is_nonzero(kasan_mem_to_shadow((void *)addr),
> > > > > > > > > -                       kasan_mem_to_shadow((void *)addr + size - 1) + 1);
> > > > > > > > > +       if ((long)size < 0)
> > > > > > > > > +               shadow_end = kasan_mem_to_shadow((void *)addr + size);
> > > > > > > >
> > > > > > > > Hi Walter,
> > > > > > > >
> > > > > > > > Thanks for working on this.
> > > > > > > >
> > > > > > > > If size<0, does it make sense to continue at all? We will still check
> > > > > > > > 1PB of shadow memory? What happens when we pass such huge range to
> > > > > > > > memory_is_nonzero?
> > > > > > > > Perhaps it's better to produce an error and bail out immediately if size<0?
> > > > > > >
> > > > > > > I agree with what you said. when size<0, it is indeed an unreasonable
> > > > > > > behavior, it should be blocked from continuing to do.
> > > > > > >
> > > > > > >
> > > > > > > > Also, what's the failure mode of the tests? Didn't they badly corrupt
> > > > > > > > memory? We tried to keep tests such that they produce the KASAN
> > > > > > > > reports, but don't badly corrupt memory b/c/ we need to run all of
> > > > > > > > them.
> > > > > > >
> > > > > > > Maybe we should first produce KASAN reports and then go to execute
> > > > > > > memmove() or do nothing? It looks like it’s doing the following.or?
> > > > > > >
> > > > > > > void *memmove(void *dest, const void *src, size_t len)
> > > > > > >  {
> > > > > > > +       if (long(len) <= 0)
> > > > > >
> > > > > > /\/\/\/\/\/\
> > > > > >
> > > > > > This check needs to be inside of check_memory_region, otherwise we
> > > > > > will have similar problems in all other places that use
> > > > > > check_memory_region.
> > > > > Thanks for your reminder.
> > > > >
> > > > >  bool check_memory_region(unsigned long addr, size_t size, bool write,
> > > > >                                 unsigned long ret_ip)
> > > > >  {
> > > > > +       if (long(size) < 0) {
> > > > > +               kasan_report_invalid_size(src, dest, len, _RET_IP_);
> > > > > +               return false;
> > > > > +       }
> > > > > +
> > > > >         return check_memory_region_inline(addr, size, write, ret_ip);
> > > > >  }
> > > > >
> > > > > > But check_memory_region already returns a bool, so we could check that
> > > > > > bool and return early.
> > > > >
> > > > > When size<0, we should only show one KASAN report, and should we only
> > > > > limit to return when size<0 is true? If yse, then __memmove() will do
> > > > > nothing.
> > > > >
> > > > >
> > > > >  void *memmove(void *dest, const void *src, size_t len)
> > > > >  {
> > > > > -       check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > > > +       if(!check_memory_region((unsigned long)src, len, false,
> > > > > _RET_IP_)
> > > > > +               && long(size) < 0)
> > > > > +               return;
> > > > > +
> > > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > > >
> > > > >         return __memmove(dest, src, len);
> > > > >
> > > > > >
> > > > Hi Dmitry,
> > > >
> > > > What do you think the following code is better than the above one.
> > > > In memmmove/memset/memcpy, they need to determine whether size < 0 is
> > > > true. we directly determine whether size is negative in memmove and
> > > > return early. it avoid to generate repeated KASAN report. Is it better?
> > > >
> > > > void *memmove(void *dest, const void *src, size_t len)
> > > > {
> > > > +       if (long(size) < 0) {
> > > > +               kasan_report_invalid_size(src, dest, len, _RET_IP_);
> > > > +               return;
> > > > +       }
> > > > +
> > > >         check_memory_region((unsigned long)src, len, false, _RET_IP_);
> > > >         check_memory_region((unsigned long)dest, len, true, _RET_IP_);
> > > >
> > > >
> > > > check_memory_region() still has to check whether the size is negative.
> > > > but memmove/memset/memcpy generate invalid size KASAN report will not be
> > > > there.
> > >
> > >
> > > If check_memory_region() will do the check, why do we need to
> > > duplicate it inside of memmove and all other range functions?
> > >
> > Yes, I know it has duplication, but if we don't have to determine size<0
> > in memmove, then all check_memory_region return false will do nothing,
> 
> But they will produce a KASAN report, right? They are asked to check
> if 18446744073709551614 bytes are good. 18446744073709551614 bytes
> can't be good.
> 
> 
> > it includes other memory corruption behaviors, this is my original
> > concern.
> >
> > > I would do:
> > >
> > > void *memmove(void *dest, const void *src, size_t len)
> > > {
> > >         if (check_memory_region((unsigned long)src, len, false, _RET_IP_))
> > >                 return;
> > if check_memory_region return TRUE is to do nothing, but it is no memory
> > corruption? Should it return early when check_memory_region return a
> > FALSE?
> 
> Maybe. I just meant the overall idea: check_memory_region should
> detect that 18446744073709551614 bytes are bad, print an error, return
> an indication that bytes were bad, memmove should return early if the
> range is bad.
> 
ok, i will send new patch.
Thanks for your review.

> 
> > > This avoids duplicating the check, adds minimal amount of code to
> > > range functions and avoids adding kasan_report_invalid_size.
> > Thanks for your suggestion.
> > We originally want to show complete information(destination address,
> > source address, and its length), but add minimal amount of code into
> > kasan_report(), it should be good.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/1570069078.19702.57.camel%40mtksdccf07.



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

  reply	other threads:[~2019-10-03  9:38 UTC|newest]

Thread overview: 131+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-27  3:43 [PATCH] kasan: fix the missing underflow in memmove and memcpy with CONFIG_KASAN_GENERIC=y Walter Wu
2019-09-27  3:43 ` Walter Wu
2019-09-27  3:43 ` Walter Wu
2019-09-27 13:07 ` Dmitry Vyukov
2019-09-27 13:07   ` Dmitry Vyukov
2019-09-27 13:07   ` Dmitry Vyukov
2019-09-27 14:22   ` Walter Wu
2019-09-27 14:22     ` Walter Wu
2019-09-27 14:22     ` Walter Wu
2019-09-27 19:41     ` Dmitry Vyukov
2019-09-27 19:41       ` Dmitry Vyukov
2019-09-27 19:41       ` Dmitry Vyukov
2019-09-30  4:36       ` Walter Wu
2019-09-30  4:36         ` Walter Wu
2019-09-30  8:57         ` Marc Gonzalez
2019-09-30  8:57           ` Marc Gonzalez
2019-10-01  2:36           ` Walter Wu
2019-10-01  2:36             ` Walter Wu
2019-10-01  3:01             ` Dmitry Vyukov
2019-10-01  3:01               ` Dmitry Vyukov
2019-10-01  3:18               ` Walter Wu
2019-10-01  3:18                 ` Walter Wu
2019-10-02 12:15         ` Walter Wu
2019-10-02 12:15           ` Walter Wu
2019-10-02 12:15           ` Walter Wu
2019-10-02 13:57           ` Dmitry Vyukov
2019-10-02 13:57             ` Dmitry Vyukov
2019-10-02 13:57             ` Dmitry Vyukov
2019-10-03  2:17             ` Walter Wu
2019-10-03  2:17               ` Walter Wu
2019-10-03  6:26               ` Dmitry Vyukov
2019-10-03  6:26                 ` Dmitry Vyukov
2019-10-03  6:26                 ` Dmitry Vyukov
2019-10-03  9:38                 ` Walter Wu [this message]
2019-10-03  9:38                   ` Walter Wu
2019-10-03  9:38                   ` Walter Wu
2019-10-03 13:51                   ` Walter Wu
2019-10-03 13:51                     ` Walter Wu
2019-10-03 13:51                     ` Walter Wu
2019-10-03 14:53                     ` Dmitry Vyukov
2019-10-03 14:53                       ` Dmitry Vyukov
2019-10-03 14:53                       ` Dmitry Vyukov
2019-10-04  4:42                       ` Walter Wu
2019-10-04  4:42                         ` Walter Wu
2019-10-04  4:42                         ` Walter Wu
2019-10-04  8:02                         ` Walter Wu
2019-10-04  8:02                           ` Walter Wu
2019-10-04  8:02                           ` Walter Wu
2019-10-04  9:18                           ` Dmitry Vyukov
2019-10-04  9:18                             ` Dmitry Vyukov
2019-10-04  9:18                             ` Dmitry Vyukov
2019-10-04  9:44                             ` Walter Wu
2019-10-04  9:44                               ` Walter Wu
2019-10-04  9:44                               ` Walter Wu
2019-10-04  9:54                               ` Dmitry Vyukov
2019-10-04  9:54                                 ` Dmitry Vyukov
2019-10-04  9:54                                 ` Dmitry Vyukov
2019-10-04  9:54                                 ` Dmitry Vyukov
2019-10-04 12:05                                 ` Walter Wu
2019-10-04 12:05                                   ` Walter Wu
2019-10-04 12:05                                   ` Walter Wu
2019-10-04 13:52                                   ` Dmitry Vyukov
2019-10-04 13:52                                     ` Dmitry Vyukov
2019-10-04 13:52                                     ` Dmitry Vyukov
2019-10-07  3:22                                     ` Walter Wu
2019-10-07  3:22                                       ` Walter Wu
2019-10-07  3:22                                       ` Walter Wu
2019-10-07  7:29                                       ` Dmitry Vyukov
2019-10-07  7:29                                         ` Dmitry Vyukov
2019-10-07  7:29                                         ` Dmitry Vyukov
2019-10-07  8:18                                         ` Walter Wu
2019-10-07  8:18                                           ` Walter Wu
2019-10-07  8:18                                           ` Walter Wu
2019-10-07  8:24                                           ` Dmitry Vyukov
2019-10-07  8:24                                             ` Dmitry Vyukov
2019-10-07  8:24                                             ` Dmitry Vyukov
2019-10-07  8:24                                             ` Dmitry Vyukov
2019-10-07  8:51                                             ` Walter Wu
2019-10-07  8:51                                               ` Walter Wu
2019-10-07  8:51                                               ` Walter Wu
2019-10-07  8:54                                               ` Dmitry Vyukov
2019-10-07  8:54                                                 ` Dmitry Vyukov
2019-10-07  8:54                                                 ` Dmitry Vyukov
2019-10-07  9:03                                                 ` Walter Wu
2019-10-07  9:03                                                   ` Walter Wu
2019-10-07  9:03                                                   ` Walter Wu
2019-10-07  9:10                                                   ` Dmitry Vyukov
2019-10-07  9:10                                                     ` Dmitry Vyukov
2019-10-07  9:10                                                     ` Dmitry Vyukov
2019-10-07  9:10                                                     ` Dmitry Vyukov
2019-10-07  9:28                                                     ` Walter Wu
2019-10-07  9:28                                                       ` Walter Wu
2019-10-07  9:28                                                       ` Walter Wu
2019-10-07  9:50                                                       ` Walter Wu
2019-10-07  9:50                                                         ` Walter Wu
2019-10-07  9:50                                                         ` Walter Wu
2019-10-07 10:51                                                         ` Dmitry Vyukov
2019-10-07 10:51                                                           ` Dmitry Vyukov
2019-10-07 10:51                                                           ` Dmitry Vyukov
2019-10-07 12:03                                                           ` Walter Wu
2019-10-07 12:03                                                             ` Walter Wu
2019-10-07 12:03                                                             ` Walter Wu
2019-10-07 12:19                                                             ` Dmitry Vyukov
2019-10-07 12:19                                                               ` Dmitry Vyukov
2019-10-07 12:19                                                               ` Dmitry Vyukov
2019-10-07 12:32                                                               ` Walter Wu
2019-10-07 12:32                                                                 ` Walter Wu
2019-10-07 12:32                                                                 ` Walter Wu
2019-10-07 13:33                                                                 ` Dmitry Vyukov
2019-10-07 13:33                                                                   ` Dmitry Vyukov
2019-10-07 13:33                                                                   ` Dmitry Vyukov
2019-10-08  6:15                                                                   ` Walter Wu
2019-10-08  6:15                                                                     ` Walter Wu
2019-10-08  6:15                                                                     ` Walter Wu
2019-10-08  9:47                                                                     ` Qian Cai
2019-10-08  9:47                                                                       ` Qian Cai
2019-10-08 11:02                                                                       ` Walter Wu
2019-10-08 11:02                                                                         ` Walter Wu
2019-10-08 11:02                                                                         ` Walter Wu
2019-10-08 11:42                                                                         ` Qian Cai
2019-10-08 11:42                                                                           ` Qian Cai
2019-10-08 12:07                                                                           ` Walter Wu
2019-10-08 12:07                                                                             ` Walter Wu
2019-10-08 12:07                                                                             ` Walter Wu
2019-10-08 12:11                                                                           ` Dmitry Vyukov
2019-10-08 12:11                                                                             ` Dmitry Vyukov
2019-10-08 12:11                                                                             ` Dmitry Vyukov
2019-10-08 12:11                                                                             ` Dmitry Vyukov
2019-10-14  2:19                                                                             ` Walter Wu
2019-10-14  2:19                                                                               ` Walter Wu
2019-10-14  2:19                                                                               ` Walter Wu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1570095525.19702.59.camel@mtksdccf07 \
    --to=walter-zh.wu@mediatek.com \
    --cc=aryabinin@virtuozzo.com \
    --cc=dvyukov@google.com \
    --cc=glider@google.com \
    --cc=kasan-dev@googlegroups.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mediatek@lists.infradead.org \
    --cc=linux-mm@kvack.org \
    --cc=matthias.bgg@gmail.com \
    --cc=wsd_upstream@mediatek.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.