* [PATCH] relay: handle alloc_percpu returning NULL in relay_open
@ 2019-11-29 1:37 Daniel Axtens
2019-11-29 4:59 ` Michael Ellerman
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Daniel Axtens @ 2019-11-29 1:37 UTC (permalink / raw)
To: linux-kernel, viro
Cc: akash.goel, ajd, Daniel Axtens, syzbot+1e925b4b836afe85a1c6,
syzbot+587b2421926808309d21, syzbot+58320b7171734bf79d26,
syzbot+d6074fb08bdb2e010520
alloc_percpu() may return NULL, which means chan->buf may be set to
NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
dereference an invalid pointer:
BUG: Unable to handle kernel data access at 0x7dae0000
Faulting instruction address: 0xc0000000003f3fec
...
NIP [c0000000003f3fec] relay_open+0x29c/0x600
LR [c0000000003f3fc0] relay_open+0x270/0x600
Call Trace:
[c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
[c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
[c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
[c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
[c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
[c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
[c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
[c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
Check if alloc_percpu returns NULL. Because we can readily catch and
handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
This was found by syzkaller both on x86 and powerpc, and the reproducer
it found on powerpc is capable of hitting the issue as an unprivileged
user.
Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
Reported-by: syzbot+1e925b4b836afe85a1c6@syzkaller-ppc64.appspotmail.com
Reported-by: syzbot+587b2421926808309d21@syzkaller-ppc64.appspotmail.com
Reported-by: syzbot+58320b7171734bf79d26@syzkaller.appspotmail.com
Reported-by: syzbot+d6074fb08bdb2e010520@syzkaller.appspotmail.com
Cc: Akash Goel <akash.goel@intel.com>
Cc: Andrew Donnellan <ajd@linux.ibm.com> # syzkaller-ppc64
Cc: stable@vger.kernel.org # v4.10+
Signed-off-by: Daniel Axtens <dja@axtens.net>
--
There's a syz reproducer on the powerpc syzbot that eventually hits
the bug, but it can take up to an hour or so before it keels over on a
kernel with all the syzkaller debugging on, and even longer on a
production kernel. I have been able to reproduce it once on a stock
Ubuntu 5.0 ppc64le kernel.
I will ask MITRE for a CVE - while only the process doing the syscall
gets killed, it gets killed while holding the relay_channels_mutex,
so it blocks all future relay activity.
---
kernel/relay.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/kernel/relay.c b/kernel/relay.c
index ade14fb7ce2e..a376cc6b54ec 100644
--- a/kernel/relay.c
+++ b/kernel/relay.c
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
if (!chan)
return NULL;
- chan->buf = alloc_percpu(struct rchan_buf *);
+ chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!chan->buf) {
+ kfree(chan);
+ return NULL;
+ }
+
chan->version = RELAYFS_CHANNEL_VERSION;
chan->n_subbufs = n_subbufs;
chan->subbuf_size = subbuf_size;
--
2.20.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open
2019-11-29 1:37 [PATCH] relay: handle alloc_percpu returning NULL in relay_open Daniel Axtens
@ 2019-11-29 4:59 ` Michael Ellerman
2019-11-29 12:42 ` Andrew Donnellan
2019-11-30 6:04 ` Daniel Axtens
2019-12-23 16:36 ` Guenter Roeck
2 siblings, 1 reply; 6+ messages in thread
From: Michael Ellerman @ 2019-11-29 4:59 UTC (permalink / raw)
To: Daniel Axtens, linux-kernel, viro
Cc: akash.goel, ajd, Daniel Axtens, syzbot+1e925b4b836afe85a1c6,
syzbot+587b2421926808309d21, syzbot+58320b7171734bf79d26,
syzbot+d6074fb08bdb2e010520
Daniel Axtens <dja@axtens.net> writes:
> alloc_percpu() may return NULL, which means chan->buf may be set to
> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
> dereference an invalid pointer:
>
> BUG: Unable to handle kernel data access at 0x7dae0000
> Faulting instruction address: 0xc0000000003f3fec
> ...
> NIP [c0000000003f3fec] relay_open+0x29c/0x600
> LR [c0000000003f3fc0] relay_open+0x270/0x600
> Call Trace:
> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
>
> Check if alloc_percpu returns NULL. Because we can readily catch and
> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
>
> This was found by syzkaller both on x86 and powerpc, and the reproducer
> it found on powerpc is capable of hitting the issue as an unprivileged
> user.
>
> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
> Reported-by: syzbot+1e925b4b836afe85a1c6@syzkaller-ppc64.appspotmail.com
> Reported-by: syzbot+587b2421926808309d21@syzkaller-ppc64.appspotmail.com
> Reported-by: syzbot+58320b7171734bf79d26@syzkaller.appspotmail.com
> Reported-by: syzbot+d6074fb08bdb2e010520@syzkaller.appspotmail.com
> Cc: Akash Goel <akash.goel@intel.com>
> Cc: Andrew Donnellan <ajd@linux.ibm.com> # syzkaller-ppc64
> Cc: stable@vger.kernel.org # v4.10+
> Signed-off-by: Daniel Axtens <dja@axtens.net>
...
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
> if (!chan)
> return NULL;
>
> - chan->buf = alloc_percpu(struct rchan_buf *);
> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> + GFP_KERNEL | __GFP_NOWARN);
> + if (!chan->buf) {
> + kfree(chan);
> + return NULL;
> + }
> +
> chan->version = RELAYFS_CHANNEL_VERSION;
> chan->n_subbufs = n_subbufs;
> chan->subbuf_size = subbuf_size;
This looks right to me. The kfree + direct return is correct, there's
nothing else that needs tear down in this function.
I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per
cpu, and if that fails the system is pretty sick, so a warning could be
helpful. There's also logic in the percpu allocator to limit the number
of warnings printed. But see what others think.
Reviewed-by: Michael Ellerman <mpe@ellerman.id.au>
cheers
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open
2019-11-29 4:59 ` Michael Ellerman
@ 2019-11-29 12:42 ` Andrew Donnellan
0 siblings, 0 replies; 6+ messages in thread
From: Andrew Donnellan @ 2019-11-29 12:42 UTC (permalink / raw)
To: Michael Ellerman, Daniel Axtens, linux-kernel, viro
Cc: akash.goel, syzbot+1e925b4b836afe85a1c6,
syzbot+587b2421926808309d21, syzbot+58320b7171734bf79d26,
syzbot+d6074fb08bdb2e010520
On 29/11/19 3:59 pm, Michael Ellerman wrote:
>> diff --git a/kernel/relay.c b/kernel/relay.c
>> index ade14fb7ce2e..a376cc6b54ec 100644
>> --- a/kernel/relay.c
>> +++ b/kernel/relay.c
>> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>> if (!chan)
>> return NULL;
>>
>> - chan->buf = alloc_percpu(struct rchan_buf *);
>> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
>> + GFP_KERNEL | __GFP_NOWARN);
>> + if (!chan->buf) {
>> + kfree(chan);
>> + return NULL;
>> + }
>> +
>> chan->version = RELAYFS_CHANNEL_VERSION;
>> chan->n_subbufs = n_subbufs;
>> chan->subbuf_size = subbuf_size;
>
> This looks right to me. The kfree + direct return is correct, there's
> nothing else that needs tear down in this function.
>
> I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per
> cpu, and if that fails the system is pretty sick, so a warning could be
> helpful. There's also logic in the percpu allocator to limit the number
> of warnings printed. But see what others think.
mpe was wondering why we didn't see a message printed from the percpu
allocator - the answer appears to be that we hit this case when the
process is killed while the percpu allocator is waiting for
pcpu_alloc_mutex, in which case it bails out without printing a warning.
It looks to me like that case doesn't warrant a warning message, while a
failing allocation for other reasons should probably get a warning.
But whatever, otherwise this patch looks good to me. I've told our
powerpc syzbot to test it.
Reviewed-by: Andrew Donnellan <ajd@linux.ibm.com>
--
Andrew Donnellan OzLabs, ADL Canberra
ajd@linux.ibm.com IBM Australia Limited
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open
2019-11-29 1:37 [PATCH] relay: handle alloc_percpu returning NULL in relay_open Daniel Axtens
2019-11-29 4:59 ` Michael Ellerman
@ 2019-11-30 6:04 ` Daniel Axtens
2019-12-23 16:36 ` Guenter Roeck
2 siblings, 0 replies; 6+ messages in thread
From: Daniel Axtens @ 2019-11-30 6:04 UTC (permalink / raw)
To: linux-kernel, viro
Cc: akash.goel, ajd, syzbot+1e925b4b836afe85a1c6,
syzbot+587b2421926808309d21, syzbot+58320b7171734bf79d26,
syzbot+d6074fb08bdb2e010520
Daniel Axtens <dja@axtens.net> writes:
> --
>
> There's a syz reproducer on the powerpc syzbot that eventually hits
> the bug, but it can take up to an hour or so before it keels over on a
> kernel with all the syzkaller debugging on, and even longer on a
> production kernel. I have been able to reproduce it once on a stock
> Ubuntu 5.0 ppc64le kernel.
>
> I will ask MITRE for a CVE - while only the process doing the syscall
> gets killed, it gets killed while holding the relay_channels_mutex,
> so it blocks all future relay activity.
CVE-2019-19462 has been assigned.
Regards,
Daniel
> ---
> kernel/relay.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
> if (!chan)
> return NULL;
>
> - chan->buf = alloc_percpu(struct rchan_buf *);
> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> + GFP_KERNEL | __GFP_NOWARN);
> + if (!chan->buf) {
> + kfree(chan);
> + return NULL;
> + }
> +
> chan->version = RELAYFS_CHANNEL_VERSION;
> chan->n_subbufs = n_subbufs;
> chan->subbuf_size = subbuf_size;
> --
> 2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open
2019-11-29 1:37 [PATCH] relay: handle alloc_percpu returning NULL in relay_open Daniel Axtens
2019-11-29 4:59 ` Michael Ellerman
2019-11-30 6:04 ` Daniel Axtens
@ 2019-12-23 16:36 ` Guenter Roeck
2019-12-24 0:26 ` Daniel Axtens
2 siblings, 1 reply; 6+ messages in thread
From: Guenter Roeck @ 2019-12-23 16:36 UTC (permalink / raw)
To: Daniel Axtens
Cc: linux-kernel, viro, akash.goel, ajd, syzbot+1e925b4b836afe85a1c6,
syzbot+587b2421926808309d21, syzbot+58320b7171734bf79d26,
syzbot+d6074fb08bdb2e010520
On Fri, Nov 29, 2019 at 12:37:45PM +1100, Daniel Axtens wrote:
> alloc_percpu() may return NULL, which means chan->buf may be set to
> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
> dereference an invalid pointer:
>
> BUG: Unable to handle kernel data access at 0x7dae0000
> Faulting instruction address: 0xc0000000003f3fec
> ...
> NIP [c0000000003f3fec] relay_open+0x29c/0x600
> LR [c0000000003f3fc0] relay_open+0x270/0x600
> Call Trace:
> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
>
> Check if alloc_percpu returns NULL. Because we can readily catch and
> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
>
> This was found by syzkaller both on x86 and powerpc, and the reproducer
> it found on powerpc is capable of hitting the issue as an unprivileged
> user.
>
> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
> Reported-by: syzbot+1e925b4b836afe85a1c6@syzkaller-ppc64.appspotmail.com
> Reported-by: syzbot+587b2421926808309d21@syzkaller-ppc64.appspotmail.com
> Reported-by: syzbot+58320b7171734bf79d26@syzkaller.appspotmail.com
> Reported-by: syzbot+d6074fb08bdb2e010520@syzkaller.appspotmail.com
> Cc: Akash Goel <akash.goel@intel.com>
> Cc: Andrew Donnellan <ajd@linux.ibm.com> # syzkaller-ppc64
> Cc: stable@vger.kernel.org # v4.10+
> Signed-off-by: Daniel Axtens <dja@axtens.net>
>
So there is a CVE now, but it appears that the patch went nowhere.
Are there any plans to actually apply it ?
Thanks,
Guenter
> --
>
> There's a syz reproducer on the powerpc syzbot that eventually hits
> the bug, but it can take up to an hour or so before it keels over on a
> kernel with all the syzkaller debugging on, and even longer on a
> production kernel. I have been able to reproduce it once on a stock
> Ubuntu 5.0 ppc64le kernel.
>
> I will ask MITRE for a CVE - while only the process doing the syscall
> gets killed, it gets killed while holding the relay_channels_mutex,
> so it blocks all future relay activity.
> ---
> kernel/relay.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
> if (!chan)
> return NULL;
>
> - chan->buf = alloc_percpu(struct rchan_buf *);
> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> + GFP_KERNEL | __GFP_NOWARN);
> + if (!chan->buf) {
> + kfree(chan);
> + return NULL;
> + }
> +
> chan->version = RELAYFS_CHANNEL_VERSION;
> chan->n_subbufs = n_subbufs;
> chan->subbuf_size = subbuf_size;
> --
> 2.20.1
>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open
2019-12-23 16:36 ` Guenter Roeck
@ 2019-12-24 0:26 ` Daniel Axtens
0 siblings, 0 replies; 6+ messages in thread
From: Daniel Axtens @ 2019-12-24 0:26 UTC (permalink / raw)
To: Guenter Roeck
Cc: linux-kernel, viro, akash.goel, ajd, syzbot+1e925b4b836afe85a1c6,
syzbot+587b2421926808309d21, syzbot+58320b7171734bf79d26,
syzbot+d6074fb08bdb2e010520
Guenter Roeck <linux@roeck-us.net> writes:
> On Fri, Nov 29, 2019 at 12:37:45PM +1100, Daniel Axtens wrote:
>> alloc_percpu() may return NULL, which means chan->buf may be set to
>> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
>> dereference an invalid pointer:
>>
>> BUG: Unable to handle kernel data access at 0x7dae0000
>> Faulting instruction address: 0xc0000000003f3fec
>> ...
>> NIP [c0000000003f3fec] relay_open+0x29c/0x600
>> LR [c0000000003f3fc0] relay_open+0x270/0x600
>> Call Trace:
>> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
>> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
>> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
>> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
>> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
>> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
>> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
>> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
>>
>> Check if alloc_percpu returns NULL. Because we can readily catch and
>> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
>>
>> This was found by syzkaller both on x86 and powerpc, and the reproducer
>> it found on powerpc is capable of hitting the issue as an unprivileged
>> user.
>>
>> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
>> Reported-by: syzbot+1e925b4b836afe85a1c6@syzkaller-ppc64.appspotmail.com
>> Reported-by: syzbot+587b2421926808309d21@syzkaller-ppc64.appspotmail.com
>> Reported-by: syzbot+58320b7171734bf79d26@syzkaller.appspotmail.com
>> Reported-by: syzbot+d6074fb08bdb2e010520@syzkaller.appspotmail.com
>> Cc: Akash Goel <akash.goel@intel.com>
>> Cc: Andrew Donnellan <ajd@linux.ibm.com> # syzkaller-ppc64
>> Cc: stable@vger.kernel.org # v4.10+
>> Signed-off-by: Daniel Axtens <dja@axtens.net>
>>
>
> So there is a CVE now, but it appears that the patch went nowhere.
> Are there any plans to actually apply it ?
I sent a v2 that addresses some review comments, I guess if anything is
applied it will be that.
Daniel
>
> Thanks,
> Guenter
>
>> --
>>
>> There's a syz reproducer on the powerpc syzbot that eventually hits
>> the bug, but it can take up to an hour or so before it keels over on a
>> kernel with all the syzkaller debugging on, and even longer on a
>> production kernel. I have been able to reproduce it once on a stock
>> Ubuntu 5.0 ppc64le kernel.
>>
>> I will ask MITRE for a CVE - while only the process doing the syscall
>> gets killed, it gets killed while holding the relay_channels_mutex,
>> so it blocks all future relay activity.
>> ---
>> kernel/relay.c | 8 +++++++-
>> 1 file changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/kernel/relay.c b/kernel/relay.c
>> index ade14fb7ce2e..a376cc6b54ec 100644
>> --- a/kernel/relay.c
>> +++ b/kernel/relay.c
>> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>> if (!chan)
>> return NULL;
>>
>> - chan->buf = alloc_percpu(struct rchan_buf *);
>> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
>> + GFP_KERNEL | __GFP_NOWARN);
>> + if (!chan->buf) {
>> + kfree(chan);
>> + return NULL;
>> + }
>> +
>> chan->version = RELAYFS_CHANNEL_VERSION;
>> chan->n_subbufs = n_subbufs;
>> chan->subbuf_size = subbuf_size;
>> --
>> 2.20.1
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-12-24 0:26 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-29 1:37 [PATCH] relay: handle alloc_percpu returning NULL in relay_open Daniel Axtens
2019-11-29 4:59 ` Michael Ellerman
2019-11-29 12:42 ` Andrew Donnellan
2019-11-30 6:04 ` Daniel Axtens
2019-12-23 16:36 ` Guenter Roeck
2019-12-24 0:26 ` Daniel Axtens
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).