[PATCH] relay: handle alloc_percpu returning NULL in relay_open

[PATCH] relay: handle alloc_percpu returning NULL in relay_open

[Daniel Axtens]

alloc_percpu() may return NULL, which means chan->buf may be set to
NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
dereference an invalid pointer:

BUG: Unable to handle kernel data access at 0x7dae0000
Faulting instruction address: 0xc0000000003f3fec
...
NIP [c0000000003f3fec] relay_open+0x29c/0x600
LR [c0000000003f3fc0] relay_open+0x270/0x600
Call Trace:
[c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
[c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
[c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
[c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
[c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
[c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
[c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
[c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68

Check if alloc_percpu returns NULL. Because we can readily catch and
handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.

This was found by syzkaller both on x86 and powerpc, and the reproducer
it found on powerpc is capable of hitting the issue as an unprivileged
user.

Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
Reported-by: syzbot+1e925b4b836afe85a1c6@syzkaller-ppc64.appspotmail.com
Reported-by: syzbot+587b2421926808309d21@syzkaller-ppc64.appspotmail.com
Reported-by: syzbot+58320b7171734bf79d26@syzkaller.appspotmail.com
Reported-by: syzbot+d6074fb08bdb2e010520@syzkaller.appspotmail.com
Cc: Akash Goel <akash.goel@intel.com>
Cc: Andrew Donnellan <ajd@linux.ibm.com> # syzkaller-ppc64
Cc: stable@vger.kernel.org # v4.10+
Signed-off-by: Daniel Axtens <dja@axtens.net>

--

There's a syz reproducer on the powerpc syzbot that eventually hits
the bug, but it can take up to an hour or so before it keels over on a
kernel with all the syzkaller debugging on, and even longer on a
production kernel. I have been able to reproduce it once on a stock
Ubuntu 5.0 ppc64le kernel.

I will ask MITRE for a CVE - while only the process doing the syscall
gets killed, it gets killed while holding the relay_channels_mutex,
so it blocks all future relay activity.
---
 kernel/relay.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kernel/relay.c b/kernel/relay.c
index ade14fb7ce2e..a376cc6b54ec 100644
--- a/kernel/relay.c
+++ b/kernel/relay.c
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
 	if (!chan)
 		return NULL;
 
-	chan->buf = alloc_percpu(struct rchan_buf *);
+	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+				     GFP_KERNEL | __GFP_NOWARN);
+	if (!chan->buf) {
+		kfree(chan);
+		return NULL;
+	}
+
 	chan->version = RELAYFS_CHANNEL_VERSION;
 	chan->n_subbufs = n_subbufs;
 	chan->subbuf_size = subbuf_size;
-- 
2.20.1

Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open

[Michael Ellerman]

Daniel Axtens <dja@axtens.net> writes:
> alloc_percpu() may return NULL, which means chan->buf may be set to
> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
> dereference an invalid pointer:
>
> BUG: Unable to handle kernel data access at 0x7dae0000
> Faulting instruction address: 0xc0000000003f3fec
> ...
> NIP [c0000000003f3fec] relay_open+0x29c/0x600
> LR [c0000000003f3fc0] relay_open+0x270/0x600
> Call Trace:
> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
>
> Check if alloc_percpu returns NULL. Because we can readily catch and
> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
>
> This was found by syzkaller both on x86 and powerpc, and the reproducer
> it found on powerpc is capable of hitting the issue as an unprivileged
> user.
>
> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
> Reported-by: syzbot+1e925b4b836afe85a1c6@syzkaller-ppc64.appspotmail.com
> Reported-by: syzbot+587b2421926808309d21@syzkaller-ppc64.appspotmail.com
> Reported-by: syzbot+58320b7171734bf79d26@syzkaller.appspotmail.com
> Reported-by: syzbot+d6074fb08bdb2e010520@syzkaller.appspotmail.com
> Cc: Akash Goel <akash.goel@intel.com>
> Cc: Andrew Donnellan <ajd@linux.ibm.com> # syzkaller-ppc64
> Cc: stable@vger.kernel.org # v4.10+
> Signed-off-by: Daniel Axtens <dja@axtens.net>
...
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>  	if (!chan)
>  		return NULL;
>  
> -	chan->buf = alloc_percpu(struct rchan_buf *);
> +	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> +				     GFP_KERNEL | __GFP_NOWARN);
> +	if (!chan->buf) {
> +		kfree(chan);
> +		return NULL;
> +	}
> +
>  	chan->version = RELAYFS_CHANNEL_VERSION;
>  	chan->n_subbufs = n_subbufs;
>  	chan->subbuf_size = subbuf_size;

This looks right to me. The kfree + direct return is correct, there's
nothing else that needs tear down in this function.

I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per
cpu, and if that fails the system is pretty sick, so a warning could be
helpful. There's also logic in the percpu allocator to limit the number
of warnings printed. But see what others think.

Reviewed-by: Michael Ellerman <mpe@ellerman.id.au>

cheers

Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open

[Andrew Donnellan]

On 29/11/19 3:59 pm, Michael Ellerman wrote:
>> diff --git a/kernel/relay.c b/kernel/relay.c
>> index ade14fb7ce2e..a376cc6b54ec 100644
>> --- a/kernel/relay.c
>> +++ b/kernel/relay.c
>> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>>   	if (!chan)
>>   		return NULL;
>>   
>> -	chan->buf = alloc_percpu(struct rchan_buf *);
>> +	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
>> +				     GFP_KERNEL | __GFP_NOWARN);
>> +	if (!chan->buf) {
>> +		kfree(chan);
>> +		return NULL;
>> +	}
>> +
>>   	chan->version = RELAYFS_CHANNEL_VERSION;
>>   	chan->n_subbufs = n_subbufs;
>>   	chan->subbuf_size = subbuf_size;
> 
> This looks right to me. The kfree + direct return is correct, there's
> nothing else that needs tear down in this function.
> 
> I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per
> cpu, and if that fails the system is pretty sick, so a warning could be
> helpful. There's also logic in the percpu allocator to limit the number
> of warnings printed. But see what others think.

mpe was wondering why we didn't see a message printed from the percpu 
allocator - the answer appears to be that we hit this case when the 
process is killed while the percpu allocator is waiting for 
pcpu_alloc_mutex, in which case it bails out without printing a warning.

It looks to me like that case doesn't warrant a warning message, while a 
failing allocation for other reasons should probably get a warning.

But whatever, otherwise this patch looks good to me. I've told our 
powerpc syzbot to test it.

Reviewed-by: Andrew Donnellan <ajd@linux.ibm.com>


-- 
Andrew Donnellan              OzLabs, ADL Canberra
ajd@linux.ibm.com             IBM Australia Limited

Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open

[Daniel Axtens]

Daniel Axtens <dja@axtens.net> writes:
> --
>
> There's a syz reproducer on the powerpc syzbot that eventually hits
> the bug, but it can take up to an hour or so before it keels over on a
> kernel with all the syzkaller debugging on, and even longer on a
> production kernel. I have been able to reproduce it once on a stock
> Ubuntu 5.0 ppc64le kernel.
>
> I will ask MITRE for a CVE - while only the process doing the syscall
> gets killed, it gets killed while holding the relay_channels_mutex,
> so it blocks all future relay activity.

CVE-2019-19462 has been assigned.

Regards,
Daniel


> ---
>  kernel/relay.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>  	if (!chan)
>  		return NULL;
>  
> -	chan->buf = alloc_percpu(struct rchan_buf *);
> +	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> +				     GFP_KERNEL | __GFP_NOWARN);
> +	if (!chan->buf) {
> +		kfree(chan);
> +		return NULL;
> +	}
> +
>  	chan->version = RELAYFS_CHANNEL_VERSION;
>  	chan->n_subbufs = n_subbufs;
>  	chan->subbuf_size = subbuf_size;
> -- 
> 2.20.1

Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open

[Guenter Roeck]

On Fri, Nov 29, 2019 at 12:37:45PM +1100, Daniel Axtens wrote:
> alloc_percpu() may return NULL, which means chan->buf may be set to
> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
> dereference an invalid pointer:
> 
> BUG: Unable to handle kernel data access at 0x7dae0000
> Faulting instruction address: 0xc0000000003f3fec
> ...
> NIP [c0000000003f3fec] relay_open+0x29c/0x600
> LR [c0000000003f3fc0] relay_open+0x270/0x600
> Call Trace:
> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
> 
> Check if alloc_percpu returns NULL. Because we can readily catch and
> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
> 
> This was found by syzkaller both on x86 and powerpc, and the reproducer
> it found on powerpc is capable of hitting the issue as an unprivileged
> user.
> 
> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
> Reported-by: syzbot+1e925b4b836afe85a1c6@syzkaller-ppc64.appspotmail.com
> Reported-by: syzbot+587b2421926808309d21@syzkaller-ppc64.appspotmail.com
> Reported-by: syzbot+58320b7171734bf79d26@syzkaller.appspotmail.com
> Reported-by: syzbot+d6074fb08bdb2e010520@syzkaller.appspotmail.com
> Cc: Akash Goel <akash.goel@intel.com>
> Cc: Andrew Donnellan <ajd@linux.ibm.com> # syzkaller-ppc64
> Cc: stable@vger.kernel.org # v4.10+
> Signed-off-by: Daniel Axtens <dja@axtens.net>
> 

So there is a CVE now, but it appears that the patch went nowhere.
Are there any plans to actually apply it ?

Thanks,
Guenter

> --
> 
> There's a syz reproducer on the powerpc syzbot that eventually hits
> the bug, but it can take up to an hour or so before it keels over on a
> kernel with all the syzkaller debugging on, and even longer on a
> production kernel. I have been able to reproduce it once on a stock
> Ubuntu 5.0 ppc64le kernel.
> 
> I will ask MITRE for a CVE - while only the process doing the syscall
> gets killed, it gets killed while holding the relay_channels_mutex,
> so it blocks all future relay activity.
> ---
>  kernel/relay.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>  	if (!chan)
>  		return NULL;
>  
> -	chan->buf = alloc_percpu(struct rchan_buf *);
> +	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> +				     GFP_KERNEL | __GFP_NOWARN);
> +	if (!chan->buf) {
> +		kfree(chan);
> +		return NULL;
> +	}
> +
>  	chan->version = RELAYFS_CHANNEL_VERSION;
>  	chan->n_subbufs = n_subbufs;
>  	chan->subbuf_size = subbuf_size;
> -- 
> 2.20.1
> 

Re: [PATCH] relay: handle alloc_percpu returning NULL in relay_open

[Daniel Axtens]

Guenter Roeck <linux@roeck-us.net> writes:

> On Fri, Nov 29, 2019 at 12:37:45PM +1100, Daniel Axtens wrote:
>> alloc_percpu() may return NULL, which means chan->buf may be set to
>> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
>> dereference an invalid pointer:
>> 
>> BUG: Unable to handle kernel data access at 0x7dae0000
>> Faulting instruction address: 0xc0000000003f3fec
>> ...
>> NIP [c0000000003f3fec] relay_open+0x29c/0x600
>> LR [c0000000003f3fc0] relay_open+0x270/0x600
>> Call Trace:
>> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
>> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
>> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
>> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
>> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
>> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
>> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
>> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
>> 
>> Check if alloc_percpu returns NULL. Because we can readily catch and
>> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
>> 
>> This was found by syzkaller both on x86 and powerpc, and the reproducer
>> it found on powerpc is capable of hitting the issue as an unprivileged
>> user.
>> 
>> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
>> Reported-by: syzbot+1e925b4b836afe85a1c6@syzkaller-ppc64.appspotmail.com
>> Reported-by: syzbot+587b2421926808309d21@syzkaller-ppc64.appspotmail.com
>> Reported-by: syzbot+58320b7171734bf79d26@syzkaller.appspotmail.com
>> Reported-by: syzbot+d6074fb08bdb2e010520@syzkaller.appspotmail.com
>> Cc: Akash Goel <akash.goel@intel.com>
>> Cc: Andrew Donnellan <ajd@linux.ibm.com> # syzkaller-ppc64
>> Cc: stable@vger.kernel.org # v4.10+
>> Signed-off-by: Daniel Axtens <dja@axtens.net>
>> 
>
> So there is a CVE now, but it appears that the patch went nowhere.
> Are there any plans to actually apply it ?

I sent a v2 that addresses some review comments, I guess if anything is
applied it will be that.

Daniel

>
> Thanks,
> Guenter
>
>> --
>> 
>> There's a syz reproducer on the powerpc syzbot that eventually hits
>> the bug, but it can take up to an hour or so before it keels over on a
>> kernel with all the syzkaller debugging on, and even longer on a
>> production kernel. I have been able to reproduce it once on a stock
>> Ubuntu 5.0 ppc64le kernel.
>> 
>> I will ask MITRE for a CVE - while only the process doing the syscall
>> gets killed, it gets killed while holding the relay_channels_mutex,
>> so it blocks all future relay activity.
>> ---
>>  kernel/relay.c | 8 +++++++-
>>  1 file changed, 7 insertions(+), 1 deletion(-)
>> 
>> diff --git a/kernel/relay.c b/kernel/relay.c
>> index ade14fb7ce2e..a376cc6b54ec 100644
>> --- a/kernel/relay.c
>> +++ b/kernel/relay.c
>> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>>  	if (!chan)
>>  		return NULL;
>>  
>> -	chan->buf = alloc_percpu(struct rchan_buf *);
>> +	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
>> +				     GFP_KERNEL | __GFP_NOWARN);
>> +	if (!chan->buf) {
>> +		kfree(chan);
>> +		return NULL;
>> +	}
>> +
>>  	chan->version = RELAYFS_CHANNEL_VERSION;
>>  	chan->n_subbufs = n_subbufs;
>>  	chan->subbuf_size = subbuf_size;
>> -- 
>> 2.20.1
>>