[PATCH 0/1] autofs: fix memory leak of waitqueues in autofs_catatonic_mode

[PATCH 0/1] autofs: fix memory leak of waitqueues in autofs_catatonic_mode

[Fedor Pchelkin]

Syzkaller reports the leak [1]. It is reproducible.

The following patch fixes the leak. It was proposed by Takeshi Misawa and
tested by Syzbot.

In other places of the code the waitqueue is freed when its wait_ctr
becomes zero (see autofs_wait_release). So I think it is not actually
supposed that inside autofs_catatonic_mode wait_ctr cannot be decreased to
zero. Please correct me if I'm wrong.

Also, looking at the discussion [2] of the '[PATCH] autofs4: use wake_up()
instead of wake_up_interruptible', shouldn't wake_up_interruptible()
inside autofs_catatonic_mode() be replaced with wake_up()?

[1] https://syzkaller.appspot.com/bug?id=a9412f636e2d733130f8def7975897d0b57f6e37
[2] https://www.spinics.net/lists/autofs/msg01875.html

[PATCH 1/1] autofs: fix memory leak of waitqueues in autofs_catatonic_mode

[Fedor Pchelkin]

Syzkaller reports a memory leak:

BUG: memory leak
unreferenced object 0xffff88810b279e00 (size 96):
  comm "syz-executor399", pid 3631, jiffies 4294964921 (age 23.870s)
  hex dump (first 32 bytes):
    00 00 00 00 00 00 00 00 08 9e 27 0b 81 88 ff ff  ..........'.....
    08 9e 27 0b 81 88 ff ff 00 00 00 00 00 00 00 00  ..'.............
  backtrace:
    [<ffffffff814cfc90>] kmalloc_trace+0x20/0x90 mm/slab_common.c:1046
    [<ffffffff81bb75ca>] kmalloc include/linux/slab.h:576 [inline]
    [<ffffffff81bb75ca>] autofs_wait+0x3fa/0x9a0 fs/autofs/waitq.c:378
    [<ffffffff81bb88a7>] autofs_do_expire_multi+0xa7/0x3e0 fs/autofs/expire.c:593
    [<ffffffff81bb8c33>] autofs_expire_multi+0x53/0x80 fs/autofs/expire.c:619
    [<ffffffff81bb6972>] autofs_root_ioctl_unlocked+0x322/0x3b0 fs/autofs/root.c:897
    [<ffffffff81bb6a95>] autofs_root_ioctl+0x25/0x30 fs/autofs/root.c:910
    [<ffffffff81602a9c>] vfs_ioctl fs/ioctl.c:51 [inline]
    [<ffffffff81602a9c>] __do_sys_ioctl fs/ioctl.c:870 [inline]
    [<ffffffff81602a9c>] __se_sys_ioctl fs/ioctl.c:856 [inline]
    [<ffffffff81602a9c>] __x64_sys_ioctl+0xfc/0x140 fs/ioctl.c:856
    [<ffffffff84608225>] do_syscall_x64 arch/x86/entry/common.c:50 [inline]
    [<ffffffff84608225>] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
    [<ffffffff84800087>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

autofs_wait_queue structs should be freed if their wait_ctr becomes zero.
Otherwise they will be lost.

In this case an AUTOFS_IOC_EXPIRE_MULTI ioctl is done, then a new
waitqueue struct is allocated in autofs_wait(), its initial wait_ctr
equals 2. After that wait_event_killable() is interrupted (it returns
-ERESTARTSYS), so that 'wq->name.name == NULL' condition may be not
satisfied. Actually, this condition can be satisfied when
autofs_wait_release() or autofs_catatonic_mode() is called and, what is
also important, wait_ctr is decremented in those places. Upon the exit of
autofs_wait(), wait_ctr is decremented to 1. Then the unmounting process
begins: kill_sb calls autofs_catatonic_mode(), which should have freed the
waitqueues, but it only decrements its usage counter to zero which is not
a correct behaviour.

So in catatonic mode we should free waitqueues which counter becomes zero.

Reported-by: syzbot+5e53f70e69ff0c0a1c0c@syzkaller.appspotmail.com
Suggested-by: Takeshi Misawa <jeliantsurux@gmail.com>
Signed-off-by: Fedor Pchelkin <pchelkin@ispras.ru>
Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru>
---
 fs/autofs/waitq.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/autofs/waitq.c b/fs/autofs/waitq.c
index 54c1f8b8b075..efdc76732fae 100644
--- a/fs/autofs/waitq.c
+++ b/fs/autofs/waitq.c
@@ -32,8 +32,9 @@ void autofs_catatonic_mode(struct autofs_sb_info *sbi)
 		wq->status = -ENOENT; /* Magic is gone - report failure */
 		kfree(wq->name.name - wq->offset);
 		wq->name.name = NULL;
-		wq->wait_ctr--;
 		wake_up_interruptible(&wq->queue);
+		if (!--wq->wait_ctr)
+			kfree(wq);
 		wq = nwq;
 	}
 	fput(sbi->pipe);	/* Close the pipe */
-- 
2.34.1

Re: [PATCH 0/1] autofs: fix memory leak of waitqueues in autofs_catatonic_mode

[Ian Kent]

On 12/2/23 03:59, Fedor Pchelkin wrote:
> Syzkaller reports the leak [1]. It is reproducible.
>
> The following patch fixes the leak. It was proposed by Takeshi Misawa and
> tested by Syzbot.
>
> In other places of the code the waitqueue is freed when its wait_ctr
> becomes zero (see autofs_wait_release). So I think it is not actually
> supposed that inside autofs_catatonic_mode wait_ctr cannot be decreased to
> zero. Please correct me if I'm wrong.

Clearly there's a problem here but I'll need to think about what's going

a bit more myself.


>
> Also, looking at the discussion [2] of the '[PATCH] autofs4: use wake_up()
> instead of wake_up_interruptible', shouldn't wake_up_interruptible()
> inside autofs_catatonic_mode() be replaced with wake_up()?

Yes, I think so but that also deserves a bit of thought.


Ian

Re: [PATCH 0/1] autofs: fix memory leak of waitqueues in autofs_catatonic_mode

[Ian Kent]

On 12/2/23 03:59, Fedor Pchelkin wrote:
> Syzkaller reports the leak [1]. It is reproducible.
>
> The following patch fixes the leak. It was proposed by Takeshi Misawa and
> tested by Syzbot.
>
> In other places of the code the waitqueue is freed when its wait_ctr
> becomes zero (see autofs_wait_release). So I think it is not actually
> supposed that inside autofs_catatonic_mode wait_ctr cannot be decreased to
> zero. Please correct me if I'm wrong.

This is a bit had to read but I think your saying there's an assumption

that wait_ctr can't become zero in autofs_catatonic_mode().


That's correct, the case of a waiting process getting sent a signal is

not accounted for and this can (as you observed) lead to the wait not

being freed and also not being freed at umount.


I think the change here should be sufficient to resolve the leak and

I can't think of any cases where this could cause a further problem.


>
> Also, looking at the discussion [2] of the '[PATCH] autofs4: use wake_up()
> instead of wake_up_interruptible', shouldn't wake_up_interruptible()
> inside autofs_catatonic_mode() be replaced with wake_up()?

This does imply that [2] should have been applied to autofs_catatonic_mode()

as well, I'm still trying to grok if that change would cause side effects

for the change here but I think not.


Ian

Re: [PATCH 0/1] autofs: fix memory leak of waitqueues in autofs_catatonic_mode

[Ian Kent]

On 13/2/23 12:27, Ian Kent wrote:
> On 12/2/23 03:59, Fedor Pchelkin wrote:
>> Syzkaller reports the leak [1]. It is reproducible.
>>
>> The following patch fixes the leak. It was proposed by Takeshi Misawa 
>> and
>> tested by Syzbot.
>>
>> In other places of the code the waitqueue is freed when its wait_ctr
>> becomes zero (see autofs_wait_release). So I think it is not actually
>> supposed that inside autofs_catatonic_mode wait_ctr cannot be 
>> decreased to
>> zero. Please correct me if I'm wrong.
>
> This is a bit had to read but I think your saying there's an assumption
>
> that wait_ctr can't become zero in autofs_catatonic_mode().
>
>
> That's correct, the case of a waiting process getting sent a signal is
>
> not accounted for and this can (as you observed) lead to the wait not
>
> being freed and also not being freed at umount.
>
>
> I think the change here should be sufficient to resolve the leak and
>
> I can't think of any cases where this could cause a further problem.
>
>
>>
>> Also, looking at the discussion [2] of the '[PATCH] autofs4: use 
>> wake_up()
>> instead of wake_up_interruptible', shouldn't wake_up_interruptible()
>> inside autofs_catatonic_mode() be replaced with wake_up()?
>
> This does imply that [2] should have been applied to 
> autofs_catatonic_mode()
>
> as well, I'm still trying to grok if that change would cause side effects
>
> for the change here but I think not.

I was going to Ack the patch but I wondering if we should wait a little

while and perhaps (probably) include the wake up call change as well.


In any case we need Al to accept it (cc'd).

Hopefully Al will offer his opinion on the changes too.


>
>
> Ian
>

Re: [PATCH 0/1] autofs: fix memory leak of waitqueues in autofs_catatonic_mode

[Fedor Pchelkin]

On Mon, Feb 13, 2023 at 12:37:16PM +0800, Ian Kent wrote:
> 
> I was going to Ack the patch but I wondering if we should wait a little
> 
> while and perhaps (probably) include the wake up call change as well.
>

Hmm, those would be separate patches?

An interesting thing is that the code itself supposes the wake up calls
from autofs_wait_release() and autofs_catatonic_mode() to be related in
some way (see autofs_wait fragment):

	/*
	 * wq->name.name is NULL iff the lock is already released
	 * or the mount has been made catatonic.
	 */
	wait_event_killable(wq->queue, wq->name.name == NULL);
	status = wq->status;

It seems 'the lock is already released' refers to autofs_wait_release()
as there is no alternative except the call to catatonic function where
wq->name.name is NULL. So apparently the wake up calls should be the same
(although I don't know if autofs_catatonic_mode has some different
behaviour in such case, but probably it doesn't differ here).

It's also strange that autofs_kill_sb() calls autofs_catatonic_mode() and
currently it just decrements the wait_ctr's and it is not clear to me
where the waitqueues are eventually freed in such case. Only if
autofs_wait_release() or autofs_wait() are called? I'm not sure whether
they are definitely called after that or not.

[1] https://www.spinics.net/lists/autofs/msg01878.html
> 
> In any case we need Al to accept it (cc'd).
> 
> Hopefully Al will offer his opinion on the changes too.
> 

It would be very nice if probably Al would make it more clear.

At the moment I think that the leak issue should be fixed with the
currenly discussed patch and the wake up call issue should be fixed like
in [1], but perhaps I'm missing something.

Re: [PATCH 0/1] autofs: fix memory leak of waitqueues in autofs_catatonic_mode

[Ian Kent]

On 11/3/23 01:56, Fedor Pchelkin wrote:
> On Mon, Feb 13, 2023 at 12:37:16PM +0800, Ian Kent wrote:
>> I was going to Ack the patch but I wondering if we should wait a little
>>
>> while and perhaps (probably) include the wake up call change as well.
>>
> Hmm, those would be separate patches?
>
> An interesting thing is that the code itself supposes the wake up calls
> from autofs_wait_release() and autofs_catatonic_mode() to be related in
> some way (see autofs_wait fragment):
>
> 	/*
> 	 * wq->name.name is NULL iff the lock is already released
> 	 * or the mount has been made catatonic.
> 	 */
> 	wait_event_killable(wq->queue, wq->name.name == NULL);
> 	status = wq->status;
>
> It seems 'the lock is already released' refers to autofs_wait_release()
> as there is no alternative except the call to catatonic function where
> wq->name.name is NULL. So apparently the wake up calls should be the same
> (although I don't know if autofs_catatonic_mode has some different
> behaviour in such case, but probably it doesn't differ here).

I think that, because there are processes waiting, they will always go

via the tail of autofs_wait() so the wait will be freed at that point.


Alternately autofs_wait_release() will be called from user space daemon

to tell the kernel it's done with the current notification.


I think there was an order of execution problem at some point between

autofs_wait() and autofs_wait_release() hence the code there. The same

may be the case for autofs_catatonic_mode() which is what the patch

implies.


These mount points can be left mounted after the user space daemon

exits with the processes still blocked so umounting the mount should

trigger the freeing of the name or they may be set catatonic by the

daemon at exit, again freeing the name, and in both cases unblocking

the processes to free the wait.


So I didn't think there was a memory leak here but SyZkaller says

there is.


>
> It's also strange that autofs_kill_sb() calls autofs_catatonic_mode() and
> currently it just decrements the wait_ctr's and it is not clear to me
> where the waitqueues are eventually freed in such case. Only if
> autofs_wait_release() or autofs_wait() are called? I'm not sure whether
> they are definitely called after that or not.
>
> [1] https://www.spinics.net/lists/autofs/msg01878.html
>> In any case we need Al to accept it (cc'd).
>>
>> Hopefully Al will offer his opinion on the changes too.
>>
> It would be very nice if probably Al would make it more clear.
>
> At the moment I think that the leak issue should be fixed with the
> currenly discussed patch and the wake up call issue should be fixed like
> in [1], but perhaps I'm missing something.

The question I have is, is it possible a process waiting on the wait

queue gets unblocked after the wait is freed in autofs_catatonic_mode?


Ian