* Re: [PATCH] ipc/msg.c: Fix lost wakeup in msgsnd().
2013-09-03 14:00 [PATCH] ipc/msg.c: Fix lost wakeup in msgsnd() Manfred Spraul
@ 2013-09-03 15:08 ` Rik van Riel
2013-09-03 16:13 ` Sedat Dilek
2013-09-03 18:07 ` Davidlohr Bueso
2 siblings, 0 replies; 6+ messages in thread
From: Rik van Riel @ 2013-09-03 15:08 UTC (permalink / raw)
To: Manfred Spraul
Cc: Linus Torvalds, Andrew Morton, Davidlohr Bueso, Sedat Dilek,
Davidlohr Bueso, linux-next, LKML, Stephen Rothwell, linux-mm,
Andi Kleen, Jonathan Gonzalez, Vineet Gupta
On 09/03/2013 10:00 AM, Manfred Spraul wrote:
> The check if the queue is full and adding current to the wait queue of pending
> msgsnd() operations (ss_add()) must be atomic.
>
> Otherwise:
> - the thread that performs msgsnd() finds a full queue and decides to sleep.
> - the thread that performs msgrcv() calls first reads all messages from the
> queue and then sleep, because the queue is empty.
> - the msgrcv() calls do not perform any wakeups, because the msgsnd() task
> has not yet called ss_add().
> - then the msgsnd()-thread first calls ss_add() and then sleeps.
> Net result: msgsnd() and msgrcv() both sleep forever.
>
> Observed with msgctl08 from ltp with a preemptible kernel.
>
> Fix: Call ipc_lock_object() before performing the check.
>
> The patch also moves security_msg_queue_msgsnd() under ipc_lock_object:
> - msgctl(IPC_SET) explicitely mentions that it tries to expunge any pending
> operations that are not allowed anymore with the new permissions.
> If security_msg_queue_msgsnd() is called without locks, then there might be
> races.
> - it makes the patch much simpler.
>
> Reported-by: Vineet Gupta <Vineet.Gupta1@synopsys.com>
> Signed-off-by: Manfred Spraul <manfred@colorfullife.com>
Acked-by: Rik van Riel <riel@redhat.com>
--
All rights reversed
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] ipc/msg.c: Fix lost wakeup in msgsnd().
2013-09-03 14:00 [PATCH] ipc/msg.c: Fix lost wakeup in msgsnd() Manfred Spraul
2013-09-03 15:08 ` Rik van Riel
@ 2013-09-03 16:13 ` Sedat Dilek
2013-09-03 16:33 ` Manfred Spraul
2013-09-03 18:07 ` Davidlohr Bueso
2 siblings, 1 reply; 6+ messages in thread
From: Sedat Dilek @ 2013-09-03 16:13 UTC (permalink / raw)
To: Manfred Spraul
Cc: Linus Torvalds, Andrew Morton, Davidlohr Bueso, Davidlohr Bueso,
linux-next, LKML, Stephen Rothwell, linux-mm, Andi Kleen,
Rik van Riel, Jonathan Gonzalez, Vineet Gupta
On Tue, Sep 3, 2013 at 4:00 PM, Manfred Spraul <manfred@colorfullife.com> wrote:
> The check if the queue is full and adding current to the wait queue of pending
> msgsnd() operations (ss_add()) must be atomic.
>
> Otherwise:
> - the thread that performs msgsnd() finds a full queue and decides to sleep.
> - the thread that performs msgrcv() calls first reads all messages from the
> queue and then sleep, because the queue is empty.
reads -> sleeps
> - the msgrcv() calls do not perform any wakeups, because the msgsnd() task
> has not yet called ss_add().
> - then the msgsnd()-thread first calls ss_add() and then sleeps.
> Net result: msgsnd() and msgrcv() both sleep forever.
>
I don't know what and why "net result" - net in sense of networking?
> Observed with msgctl08 from ltp with a preemptible kernel.
>
...on ARC arch (that sounds funny somehow).
> Fix: Call ipc_lock_object() before performing the check.
>
> The patch also moves security_msg_queue_msgsnd() under ipc_lock_object:
> - msgctl(IPC_SET) explicitely mentions that it tries to expunge any pending
> operations that are not allowed anymore with the new permissions.
> If security_msg_queue_msgsnd() is called without locks, then there might be
> races.
> - it makes the patch much simpler.
>
> Reported-by: Vineet Gupta <Vineet.Gupta1@synopsys.com>
> Signed-off-by: Manfred Spraul <manfred@colorfullife.com>
I guess this is missing a "CC: stable" as Vineet reported against
Linux v3.11-rc7 (and should enter v3.11.1)?
- Sedat -
> ---
> ipc/msg.c | 12 +++++-------
> 1 file changed, 5 insertions(+), 7 deletions(-)
>
> diff --git a/ipc/msg.c b/ipc/msg.c
> index 9f29d9e..b65fdf1 100644
> --- a/ipc/msg.c
> +++ b/ipc/msg.c
> @@ -680,16 +680,18 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
> goto out_unlock1;
> }
>
> + ipc_lock_object(&msq->q_perm);
> +
> for (;;) {
> struct msg_sender s;
>
> err = -EACCES;
> if (ipcperms(ns, &msq->q_perm, S_IWUGO))
> - goto out_unlock1;
> + goto out_unlock0;
>
> err = security_msg_queue_msgsnd(msq, msg, msgflg);
> if (err)
> - goto out_unlock1;
> + goto out_unlock0;
>
> if (msgsz + msq->q_cbytes <= msq->q_qbytes &&
> 1 + msq->q_qnum <= msq->q_qbytes) {
> @@ -699,10 +701,9 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
> /* queue full, wait: */
> if (msgflg & IPC_NOWAIT) {
> err = -EAGAIN;
> - goto out_unlock1;
> + goto out_unlock0;
> }
>
> - ipc_lock_object(&msq->q_perm);
> ss_add(msq, &s);
>
> if (!ipc_rcu_getref(msq)) {
> @@ -730,10 +731,7 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
> goto out_unlock0;
> }
>
> - ipc_unlock_object(&msq->q_perm);
> }
> -
> - ipc_lock_object(&msq->q_perm);
> msq->q_lspid = task_tgid_vnr(current);
> msq->q_stime = get_seconds();
>
> --
> 1.8.3.1
>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] ipc/msg.c: Fix lost wakeup in msgsnd().
2013-09-03 16:13 ` Sedat Dilek
@ 2013-09-03 16:33 ` Manfred Spraul
2013-09-03 21:56 ` Sedat Dilek
0 siblings, 1 reply; 6+ messages in thread
From: Manfred Spraul @ 2013-09-03 16:33 UTC (permalink / raw)
To: sedat.dilek, Greg KH
Cc: Linus Torvalds, Andrew Morton, Davidlohr Bueso, Davidlohr Bueso,
linux-next, LKML, Stephen Rothwell, linux-mm, Andi Kleen,
Rik van Riel, Jonathan Gonzalez, Vineet Gupta
Hi Sedat,
On 09/03/2013 06:13 PM, Sedat Dilek wrote:
> On Tue, Sep 3, 2013 at 4:00 PM, Manfred Spraul <manfred@colorfullife.com> wrote:
>> The check if the queue is full and adding current to the wait queue of pending
>> msgsnd() operations (ss_add()) must be atomic.
>>
>> Otherwise:
>> - the thread that performs msgsnd() finds a full queue and decides to sleep.
>> - the thread that performs msgrcv() calls first reads all messages from the
>> queue and then sleep, because the queue is empty.
> reads -> sleeps
Correct.
>> - the msgrcv() calls do not perform any wakeups, because the msgsnd() task
>> has not yet called ss_add().
>> - then the msgsnd()-thread first calls ss_add() and then sleeps.
>> Net result: msgsnd() and msgrcv() both sleep forever.
>>
> I don't know what and why "net result" - net in sense of networking?
http://en.wiktionary.org/wiki/net#Adjective
I.e.: Ignore/remove the "Net".
>> Observed with msgctl08 from ltp with a preemptible kernel.
>>
> ...on ARC arch (that sounds funny somehow).
>
>> Fix: Call ipc_lock_object() before performing the check.
>>
>> The patch also moves security_msg_queue_msgsnd() under ipc_lock_object:
>> - msgctl(IPC_SET) explicitely mentions that it tries to expunge any pending
>> operations that are not allowed anymore with the new permissions.
>> If security_msg_queue_msgsnd() is called without locks, then there might be
>> races.
>> - it makes the patch much simpler.
>>
>> Reported-by: Vineet Gupta <Vineet.Gupta1@synopsys.com>
>> Signed-off-by: Manfred Spraul <manfred@colorfullife.com>
> I guess this is missing a "CC: stable" as Vineet reported against
> Linux v3.11-rc7 (and should enter v3.11.1)?
Yes. I didn't notice that Linus already released 3.11.
--
Manfred
> - Sedat -
>
>> ---
>> ipc/msg.c | 12 +++++-------
>> 1 file changed, 5 insertions(+), 7 deletions(-)
>>
>> diff --git a/ipc/msg.c b/ipc/msg.c
>> index 9f29d9e..b65fdf1 100644
>> --- a/ipc/msg.c
>> +++ b/ipc/msg.c
>> @@ -680,16 +680,18 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
>> goto out_unlock1;
>> }
>>
>> + ipc_lock_object(&msq->q_perm);
>> +
>> for (;;) {
>> struct msg_sender s;
>>
>> err = -EACCES;
>> if (ipcperms(ns, &msq->q_perm, S_IWUGO))
>> - goto out_unlock1;
>> + goto out_unlock0;
>>
>> err = security_msg_queue_msgsnd(msq, msg, msgflg);
>> if (err)
>> - goto out_unlock1;
>> + goto out_unlock0;
>>
>> if (msgsz + msq->q_cbytes <= msq->q_qbytes &&
>> 1 + msq->q_qnum <= msq->q_qbytes) {
>> @@ -699,10 +701,9 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
>> /* queue full, wait: */
>> if (msgflg & IPC_NOWAIT) {
>> err = -EAGAIN;
>> - goto out_unlock1;
>> + goto out_unlock0;
>> }
>>
>> - ipc_lock_object(&msq->q_perm);
>> ss_add(msq, &s);
>>
>> if (!ipc_rcu_getref(msq)) {
>> @@ -730,10 +731,7 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
>> goto out_unlock0;
>> }
>>
>> - ipc_unlock_object(&msq->q_perm);
>> }
>> -
>> - ipc_lock_object(&msq->q_perm);
>> msq->q_lspid = task_tgid_vnr(current);
>> msq->q_stime = get_seconds();
>>
>> --
>> 1.8.3.1
>>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] ipc/msg.c: Fix lost wakeup in msgsnd().
2013-09-03 16:33 ` Manfred Spraul
@ 2013-09-03 21:56 ` Sedat Dilek
0 siblings, 0 replies; 6+ messages in thread
From: Sedat Dilek @ 2013-09-03 21:56 UTC (permalink / raw)
To: Manfred Spraul
Cc: Greg KH, Linus Torvalds, Andrew Morton, Davidlohr Bueso,
Davidlohr Bueso, linux-next, LKML, Stephen Rothwell, linux-mm,
Andi Kleen, Rik van Riel, Jonathan Gonzalez, Vineet Gupta
On Tue, Sep 3, 2013 at 6:33 PM, Manfred Spraul <manfred@colorfullife.com> wrote:
> Hi Sedat,
>
>
> On 09/03/2013 06:13 PM, Sedat Dilek wrote:
>>
>> On Tue, Sep 3, 2013 at 4:00 PM, Manfred Spraul <manfred@colorfullife.com>
>> wrote:
>>>
>>> The check if the queue is full and adding current to the wait queue of
>>> pending
>>> msgsnd() operations (ss_add()) must be atomic.
>>>
>>> Otherwise:
>>> - the thread that performs msgsnd() finds a full queue and decides to
>>> sleep.
>>> - the thread that performs msgrcv() calls first reads all messages from
>>> the
>>> queue and then sleep, because the queue is empty.
>>
>> reads -> sleeps
>
> Correct.
>
>>> - the msgrcv() calls do not perform any wakeups, because the msgsnd()
>>> task
>>> has not yet called ss_add().
>>> - then the msgsnd()-thread first calls ss_add() and then sleeps.
>>> Net result: msgsnd() and msgrcv() both sleep forever.
>>>
>> I don't know what and why "net result" - net in sense of networking?
>
> http://en.wiktionary.org/wiki/net#Adjective
> I.e.: Ignore/remove the "Net".
>
>
>>> Observed with msgctl08 from ltp with a preemptible kernel.
>>>
>> ...on ARC arch (that sounds funny somehow).
>>
>>> Fix: Call ipc_lock_object() before performing the check.
>>>
>>> The patch also moves security_msg_queue_msgsnd() under ipc_lock_object:
>>> - msgctl(IPC_SET) explicitely mentions that it tries to expunge any
>>> pending
>>> operations that are not allowed anymore with the new permissions.
>>> If security_msg_queue_msgsnd() is called without locks, then there
>>> might be
>>> races.
>>> - it makes the patch much simpler.
>>>
>>> Reported-by: Vineet Gupta <Vineet.Gupta1@synopsys.com>
>>> Signed-off-by: Manfred Spraul <manfred@colorfullife.com>
>>
>> I guess this is missing a "CC: stable" as Vineet reported against
>> Linux v3.11-rc7 (and should enter v3.11.1)?
>
> Yes. I didn't notice that Linus already released 3.11.
>
Linus pushed your patch upstream... with typos fixed and "CC: stable #3.11".
Thanks to all involved people!
- Sedat -
[1] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=bebcb928c820d0ee83aca4b192adc195e43e66a2
> --
> Manfred
>
>> - Sedat -
>>
>>> ---
>>> ipc/msg.c | 12 +++++-------
>>> 1 file changed, 5 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/ipc/msg.c b/ipc/msg.c
>>> index 9f29d9e..b65fdf1 100644
>>> --- a/ipc/msg.c
>>> +++ b/ipc/msg.c
>>> @@ -680,16 +680,18 @@ long do_msgsnd(int msqid, long mtype, void __user
>>> *mtext,
>>> goto out_unlock1;
>>> }
>>>
>>> + ipc_lock_object(&msq->q_perm);
>>> +
>>> for (;;) {
>>> struct msg_sender s;
>>>
>>> err = -EACCES;
>>> if (ipcperms(ns, &msq->q_perm, S_IWUGO))
>>> - goto out_unlock1;
>>> + goto out_unlock0;
>>>
>>> err = security_msg_queue_msgsnd(msq, msg, msgflg);
>>> if (err)
>>> - goto out_unlock1;
>>> + goto out_unlock0;
>>>
>>> if (msgsz + msq->q_cbytes <= msq->q_qbytes &&
>>> 1 + msq->q_qnum <= msq->q_qbytes) {
>>> @@ -699,10 +701,9 @@ long do_msgsnd(int msqid, long mtype, void __user
>>> *mtext,
>>> /* queue full, wait: */
>>> if (msgflg & IPC_NOWAIT) {
>>> err = -EAGAIN;
>>> - goto out_unlock1;
>>> + goto out_unlock0;
>>> }
>>>
>>> - ipc_lock_object(&msq->q_perm);
>>> ss_add(msq, &s);
>>>
>>> if (!ipc_rcu_getref(msq)) {
>>> @@ -730,10 +731,7 @@ long do_msgsnd(int msqid, long mtype, void __user
>>> *mtext,
>>> goto out_unlock0;
>>> }
>>>
>>> - ipc_unlock_object(&msq->q_perm);
>>> }
>>> -
>>> - ipc_lock_object(&msq->q_perm);
>>> msq->q_lspid = task_tgid_vnr(current);
>>> msq->q_stime = get_seconds();
>>>
>>> --
>>> 1.8.3.1
>>>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] ipc/msg.c: Fix lost wakeup in msgsnd().
2013-09-03 14:00 [PATCH] ipc/msg.c: Fix lost wakeup in msgsnd() Manfred Spraul
2013-09-03 15:08 ` Rik van Riel
2013-09-03 16:13 ` Sedat Dilek
@ 2013-09-03 18:07 ` Davidlohr Bueso
2 siblings, 0 replies; 6+ messages in thread
From: Davidlohr Bueso @ 2013-09-03 18:07 UTC (permalink / raw)
To: Manfred Spraul
Cc: Linus Torvalds, Andrew Morton, Davidlohr Bueso, Sedat Dilek,
Davidlohr Bueso, linux-next, LKML, Stephen Rothwell, linux-mm,
Andi Kleen, Rik van Riel, Jonathan Gonzalez, Vineet Gupta
On Tue, 2013-09-03 at 16:00 +0200, Manfred Spraul wrote:
> The check if the queue is full and adding current to the wait queue of pending
> msgsnd() operations (ss_add()) must be atomic.
>
> Otherwise:
> - the thread that performs msgsnd() finds a full queue and decides to sleep.
> - the thread that performs msgrcv() calls first reads all messages from the
> queue and then sleep, because the queue is empty.
> - the msgrcv() calls do not perform any wakeups, because the msgsnd() task
> has not yet called ss_add().
> - then the msgsnd()-thread first calls ss_add() and then sleeps.
> Net result: msgsnd() and msgrcv() both sleep forever.
>
> Observed with msgctl08 from ltp with a preemptible kernel.
Good catch, thanks for looking into this Manfred.
FWIW similar changes that aim at reducing the kern_ipc_perm.lock
contention in shm have already been in linux-next for a good while and
should be going into 3.12. While both Sedat and I have tested them
through LTP, I will keep an eye open for regressions so that we don't
run into issues like this, late in the release cycle.
>
> Fix: Call ipc_lock_object() before performing the check.
>
> The patch also moves security_msg_queue_msgsnd() under ipc_lock_object:
> - msgctl(IPC_SET) explicitely mentions that it tries to expunge any pending
> operations that are not allowed anymore with the new permissions.
> If security_msg_queue_msgsnd() is called without locks, then there might be
> races.
Right.
> - it makes the patch much simpler.
>
> Reported-by: Vineet Gupta <Vineet.Gupta1@synopsys.com>
> Signed-off-by: Manfred Spraul <manfred@colorfullife.com>
Signed-off-by: Davidlohr Bueso <davidlohr@hp.com>
> ---
> ipc/msg.c | 12 +++++-------
> 1 file changed, 5 insertions(+), 7 deletions(-)
>
> diff --git a/ipc/msg.c b/ipc/msg.c
> index 9f29d9e..b65fdf1 100644
> --- a/ipc/msg.c
> +++ b/ipc/msg.c
> @@ -680,16 +680,18 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
> goto out_unlock1;
> }
>
> + ipc_lock_object(&msq->q_perm);
> +
> for (;;) {
> struct msg_sender s;
>
> err = -EACCES;
> if (ipcperms(ns, &msq->q_perm, S_IWUGO))
> - goto out_unlock1;
> + goto out_unlock0;
>
> err = security_msg_queue_msgsnd(msq, msg, msgflg);
> if (err)
> - goto out_unlock1;
> + goto out_unlock0;
>
> if (msgsz + msq->q_cbytes <= msq->q_qbytes &&
> 1 + msq->q_qnum <= msq->q_qbytes) {
> @@ -699,10 +701,9 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
> /* queue full, wait: */
> if (msgflg & IPC_NOWAIT) {
> err = -EAGAIN;
> - goto out_unlock1;
> + goto out_unlock0;
> }
>
> - ipc_lock_object(&msq->q_perm);
> ss_add(msq, &s);
>
> if (!ipc_rcu_getref(msq)) {
> @@ -730,10 +731,7 @@ long do_msgsnd(int msqid, long mtype, void __user *mtext,
> goto out_unlock0;
> }
>
> - ipc_unlock_object(&msq->q_perm);
> }
> -
> - ipc_lock_object(&msq->q_perm);
> msq->q_lspid = task_tgid_vnr(current);
> msq->q_stime = get_seconds();
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread