* [Xen-devel] [PATCH v3 1/2] xenbus: req->body should be updated before req->state
@ 2020-03-03 22:14 Dongli Zhang
2020-03-03 22:14 ` [Xen-devel] [PATCH v3 2/2] xenbus: req->err " Dongli Zhang
2020-03-04 13:50 ` [Xen-devel] [PATCH v3 1/2] xenbus: req->body " Julien Grall
0 siblings, 2 replies; 5+ messages in thread
From: Dongli Zhang @ 2020-03-03 22:14 UTC (permalink / raw)
To: xen-devel, linux-kernel
Cc: jgross, joe.jin, boris.ostrovsky, sstabellini, julien
The req->body should be updated before req->state is updated and the
order should be guaranteed by a barrier.
Otherwise, read_reply() might return req->body = NULL.
Below is sample callstack when the issue is reproduced on purpose by
reordering the updates of req->body and req->state and adding delay in
code between updates of req->state and req->body.
[ 22.356105] general protection fault: 0000 [#1] SMP PTI
[ 22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6
[ 22.366727] Hardware name: Xen HVM domU, BIOS ...
[ 22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
... ...
[ 22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246
[ 22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000
[ 22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d
[ 22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd
[ 22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000
[ 22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000
[ 22.423726] FS: 0000000000000000(0000) GS:ffff9d953bc80000(0000) knlGS:0000000000000000
[ 22.429898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0
[ 22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 22.450342] Call Trace:
[ 22.452509] simple_strtoull+0x27/0x70
[ 22.455572] xenbus_transaction_start+0x31/0x50
[ 22.459104] netback_changed+0x76c/0xcc1 [xen_netfront]
[ 22.463279] ? find_watch+0x40/0x40
[ 22.466156] xenwatch_thread+0xb4/0x150
[ 22.469309] ? wait_woken+0x80/0x80
[ 22.472198] kthread+0x10e/0x130
[ 22.474925] ? kthread_park+0x80/0x80
[ 22.477946] ret_from_fork+0x35/0x40
[ 22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront xen_blkfront
[ 22.486783] ---[ end trace a9222030a747c3f7 ]---
[ 22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
The virt_rmb() is added in the 'true' path of test_reply(). The "while"
is changed to "do while" so that test_reply() is used as a read memory
barrier.
Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
Changed since v1:
- change "barrier()" to "virt_rmb()" in test_reply()
Changed since v2:
- Use "virt_rmb()" only in 'true' path
drivers/xen/xenbus/xenbus_comms.c | 2 ++
drivers/xen/xenbus/xenbus_xs.c | 9 ++++++---
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
index d239fc3c5e3d..852ed161fc2a 100644
--- a/drivers/xen/xenbus/xenbus_comms.c
+++ b/drivers/xen/xenbus/xenbus_comms.c
@@ -313,6 +313,8 @@ static int process_msg(void)
req->msg.type = state.msg.type;
req->msg.len = state.msg.len;
req->body = state.body;
+ /* write body, then update state */
+ virt_wmb();
req->state = xb_req_state_got_reply;
req->cb(req);
} else
diff --git a/drivers/xen/xenbus/xenbus_xs.c b/drivers/xen/xenbus/xenbus_xs.c
index ddc18da61834..3a06eb699f33 100644
--- a/drivers/xen/xenbus/xenbus_xs.c
+++ b/drivers/xen/xenbus/xenbus_xs.c
@@ -191,8 +191,11 @@ static bool xenbus_ok(void)
static bool test_reply(struct xb_req_data *req)
{
- if (req->state == xb_req_state_got_reply || !xenbus_ok())
+ if (req->state == xb_req_state_got_reply || !xenbus_ok()) {
+ /* read req->state before all other fields */
+ virt_rmb();
return true;
+ }
/* Make sure to reread req->state each time. */
barrier();
@@ -202,7 +205,7 @@ static bool test_reply(struct xb_req_data *req)
static void *read_reply(struct xb_req_data *req)
{
- while (req->state != xb_req_state_got_reply) {
+ do {
wait_event(req->wq, test_reply(req));
if (!xenbus_ok())
@@ -216,7 +219,7 @@ static void *read_reply(struct xb_req_data *req)
if (req->err)
return ERR_PTR(req->err);
- }
+ } while (req->state != xb_req_state_got_reply);
return req->body;
}
--
2.17.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [Xen-devel] [PATCH v3 2/2] xenbus: req->err should be updated before req->state
2020-03-03 22:14 [Xen-devel] [PATCH v3 1/2] xenbus: req->body should be updated before req->state Dongli Zhang
@ 2020-03-03 22:14 ` Dongli Zhang
2020-03-04 13:50 ` Julien Grall
2020-03-06 11:13 ` Boris Ostrovsky
2020-03-04 13:50 ` [Xen-devel] [PATCH v3 1/2] xenbus: req->body " Julien Grall
1 sibling, 2 replies; 5+ messages in thread
From: Dongli Zhang @ 2020-03-03 22:14 UTC (permalink / raw)
To: xen-devel, linux-kernel
Cc: jgross, joe.jin, boris.ostrovsky, sstabellini, julien
This patch adds the barrier to guarantee that req->err is always updated
before req->state.
Otherwise, read_reply() would not return ERR_PTR(req->err) but
req->body, when process_writes()->xb_write() is failed.
Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
drivers/xen/xenbus/xenbus_comms.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
index 852ed161fc2a..eb5151fc8efa 100644
--- a/drivers/xen/xenbus/xenbus_comms.c
+++ b/drivers/xen/xenbus/xenbus_comms.c
@@ -397,6 +397,8 @@ static int process_writes(void)
if (state.req->state == xb_req_state_aborted)
kfree(state.req);
else {
+ /* write err, then update state */
+ virt_wmb();
state.req->state = xb_req_state_got_reply;
wake_up(&state.req->wq);
}
--
2.17.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [Xen-devel] [PATCH v3 1/2] xenbus: req->body should be updated before req->state
2020-03-03 22:14 [Xen-devel] [PATCH v3 1/2] xenbus: req->body should be updated before req->state Dongli Zhang
2020-03-03 22:14 ` [Xen-devel] [PATCH v3 2/2] xenbus: req->err " Dongli Zhang
@ 2020-03-04 13:50 ` Julien Grall
1 sibling, 0 replies; 5+ messages in thread
From: Julien Grall @ 2020-03-04 13:50 UTC (permalink / raw)
To: Dongli Zhang, xen-devel, linux-kernel
Cc: jgross, boris.ostrovsky, sstabellini, joe.jin
Hi,
On 03/03/2020 22:14, Dongli Zhang wrote:
> The req->body should be updated before req->state is updated and the
> order should be guaranteed by a barrier.
>
> Otherwise, read_reply() might return req->body = NULL.
>
> Below is sample callstack when the issue is reproduced on purpose by
> reordering the updates of req->body and req->state and adding delay in
> code between updates of req->state and req->body.
>
> [ 22.356105] general protection fault: 0000 [#1] SMP PTI
> [ 22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6
> [ 22.366727] Hardware name: Xen HVM domU, BIOS ...
> [ 22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
> ... ...
> [ 22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246
> [ 22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000
> [ 22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d
> [ 22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd
> [ 22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000
> [ 22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000
> [ 22.423726] FS: 0000000000000000(0000) GS:ffff9d953bc80000(0000) knlGS:0000000000000000
> [ 22.429898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0
> [ 22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 22.450342] Call Trace:
> [ 22.452509] simple_strtoull+0x27/0x70
> [ 22.455572] xenbus_transaction_start+0x31/0x50
> [ 22.459104] netback_changed+0x76c/0xcc1 [xen_netfront]
> [ 22.463279] ? find_watch+0x40/0x40
> [ 22.466156] xenwatch_thread+0xb4/0x150
> [ 22.469309] ? wait_woken+0x80/0x80
> [ 22.472198] kthread+0x10e/0x130
> [ 22.474925] ? kthread_park+0x80/0x80
> [ 22.477946] ret_from_fork+0x35/0x40
> [ 22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront xen_blkfront
> [ 22.486783] ---[ end trace a9222030a747c3f7 ]---
> [ 22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60
>
> The virt_rmb() is added in the 'true' path of test_reply(). The "while"
> is changed to "do while" so that test_reply() is used as a read memory
> barrier.
>
> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
Reviewed-by: Julien Grall <jgrall@amazon.com>
> ---
> Changed since v1:
> - change "barrier()" to "virt_rmb()" in test_reply()
> Changed since v2:
> - Use "virt_rmb()" only in 'true' path
>
> drivers/xen/xenbus/xenbus_comms.c | 2 ++
> drivers/xen/xenbus/xenbus_xs.c | 9 ++++++---
> 2 files changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
> index d239fc3c5e3d..852ed161fc2a 100644
> --- a/drivers/xen/xenbus/xenbus_comms.c
> +++ b/drivers/xen/xenbus/xenbus_comms.c
> @@ -313,6 +313,8 @@ static int process_msg(void)
> req->msg.type = state.msg.type;
> req->msg.len = state.msg.len;
> req->body = state.body;
> + /* write body, then update state */
> + virt_wmb();
> req->state = xb_req_state_got_reply;
> req->cb(req);
> } else
> diff --git a/drivers/xen/xenbus/xenbus_xs.c b/drivers/xen/xenbus/xenbus_xs.c
> index ddc18da61834..3a06eb699f33 100644
> --- a/drivers/xen/xenbus/xenbus_xs.c
> +++ b/drivers/xen/xenbus/xenbus_xs.c
> @@ -191,8 +191,11 @@ static bool xenbus_ok(void)
>
> static bool test_reply(struct xb_req_data *req)
> {
> - if (req->state == xb_req_state_got_reply || !xenbus_ok())
> + if (req->state == xb_req_state_got_reply || !xenbus_ok()) {
> + /* read req->state before all other fields */
> + virt_rmb();
> return true;
> + }
>
> /* Make sure to reread req->state each time. */
> barrier();
> @@ -202,7 +205,7 @@ static bool test_reply(struct xb_req_data *req)
>
> static void *read_reply(struct xb_req_data *req)
> {
> - while (req->state != xb_req_state_got_reply) {
> + do {
> wait_event(req->wq, test_reply(req));
>
> if (!xenbus_ok())
> @@ -216,7 +219,7 @@ static void *read_reply(struct xb_req_data *req)
> if (req->err)
> return ERR_PTR(req->err);
>
> - }
> + } while (req->state != xb_req_state_got_reply);
>
> return req->body;
> }
>
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-devel] [PATCH v3 2/2] xenbus: req->err should be updated before req->state
2020-03-03 22:14 ` [Xen-devel] [PATCH v3 2/2] xenbus: req->err " Dongli Zhang
@ 2020-03-04 13:50 ` Julien Grall
2020-03-06 11:13 ` Boris Ostrovsky
1 sibling, 0 replies; 5+ messages in thread
From: Julien Grall @ 2020-03-04 13:50 UTC (permalink / raw)
To: Dongli Zhang, xen-devel, linux-kernel
Cc: jgross, boris.ostrovsky, sstabellini, joe.jin
Hi,
On 03/03/2020 22:14, Dongli Zhang wrote:
> This patch adds the barrier to guarantee that req->err is always updated
> before req->state.
>
> Otherwise, read_reply() would not return ERR_PTR(req->err) but
> req->body, when process_writes()->xb_write() is failed.
>
> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
Reviewed-by: Julien Grall <jgrall@amazon.com>
Cheers,
---
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-devel] [PATCH v3 2/2] xenbus: req->err should be updated before req->state
2020-03-03 22:14 ` [Xen-devel] [PATCH v3 2/2] xenbus: req->err " Dongli Zhang
2020-03-04 13:50 ` Julien Grall
@ 2020-03-06 11:13 ` Boris Ostrovsky
1 sibling, 0 replies; 5+ messages in thread
From: Boris Ostrovsky @ 2020-03-06 11:13 UTC (permalink / raw)
To: Dongli Zhang, xen-devel, linux-kernel
Cc: jgross, joe.jin, sstabellini, julien
On 3/3/20 5:14 PM, Dongli Zhang wrote:
> This patch adds the barrier to guarantee that req->err is always updated
> before req->state.
>
> Otherwise, read_reply() would not return ERR_PTR(req->err) but
> req->body, when process_writes()->xb_write() is failed.
>
> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
Applied to for-linus-5.6b.
-boris
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-03-06 11:13 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-03 22:14 [Xen-devel] [PATCH v3 1/2] xenbus: req->body should be updated before req->state Dongli Zhang
2020-03-03 22:14 ` [Xen-devel] [PATCH v3 2/2] xenbus: req->err " Dongli Zhang
2020-03-04 13:50 ` Julien Grall
2020-03-06 11:13 ` Boris Ostrovsky
2020-03-04 13:50 ` [Xen-devel] [PATCH v3 1/2] xenbus: req->body " Julien Grall
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).