* Re: xen-blkfront hang
[not found] ` <456a695c-8302-ee25-3e01-db3fe60425d9@oracle.com>
@ 2017-07-31 1:13 ` Dongli Zhang
[not found] ` <20170731063033.vjr7zizjazirzf5x@gavran.carpriv.carnet.hr>
2017-08-04 8:00 ` Valentin Vidic
2 siblings, 0 replies; 6+ messages in thread
From: Dongli Zhang @ 2017-07-31 1:13 UTC (permalink / raw)
To: Valentin Vidic, Konrad Rzeszutek Wilk
Cc: xen-devel, Junxiao Bi, Ankur Arora, Roger Pau Monné
CCed xen-devel so that more people would be able to help.
Dongli Zhang
On 07/31/2017 09:09 AM, Dongli Zhang wrote:
> Hi Valentin,
>
> On 07/30/2017 03:42 PM, Valentin Vidic wrote:
>> I'm having a problem with a domU hang in disk IO, described here:
>>
>> https://lists.xen.org/archives/html/xen-users/2017-07/msg00057.html
>>
>> Do you think this is a multi-queue issue and applying one of these
>> latest changes would help?
>>
>> xen/blkfront: always allocate grants first from per-queue persistent grants
>> https://github.com/torvalds/linux/commit/bd912ef3e46b6edb51bb8af4b73fd2be7817e305
>
> This patch is not able to fix the lack of grant issue permanently. It is used to
> optimize the utilization of grant table entires.
>
> To verify whether the above patch would help, please check the nr_grant_frames
> value in guest domU. If this value is exactly the same of maximum grant frames
> (by default, xen mainline uses 32) and the number of free grant references is
> very small, the above patch might help.
>
> The best way is to increase the gnttab_max_frames to larger value (e.g., 256)
> in dom0 xen.gz grub.
>
> Dongli Zhang
>
>>
>> xen-blkfront: fix mq start/stop race
>> https://github.com/torvalds/linux/commit/4b422cb99836de3d261faec20a0329385bdec43d
>>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xen-blkfront hang
[not found] ` <20170731063033.vjr7zizjazirzf5x@gavran.carpriv.carnet.hr>
@ 2017-07-31 6:40 ` Dongli Zhang
0 siblings, 0 replies; 6+ messages in thread
From: Dongli Zhang @ 2017-07-31 6:40 UTC (permalink / raw)
To: Valentin Vidic; +Cc: xen-devel, Roger Pau Monné, Ankur Arora, Junxiao Bi
Here are the options:
1. Dump a vmcore of guest and print nr_grant_frames with crash utility.
2. Implement a kernel module in guest to dump nr_grant_frames if you still have
access to your hung guest domU.
3. There is a new utility in xen toolstack at tools/misc/xen-diag.c to dump
grant table usage for arbitrary guest domU (including dom0)
./xen-diag gnttab_query_size [domid]
4. If your host's xen toolstack does not have xen-diag, feel free to implement
one your self via GNTTABOP_query_size hypercall and compile with -lxenctrl.
Dongli Zhang
On 07/31/2017 02:30 PM, Valentin Vidic wrote:
> On Mon, Jul 31, 2017 at 09:09:19AM +0800, Dongli Zhang wrote:
>> This patch is not able to fix the lack of grant issue permanently. It is used to
>> optimize the utilization of grant table entires.
>>
>> To verify whether the above patch would help, please check the nr_grant_frames
>> value in guest domU. If this value is exactly the same of maximum grant frames
>> (by default, xen mainline uses 32) and the number of free grant references is
>> very small, the above patch might help.
>
> I can try that, but how do I get the nr_grant_frames value in domU?
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xen-blkfront hang
[not found] ` <456a695c-8302-ee25-3e01-db3fe60425d9@oracle.com>
2017-07-31 1:13 ` xen-blkfront hang Dongli Zhang
[not found] ` <20170731063033.vjr7zizjazirzf5x@gavran.carpriv.carnet.hr>
@ 2017-08-04 8:00 ` Valentin Vidic
2017-08-04 9:13 ` Roger Pau Monné
2 siblings, 1 reply; 6+ messages in thread
From: Valentin Vidic @ 2017-08-04 8:00 UTC (permalink / raw)
To: Dongli Zhang; +Cc: xen-devel, Roger Pau Monné, Ankur Arora, Junxiao Bi
On Mon, Jul 31, 2017 at 09:09:19AM +0800, Dongli Zhang wrote:
> To verify whether the above patch would help, please check the nr_grant_frames
> value in guest domU. If this value is exactly the same of maximum grant frames
> (by default, xen mainline uses 32) and the number of free grant references is
> very small, the above patch might help.
You are right, this is what I get after the machine hangs:
crash> print nr_grant_frames
$1 = 32
crash> print gnttab_free_count
$2 = 9
> The best way is to increase the gnttab_max_frames to larger value (e.g., 256)
> in dom0 xen.gz grub.
Thank you, this seems to help. The test machine does not hang now and
the numbers are looking better now:
crash> print nr_grant_frames
$1 = 59
crash> print gnttab_free_count
$2 = 356
--
Valentin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xen-blkfront hang
2017-08-04 8:00 ` Valentin Vidic
@ 2017-08-04 9:13 ` Roger Pau Monné
2017-08-04 9:36 ` Dongli Zhang
0 siblings, 1 reply; 6+ messages in thread
From: Roger Pau Monné @ 2017-08-04 9:13 UTC (permalink / raw)
To: Valentin Vidic; +Cc: Dongli Zhang, Junxiao Bi, xen-devel, Ankur Arora
On Fri, Aug 04, 2017 at 10:00:09AM +0200, Valentin Vidic wrote:
> On Mon, Jul 31, 2017 at 09:09:19AM +0800, Dongli Zhang wrote:
> > To verify whether the above patch would help, please check the nr_grant_frames
> > value in guest domU. If this value is exactly the same of maximum grant frames
> > (by default, xen mainline uses 32) and the number of free grant references is
> > very small, the above patch might help.
>
> You are right, this is what I get after the machine hangs:
>
> crash> print nr_grant_frames
> $1 = 32
> crash> print gnttab_free_count
> $2 = 9
>
> > The best way is to increase the gnttab_max_frames to larger value (e.g., 256)
> > in dom0 xen.gz grub.
>
> Thank you, this seems to help. The test machine does not hang now and
> the numbers are looking better now:
>
> crash> print nr_grant_frames
> $1 = 59
> crash> print gnttab_free_count
> $2 = 356
At some point I've already expressed my opinion that having a
per-queue list of persistent grants was not a good idea. Now I think
the only solution is to remove persistent grants, or to lower the
default number of per-queue persistent grants to a ridiculously low
value.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xen-blkfront hang
2017-08-04 9:13 ` Roger Pau Monné
@ 2017-08-04 9:36 ` Dongli Zhang
2017-08-04 10:00 ` Roger Pau Monné
0 siblings, 1 reply; 6+ messages in thread
From: Dongli Zhang @ 2017-08-04 9:36 UTC (permalink / raw)
To: Roger Pau Monné, Valentin Vidic; +Cc: xen-devel, Junxiao Bi, Ankur Arora
On 08/04/2017 05:13 PM, Roger Pau Monné wrote:
> On Fri, Aug 04, 2017 at 10:00:09AM +0200, Valentin Vidic wrote:
>> On Mon, Jul 31, 2017 at 09:09:19AM +0800, Dongli Zhang wrote:
>>> To verify whether the above patch would help, please check the nr_grant_frames
>>> value in guest domU. If this value is exactly the same of maximum grant frames
>>> (by default, xen mainline uses 32) and the number of free grant references is
>>> very small, the above patch might help.
>>
>> You are right, this is what I get after the machine hangs:
>>
>> crash> print nr_grant_frames
>> $1 = 32
>> crash> print gnttab_free_count
>> $2 = 9
>>
>>> The best way is to increase the gnttab_max_frames to larger value (e.g., 256)
>>> in dom0 xen.gz grub.
>>
>> Thank you, this seems to help. The test machine does not hang now and
>> the numbers are looking better now:
>>
>> crash> print nr_grant_frames
>> $1 = 59
>> crash> print gnttab_free_count
>> $2 = 356
>
> At some point I've already expressed my opinion that having a
> per-queue list of persistent grants was not a good idea. Now I think
> the only solution is to remove persistent grants, or to lower the
> default number of per-queue persistent grants to a ridiculously low
> value.
It would be more efficient if (1) persistent grants are shared by all queues and
(2) there is a new mechanism to allow frontend to actively ask backend to unmap
existing persistent grants. So far, it is backend's responsibility to decide
when to unmap persistent grants.
Dongli Zhang
>
> Roger.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xen-blkfront hang
2017-08-04 9:36 ` Dongli Zhang
@ 2017-08-04 10:00 ` Roger Pau Monné
0 siblings, 0 replies; 6+ messages in thread
From: Roger Pau Monné @ 2017-08-04 10:00 UTC (permalink / raw)
To: Dongli Zhang; +Cc: xen-devel, Ankur Arora, Junxiao Bi, Valentin Vidic
On Fri, Aug 04, 2017 at 05:36:29PM +0800, Dongli Zhang wrote:
>
>
> On 08/04/2017 05:13 PM, Roger Pau Monné wrote:
> > On Fri, Aug 04, 2017 at 10:00:09AM +0200, Valentin Vidic wrote:
> >> On Mon, Jul 31, 2017 at 09:09:19AM +0800, Dongli Zhang wrote:
> >>> To verify whether the above patch would help, please check the nr_grant_frames
> >>> value in guest domU. If this value is exactly the same of maximum grant frames
> >>> (by default, xen mainline uses 32) and the number of free grant references is
> >>> very small, the above patch might help.
> >>
> >> You are right, this is what I get after the machine hangs:
> >>
> >> crash> print nr_grant_frames
> >> $1 = 32
> >> crash> print gnttab_free_count
> >> $2 = 9
> >>
> >>> The best way is to increase the gnttab_max_frames to larger value (e.g., 256)
> >>> in dom0 xen.gz grub.
> >>
> >> Thank you, this seems to help. The test machine does not hang now and
> >> the numbers are looking better now:
> >>
> >> crash> print nr_grant_frames
> >> $1 = 59
> >> crash> print gnttab_free_count
> >> $2 = 356
> >
> > At some point I've already expressed my opinion that having a
> > per-queue list of persistent grants was not a good idea. Now I think
> > the only solution is to remove persistent grants, or to lower the
> > default number of per-queue persistent grants to a ridiculously low
> > value.
>
> It would be more efficient if (1) persistent grants are shared by all queues and
There was a complain that having a shared pool of persistent grants
introduced too much contention.
> (2) there is a new mechanism to allow frontend to actively ask backend to unmap
> existing persistent grants. So far, it is backend's responsibility to decide
> when to unmap persistent grants.
Hm, I would rather remove them than make the protocol more complex.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-08-04 10:00 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20170730074208.6gjcdieop76skdrx@gavran.carpriv.carnet.hr>
[not found] ` <456a695c-8302-ee25-3e01-db3fe60425d9@oracle.com>
2017-07-31 1:13 ` xen-blkfront hang Dongli Zhang
[not found] ` <20170731063033.vjr7zizjazirzf5x@gavran.carpriv.carnet.hr>
2017-07-31 6:40 ` Dongli Zhang
2017-08-04 8:00 ` Valentin Vidic
2017-08-04 9:13 ` Roger Pau Monné
2017-08-04 9:36 ` Dongli Zhang
2017-08-04 10:00 ` Roger Pau Monné
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.