Re: null domains after xl destroy

[Glenn Enright <glenn@rimuhosting.com>]

On 15/05/17 21:57, Juergen Gross wrote:
> On 13/05/17 06:02, Glenn Enright wrote:
>> On 09/05/17 21:24, Roger Pau Monné wrote:
>>> On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote:
>>>> On 04/05/17 00:17, Glenn Enright wrote:
>>>>> On 04/05/17 04:58, Steven Haigh wrote:
>>>>>> On 04/05/17 01:53, Juergen Gross wrote:
>>>>>>> On 03/05/17 12:45, Steven Haigh wrote:
>>>>>>>> Just wanted to give this a little nudge now people seem to be
>>>>>>>> back on
>>>>>>>> deck...
>>>>>>>
>>>>>>> Glenn, could you please give the attached patch a try?
>>>>>>>
>>>>>>> It should be applied on top of the other correction, the old debug
>>>>>>> patch should not be applied.
>>>>>>>
>>>>>>> I have added some debug output to make sure we see what is happening.
>>>>>>
>>>>>> This patch is included in kernel-xen-4.9.26-1
>>>>>>
>>>>>> It should be in the repos now.
>>>>>>
>>>>>
>>>>> Still seeing the same issue. Without the extra debug patch all I see in
>>>>> the logs after destroy is this...
>>>>>
>>>>> xen-blkback: xen_blkif_disconnect: busy
>>>>> xen-blkback: xen_blkif_free: delayed = 0
>>>>
>>>> Hmm, to me it seems as if some grant isn't being unmapped.
>>>>
>>>> Looking at gnttab_unmap_refs_async() I wonder how this is supposed to
>>>> work:
>>>>
>>>> I don't see how a grant would ever be unmapped in case of
>>>> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it
>>>> does is deferring the call to the unmap operation again and again. Or
>>>> am I missing something here?
>>>
>>> No, I don't think you are missing anything, but I cannot see how this
>>> can be
>>> solved in a better way, unmapping a page that's still referenced is
>>> certainly
>>> not the best option, or else we risk triggering a page-fault elsewhere.
>>>
>>> IMHO, gnttab_unmap_refs_async should have a timeout, and return an
>>> error at
>>> some point. Also, I'm wondering whether there's a way to keep track of
>>> who has
>>> references on a specific page, but so far I haven't been able to
>>> figure out how
>>> to get this information from Linux.
>>>
>>> Also, I've noticed that __gnttab_unmap_refs_async uses page_count,
>>> shouldn't it
>>> use page_ref_count instead?
>>>
>>> Roger.
>>>
>>
>> In case it helps, I have continued to work on this. I notices processed
>> left behind (under 4.9.27). The same issue is ongoing.
>>
>> # ps auxf | grep [x]vda
>> root      2983  0.0  0.0      0     0 ?        S    01:44   0:00  \_
>> [1.xvda1-1]
>> root      5457  0.0  0.0      0     0 ?        S    02:06   0:00  \_
>> [3.xvda1-1]
>> root      7382  0.0  0.0      0     0 ?        S    02:36   0:00  \_
>> [4.xvda1-1]
>> root      9668  0.0  0.0      0     0 ?        S    02:51   0:00  \_
>> [6.xvda1-1]
>> root     11080  0.0  0.0      0     0 ?        S    02:57   0:00  \_
>> [7.xvda1-1]
>>
>> # xl list
>> Name                              ID   Mem VCPUs      State   Time(s)
>> Domain-0                          0  1512     2     r-----     118.5
>> (null)                            1     8     4     --p--d      43.8
>> (null)                            3     8     4     --p--d       6.3
>> (null)                            4     8     4     --p--d      73.4
>> (null)                            6     8     4     --p--d      14.7
>> (null)                            7     8     4     --p--d      30
>>
>> Those all have...
>>
>> [root 11080]# cat wchan
>> xen_blkif_schedule
>>
>> [root 11080]# cat stack
>> [<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40
>> [<ffffffff810a0555>] kthread+0xe5/0x100
>> [<ffffffff816f1c45>] ret_from_fork+0x25/0x30
>> [<ffffffffffffffff>] 0xffffffffffffffff
>
> And found another reference count bug. Would you like to give the
> attached patch (to be applied additionally to the previous ones) a try?
>
>
> Juergen
>

This seems to have solved the issue in 4.9.28, with all three patches 
applied. Awesome!

On my main test machine I can no longer replicate what I was originally 
seeing, and in dmesg I now see this flow...

xen-blkback: xen_blkif_disconnect: busy
xen-blkback: xen_blkif_free: delayed = 1
xen-blkback: xen_blkif_free: delayed = 0

xl list is clean, xenstore looks right. No extraneous processes left over.

Thankyou Juergen, so much. Really appreciate your persistence with this. 
Anything I can do to help push this upstream please let me know. Feel 
free to add a reported-by line with my name if you think it appropriate.

Regards, Glenn
http://rimuhosting.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel