obtain a list of granttable PFNs from toolstack

obtain a list of granttable PFNs from toolstack

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 118 bytes --]

Is there an API to get a list of PFNs which are currently being active in the grant table for a given domid?

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: obtain a list of granttable PFNs from toolstack

[Jan Beulich]

On 28.04.2021 09:50, Olaf Hering wrote:
> Is there an API to get a list of PFNs which are currently being active in the grant table for a given domid?

Do you really mean PFNs? For PV, the P <=> M relationship is
guest-managed, and hence any value returned here would be stale by
the time the caller gets to use it, even if the grant table itself
didn't change.

Also, what do you mean by "active in the grant table"? A grant
present in the table referencing said page? Or a page comprising
the grant table? In the latter case, the acquire-resource interface
would allow you to obtain the MFNs from a PV guest; a HVM caller
would specify GFNs as input, for the MFNs to get (foreign-)mapped
at instead.

Jan

Re: obtain a list of granttable PFNs from toolstack

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

Am Wed, 28 Apr 2021 11:49:59 +0200
schrieb Jan Beulich <jbeulich@suse.com>:

> On 28.04.2021 09:50, Olaf Hering wrote:
> > Is there an API to get a list of PFNs which are currently being active in the grant table for a given domid?  
> Do you really mean PFNs?

The save/restore code uses the term "pfn" to refer to page indices from the domU point of view. Since I usually just follow the HVM code paths, the term might indeed be incorrect for PV.

> Also, what do you mean by "active in the grant table"?

I'm not familiar with grants. My guess is, a domU pro-actively shares some of its pages with someone else. These shared pages have to be tracked somewhere, and I guess Xen has to establish proper access permissions for such pages. I guess the "grant table" is the place where things are collected.

Looking through xen/grant_table.h, I see no API. But that might be the wrong place anyway.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: obtain a list of granttable PFNs from toolstack

[Jan Beulich]

On 28.04.2021 12:17, Olaf Hering wrote:
> Am Wed, 28 Apr 2021 11:49:59 +0200
> schrieb Jan Beulich <jbeulich@suse.com>:
>> On 28.04.2021 09:50, Olaf Hering wrote:
>>> Is there an API to get a list of PFNs which are currently being active in the grant table for a given domid?  
>> 
>> Also, what do you mean by "active in the grant table"?
> 
> I'm not familiar with grants. My guess is, a domU pro-actively shares some of its pages with someone else. These shared pages have to be tracked somewhere, and I guess Xen has to establish proper access permissions for such pages. I guess the "grant table" is the place where things are collected.

The grant table is where the guest stores which pages it allows to
be accessed by which other domains. Upon request to access a grant,
Xen then updates that table entry to indicate what access is
currently in progress.

> Looking through xen/grant_table.h, I see no API. But that might be the wrong place anyway.

There's none, and so far I can't see a reason why one would be
wanted / needed. Any inspection results are immediately stale
afterwards, as there's no synchronization at all to what guests
may put when into their grant table.

It might help if you would outline what you're actually after by
knowing a domain's active grants.

Jan

Re: obtain a list of granttable PFNs from toolstack

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

Am Wed, 28 Apr 2021 12:42:54 +0200
schrieb Jan Beulich <jbeulich@suse.com>:

> It might help if you would outline what you're actually after by
> knowing a domain's active grants.

There is REC_TYPE_VERIFY, which will re-transmit all memory of a paused domU and compare what was previously transferred. Last time I used it a large number of pages are different. I did not spent time to dig further into it, what pfns are affected, how the pages differ. I suspect the backends do still write into the granted pages, hence the question.

I think the domU may undo all grants during its suspend anyway, that is something to explore as well.


Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: obtain a list of granttable PFNs from toolstack

[Andrew Cooper]

On 28/04/2021 11:17, Olaf Hering wrote:
> Am Wed, 28 Apr 2021 11:49:59 +0200
> schrieb Jan Beulich <jbeulich@suse.com>:
>
>> On 28.04.2021 09:50, Olaf Hering wrote:
>>> Is there an API to get a list of PFNs which are currently being active in the grant table for a given domid?  
>> Do you really mean PFNs?
> The save/restore code uses the term "pfn" to refer to page indices from the domU point of view. Since I usually just follow the HVM code paths, the term might indeed be incorrect for PV.

pfn's is correct here.  The migration logic tracks dirty pages in the
same address space as Xen's logdirty bitmap.

For HVM guests, this is ==gfn.

For PV guests, this is the guest's logical idea of its physmap layout,
as written in the M2P, and specifically not mfn/gfn's.

~Andrew

Re: obtain a list of granttable PFNs from toolstack

[Andrew Cooper]

On 28/04/2021 12:03, Olaf Hering wrote:
> Am Wed, 28 Apr 2021 12:42:54 +0200
> schrieb Jan Beulich <jbeulich@suse.com>:
>
>> It might help if you would outline what you're actually after by
>> knowing a domain's active grants.
> There is REC_TYPE_VERIFY, which will re-transmit all memory of a paused domU and compare what was previously transferred. Last time I used it a large number of pages are different. I did not spent time to dig further into it, what pfns are affected, how the pages differ. I suspect the backends do still write into the granted pages, hence the question.
>
> I think the domU may undo all grants during its suspend anyway, that is something to explore as well.

Grants are not properly accounted for in the logdirty bitmap, because
they're not unmapped during the final phase.

This came as an unexpected surprise when I was doing migration v2.  You
really will see `--debug` identify dirty frames, but its "fine", allegedly.

The reason is that frontends and backends can't disconnect while the VM
is paused, and will continue to process outstanding IO.  The reason this
doesn't explode on the destination is because the guest kernel replays
the command ring.

Disks get possibly re-issued read/writes.  Networks really does drop
some to-guest packets, and may send duplicate to-host packets - TCP
handles this normal mechanisms, and UDP is UDP.  Other PV protocols will
encounter similar problems.

I didn't get the impression that this behaviour was intentional in the
first place, and its certainly not something you'd deliberately design. 
It appears to have been a consequence of how PV guests evolved, where
kernel cooperation on migrate is mandatory, which caused kernel
cooperation to be mandatory for the paravirt driver pairs too.

~Andrew

Re: obtain a list of granttable PFNs from toolstack

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

Am Wed, 28 Apr 2021 12:43:45 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> Grants are not properly accounted for in the logdirty bitmap, because
> they're not unmapped during the final phase.

Yeah, which means the verify code can do no useful work.

This can be addressed by either removing it. Or by giving the granted pages a dedicate type so that the verify code can skip them. I assume if assigning a dedicated type is easy, it would have been done already.


Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]