vsock CID questions

vsock CID questions

[ted.h.kim]

Hi Stefano (and Stefan),

I have some questions about vsock CIDs, particularly when migration happens.

1. Is there an API to lookup CIDs of guests from the host side (in 
libvirt)?

2. In the vsock(7) man page, it says the CID might change upon 
migration, if it is not available.
Is there some notification when CID reassignment happens?

3. if CID reassignment happens, is this persistent? (i.e. will I see 
updated vsock definition in XML for the guest)

4. I would like to minimize the chance of CID collision. If I understand 
correctly, the CID is a 32-bit unsigned. So for my application, it might 
work to put an IPv4 address. But if I adopt this convention, then I need 
to look forward to possibly using IPv6. Anyway, would it be hard to 
potentially expand the size of the CID to 64 bits or even 128?

Thanks,
-ted

-- 
Ted H. Kim, PhD
ted.h.kim@oracle.com
+1 310-258-7515

Re: vsock CID questions

[Stefano Garzarella]

On Tue, Feb 18, 2020 at 02:45:38PM -0800, ted.h.kim@oracle.com wrote:
> Hi Stefano (and Stefan),

Hi Ted,

> 
> I have some questions about vsock CIDs, particularly when migration happens.
> 
> 1. Is there an API to lookup CIDs of guests from the host side (in libvirt)?

I don't know if there is a specific API, but looking at the xml, you can see
the assigned CID:

$ virsh dumpxml fedora31 | grep cid
      <cid auto='yes' address='3'/>

I'm not sure that's what you were asking, if you meant a list of all the
guest CIDs, I don't think there's an API for that.

> 
> 2. In the vsock(7) man page, it says the CID might change upon migration, if
> it is not available.
> Is there some notification when CID reassignment happens?

Connected stream sockets will receive an error after the migration and then
they'll be closed.

Usually it is not recommended to bind the guest's cid, it is preferable
to use VMADDR_CID_ANY.

> 
> 3. if CID reassignment happens, is this persistent? (i.e. will I see updated
> vsock definition in XML for the guest)

I guess so, but I didn't try.

> 
> 4. I would like to minimize the chance of CID collision. If I understand
> correctly, the CID is a 32-bit unsigned.

Right. 'struct sockaddr_vm' supports 32-bit unsigned CID.

>                                          So for my application, it might
> work to put an IPv4 address. But if I adopt this convention, then I need to
> look forward to possibly using IPv6. Anyway, would it be hard to potentially
> expand the size of the CID to 64 bits or even 128?

virtio-vsock specification [1] supports up to 64-bit CID.
The 'svm_cid' field in the 'struct sockaddr_vm' is the last one, before
the zero section, and we have 16-bit reserved on top that we can use for
some flags.
Maybe extending it to 64 bit might be feasible, but we need to check
other transports (vmci, hyperv).

Cheers,
Stefano

[1] https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-3960006

Re: vsock CID questions

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

On Tue, Feb 18, 2020 at 02:45:38PM -0800, ted.h.kim@oracle.com wrote:
> 1. Is there an API to lookup CIDs of guests from the host side (in libvirt)?

I wonder if it can be queried from libvirt (at a minimum the domain XML
might have the CID)?  I have CCed Ján Tomko who worked on the libvirt
support:

https://libvirt.org/formatdomain.html#vsock

> 2. In the vsock(7) man page, it says the CID might change upon migration, if
> it is not available.
> Is there some notification when CID reassignment happens?

All established connections are reset across live migration -
applications will notice :).

Listen sockets stay open but automatically listen on the new CID.

> 3. if CID reassignment happens, is this persistent? (i.e. will I see updated
> vsock definition in XML for the guest)

Another question for Ján.

> 4. I would like to minimize the chance of CID collision. If I understand
> correctly, the CID is a 32-bit unsigned. So for my application, it might
> work to put an IPv4 address. But if I adopt this convention, then I need to
> look forward to possibly using IPv6. Anyway, would it be hard to potentially
> expand the size of the CID to 64 bits or even 128?

A little hard, since the struct sockaddr_vm that userspace applications
use has a 32-bit CID field.  This is because the existing VMware VMCI
vsock implementation has 32-bit CIDs.

virtio-vsock is ready for 64-bit CIDs (the packet header fields are
already 64-bit) but changes to net/vmw_vsock/ core code and to the
userspace ABI would be necessary.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: vsock CID questions

[Ján Tomko]

[-- Attachment #1: Type: text/plain, Size: 2351 bytes --]

On Wed, Feb 19, 2020 at 03:43:17PM +0000, Stefan Hajnoczi wrote:
>On Tue, Feb 18, 2020 at 02:45:38PM -0800, ted.h.kim@oracle.com wrote:
>> 1. Is there an API to lookup CIDs of guests from the host side (in libvirt)?
>
>I wonder if it can be queried from libvirt (at a minimum the domain XML
>might have the CID)?  I have CCed Ján Tomko who worked on the libvirt
>support:
>
>https://libvirt.org/formatdomain.html#vsock
>

Yes, libvirt has to know the CIDs of the guest and presents them in the
domain XML:
<domain type='kvm'>
   <name>test</name>
   ...
   <devices>
     ...
     <vsock model='virtio'>
       <cid auto='no' address='4'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
     </vsock>
   </devices>
</domain>

>> 2. In the vsock(7) man page, it says the CID might change upon migration, if
>> it is not available.
>> Is there some notification when CID reassignment happens?
>
>All established connections are reset across live migration -
>applications will notice :).
>
>Listen sockets stay open but automatically listen on the new CID.
>
>> 3. if CID reassignment happens, is this persistent? (i.e. will I see updated
>> vsock definition in XML for the guest)
>
>Another question for Ján.

Depends on the setting.
For <cid auto='yes'/>, libvirt will try to acquire the first available CID
for the guest and pass it to QEMU.
For <cid auto='no'/>, no reassignment should happend and the CID
requested in the domain XML on the source will be used (or fail to be
used) on the destination.

Jano

>
>> 4. I would like to minimize the chance of CID collision. If I understand
>> correctly, the CID is a 32-bit unsigned. So for my application, it might
>> work to put an IPv4 address. But if I adopt this convention, then I need to
>> look forward to possibly using IPv6. Anyway, would it be hard to potentially
>> expand the size of the CID to 64 bits or even 128?
>
>A little hard, since the struct sockaddr_vm that userspace applications
>use has a 32-bit CID field.  This is because the existing VMware VMCI
>vsock implementation has 32-bit CIDs.
>
>virtio-vsock is ready for 64-bit CIDs (the packet header fields are
>already 64-bit) but changes to net/vmw_vsock/ core code and to the
>userspace ABI would be necessary.
>
>Stefan



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: vsock CID questions

[ted.h.kim]

Hi Jan,

Thanks for responding - let me see if I am understanding correctly.


I think you are saying on migration the process of determining the CID 
assigned is the same as when you start a domain.

<cid auto='yes'> means assignment to the first available value.
It also seems for auto='yes' that any address='<value>' part of the CID 
definition is ignored, even as a suggested value.
(I always get CID 3 when starting auto='yes' and no other domains have 
started, even if there is an specific address in the definition, e.g. 
address='12'.)

But if auto='no', either the domain gets the address field value or if 
the value is already assigned, the domain will fail to start/migrate.

Is that right?


In cases, where auto='yes', it does not seem that the host/hypervisor 
can find out what CID value was assigned to a domain. Even parsing the 
XML only reveals that it was auto-assigned and no specific value can be 
determined.

Is this correct?

If this is the case, I would advocate for a specific API which can 
lookup the current CID of a domain.
Otherwise the host/hypervisor cannot tell which  auto='yes' domain is on 
the other end of the connected socket, when there is more than one.


Thanks.
-ted


On 2/20/20 8:09 AM, Ján Tomko wrote:
> On Wed, Feb 19, 2020 at 03:43:17PM +0000, Stefan Hajnoczi wrote:
>> On Tue, Feb 18, 2020 at 02:45:38PM -0800, ted.h.kim@oracle.com wrote:
>>> 1. Is there an API to lookup CIDs of guests from the host side (in 
>>> libvirt)?
>>
>> I wonder if it can be queried from libvirt (at a minimum the domain XML
>> might have the CID)?  I have CCed Ján Tomko who worked on the libvirt
>> support:
>>
>> https://libvirt.org/formatdomain.html#vsock
>>
>
> Yes, libvirt has to know the CIDs of the guest and presents them in the
> domain XML:
> <domain type='kvm'>
>   <name>test</name>
>   ...
>   <devices>
>     ...
>     <vsock model='virtio'>
>       <cid auto='no' address='4'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07' 
> function='0x0'/>
>     </vsock>
>   </devices>
> </domain>
>
>>> 2. In the vsock(7) man page, it says the CID might change upon 
>>> migration, if
>>> it is not available.
>>> Is there some notification when CID reassignment happens?
>>
>> All established connections are reset across live migration -
>> applications will notice :).
>>
>> Listen sockets stay open but automatically listen on the new CID.
>>
>>> 3. if CID reassignment happens, is this persistent? (i.e. will I see 
>>> updated
>>> vsock definition in XML for the guest)
>>
>> Another question for Ján.
>
> Depends on the setting.
> For <cid auto='yes'/>, libvirt will try to acquire the first available 
> CID
> for the guest and pass it to QEMU.
> For <cid auto='no'/>, no reassignment should happend and the CID
> requested in the domain XML on the source will be used (or fail to be
> used) on the destination.
>
> Jano
>
>>
>>> 4. I would like to minimize the chance of CID collision. If I 
>>> understand
>>> correctly, the CID is a 32-bit unsigned. So for my application, it 
>>> might
>>> work to put an IPv4 address. But if I adopt this convention, then I 
>>> need to
>>> look forward to possibly using IPv6. Anyway, would it be hard to 
>>> potentially
>>> expand the size of the CID to 64 bits or even 128?
>>
>> A little hard, since the struct sockaddr_vm that userspace applications
>> use has a 32-bit CID field.  This is because the existing VMware VMCI
>> vsock implementation has 32-bit CIDs.
>>
>> virtio-vsock is ready for 64-bit CIDs (the packet header fields are
>> already 64-bit) but changes to net/vmw_vsock/ core code and to the
>> userspace ABI would be necessary.
>>
>> Stefan
>
>
-- 
Ted H. Kim, PhD
ted.h.kim@oracle.com
+1 310-258-7515

Re: vsock CID questions

[Ján Tomko]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

On Fri, Feb 21, 2020 at 11:49:06AM -0800, ted.h.kim@oracle.com wrote:
>Hi Jan,
>
>Thanks for responding - let me see if I am understanding correctly.
>
>
>I think you are saying on migration the process of determining the CID 
>assigned is the same as when you start a domain.
>
><cid auto='yes'> means assignment to the first available value.
>It also seems for auto='yes' that any address='<value>' part of the 
>CID definition is ignored, even as a suggested value.
>(I always get CID 3 when starting auto='yes' and no other domains have 
>started, even if there is an specific address in the definition, e.g. 
>address='12'.)
>
>But if auto='no', either the domain gets the address field value or if 
>the value is already assigned, the domain will fail to start/migrate.
>
>Is that right?
>

Yes.

>
>In cases, where auto='yes', it does not seem that the host/hypervisor 
>can find out what CID value was assigned to a domain. Even parsing the 
>XML only reveals that it was auto-assigned and no specific value can 
>be determined.

The value is not recorded in the inactive XML, but after domain startup,
the address should be visible in the live XML:

   <vsock model="virtio">
     <cid auto="yes" address="3"/>
     <alias name="vsock0"/>
     <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
   </vsock>

>
>Is this correct?
>
>If this is the case, I would advocate for a specific API which can 
>lookup the current CID of a domain.

This can already be queried from the XML through virDomainGetXMLDesc:
$ virsh dumpxml libvirt-fedora-31 | xmllint --xpath 'string(//cid/@address)' -
3

Jano

>Otherwise the host/hypervisor cannot tell which  auto='yes' domain is 
>on the other end of the connected socket, when there is more than one.
>
>
>Thanks.
>-ted
>
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]