Re: Question about sharing spinlock_t among VMs in Xen

From: Meng Xu <xumengpanda@gmail.com>
To: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: "xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>
Subject: Re: Question about sharing spinlock_t among VMs in Xen
Date: Mon, 13 Jun 2016 16:46:37 -0400	[thread overview]
Message-ID: <CAENZ-+nirrxqhSvQQ+MiD=xMy=g3kLEActr4+epANQQEqUni9w@mail.gmail.com> (raw)
In-Reply-To: <575EFB59.1000202@oracle.com>

On Mon, Jun 13, 2016 at 2:28 PM, Boris Ostrovsky
<boris.ostrovsky@oracle.com> wrote:
> On 06/13/2016 01:43 PM, Meng Xu wrote:
>> Hi,
>>
>> I have a quick question about using the Linux spin_lock() in Xen
>> environment to protect some host-wide shared (memory) resource among
>> VMs.
>>
>> *** The question is as follows ***
>> Suppose I have two Linux VMs sharing the same spinlock_t lock (through
>> the sharing memory) on the same host. Suppose we have one process in
>> each VM. Each process uses the linux function spin_lock(&lock) [1] to
>> grab & release the lock.
>> Will these two processes in the two VMs have race on the shared lock?
>
> You can't do this: depending on which Linux version you use you will
> find that kernel uses ticketlocks or qlocks locks which keep track of
> who is holding the lock (obviously this information is internal to VM).

Yes, we are using the ticketlocks. We expose this information to other
VMs as well. That's why I'm guessing some data can be corrupted due to
races among VMs that share the same ticketlock information.

> On top of this on Xen we use pvlocks which add another (internal)
> control layer.

I see. Will have a look.

>
>
>>
>> My speculation is that it should have the race on the shard lock when
>> the spin_lock() function in *two VMs* operate on the same lock.
>>
>> We did some quick experiment on this and we found one VM sometimes see
>> the soft lockup on the lock. But we want to make sure our
>> understanding is correct.
>>
>> We are exploring if we can use the spin_lock to protect the shared
>> resources among VMs, instead of using the PV drivers. If the
>> spin_lock() in linux can provide the host-wide atomicity (which will
>> surprise me, though), that will be great. Otherwise, we probably have
>> to expose the spin_lock in Xen to the Linux?
>
> I'd think this has to be via the hypervisor (or some other third party).
> Otherwise what happens if one of the guests dies while holding the lock?

Right. It's a good point. This could potentially cause all other VMs
that share the same lock spin on the lock forever.
We will try out to have it done within hypervisor.

Thank you so much for your explanation, Boris!

Best Regards,

Meng

-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel