* Question about sharing spinlock_t among VMs in Xen @ 2016-06-13 17:43 Meng Xu 2016-06-13 18:28 ` Boris Ostrovsky 2016-06-13 22:54 ` Andrew Cooper 0 siblings, 2 replies; 9+ messages in thread From: Meng Xu @ 2016-06-13 17:43 UTC (permalink / raw) To: xen-devel Hi, I have a quick question about using the Linux spin_lock() in Xen environment to protect some host-wide shared (memory) resource among VMs. *** The question is as follows *** Suppose I have two Linux VMs sharing the same spinlock_t lock (through the sharing memory) on the same host. Suppose we have one process in each VM. Each process uses the linux function spin_lock(&lock) [1] to grab & release the lock. Will these two processes in the two VMs have race on the shared lock? My speculation is that it should have the race on the shard lock when the spin_lock() function in *two VMs* operate on the same lock. We did some quick experiment on this and we found one VM sometimes see the soft lockup on the lock. But we want to make sure our understanding is correct. We are exploring if we can use the spin_lock to protect the shared resources among VMs, instead of using the PV drivers. If the spin_lock() in linux can provide the host-wide atomicity (which will surprise me, though), that will be great. Otherwise, we probably have to expose the spin_lock in Xen to the Linux? [1] http://lxr.free-electrons.com/source/include/linux/spinlock.h#L300 Thank you very much for your time and help in this question! Best Regards, Meng ----------- Meng Xu PhD Student in Computer and Information Science University of Pennsylvania http://www.cis.upenn.edu/~mengxu/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Question about sharing spinlock_t among VMs in Xen 2016-06-13 17:43 Question about sharing spinlock_t among VMs in Xen Meng Xu @ 2016-06-13 18:28 ` Boris Ostrovsky 2016-06-13 20:46 ` Meng Xu 2016-06-13 22:54 ` Andrew Cooper 1 sibling, 1 reply; 9+ messages in thread From: Boris Ostrovsky @ 2016-06-13 18:28 UTC (permalink / raw) To: Meng Xu, xen-devel On 06/13/2016 01:43 PM, Meng Xu wrote: > Hi, > > I have a quick question about using the Linux spin_lock() in Xen > environment to protect some host-wide shared (memory) resource among > VMs. > > *** The question is as follows *** > Suppose I have two Linux VMs sharing the same spinlock_t lock (through > the sharing memory) on the same host. Suppose we have one process in > each VM. Each process uses the linux function spin_lock(&lock) [1] to > grab & release the lock. > Will these two processes in the two VMs have race on the shared lock? You can't do this: depending on which Linux version you use you will find that kernel uses ticketlocks or qlocks locks which keep track of who is holding the lock (obviously this information is internal to VM). On top of this on Xen we use pvlocks which add another (internal) control layer. > > My speculation is that it should have the race on the shard lock when > the spin_lock() function in *two VMs* operate on the same lock. > > We did some quick experiment on this and we found one VM sometimes see > the soft lockup on the lock. But we want to make sure our > understanding is correct. > > We are exploring if we can use the spin_lock to protect the shared > resources among VMs, instead of using the PV drivers. If the > spin_lock() in linux can provide the host-wide atomicity (which will > surprise me, though), that will be great. Otherwise, we probably have > to expose the spin_lock in Xen to the Linux? I'd think this has to be via the hypervisor (or some other third party). Otherwise what happens if one of the guests dies while holding the lock? -boris > > [1] http://lxr.free-electrons.com/source/include/linux/spinlock.h#L300 > > Thank you very much for your time and help in this question! > > Best Regards, > > Meng > > ----------- > Meng Xu > PhD Student in Computer and Information Science > University of Pennsylvania > http://www.cis.upenn.edu/~mengxu/ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Question about sharing spinlock_t among VMs in Xen 2016-06-13 18:28 ` Boris Ostrovsky @ 2016-06-13 20:46 ` Meng Xu 2016-06-13 21:17 ` Boris Ostrovsky 0 siblings, 1 reply; 9+ messages in thread From: Meng Xu @ 2016-06-13 20:46 UTC (permalink / raw) To: Boris Ostrovsky; +Cc: xen-devel On Mon, Jun 13, 2016 at 2:28 PM, Boris Ostrovsky <boris.ostrovsky@oracle.com> wrote: > On 06/13/2016 01:43 PM, Meng Xu wrote: >> Hi, >> >> I have a quick question about using the Linux spin_lock() in Xen >> environment to protect some host-wide shared (memory) resource among >> VMs. >> >> *** The question is as follows *** >> Suppose I have two Linux VMs sharing the same spinlock_t lock (through >> the sharing memory) on the same host. Suppose we have one process in >> each VM. Each process uses the linux function spin_lock(&lock) [1] to >> grab & release the lock. >> Will these two processes in the two VMs have race on the shared lock? > > You can't do this: depending on which Linux version you use you will > find that kernel uses ticketlocks or qlocks locks which keep track of > who is holding the lock (obviously this information is internal to VM). Yes, we are using the ticketlocks. We expose this information to other VMs as well. That's why I'm guessing some data can be corrupted due to races among VMs that share the same ticketlock information. > On top of this on Xen we use pvlocks which add another (internal) > control layer. I see. Will have a look. > > >> >> My speculation is that it should have the race on the shard lock when >> the spin_lock() function in *two VMs* operate on the same lock. >> >> We did some quick experiment on this and we found one VM sometimes see >> the soft lockup on the lock. But we want to make sure our >> understanding is correct. >> >> We are exploring if we can use the spin_lock to protect the shared >> resources among VMs, instead of using the PV drivers. If the >> spin_lock() in linux can provide the host-wide atomicity (which will >> surprise me, though), that will be great. Otherwise, we probably have >> to expose the spin_lock in Xen to the Linux? > > I'd think this has to be via the hypervisor (or some other third party). > Otherwise what happens if one of the guests dies while holding the lock? Right. It's a good point. This could potentially cause all other VMs that share the same lock spin on the lock forever. We will try out to have it done within hypervisor. Thank you so much for your explanation, Boris! Best Regards, Meng ----------- Meng Xu PhD Student in Computer and Information Science University of Pennsylvania http://www.cis.upenn.edu/~mengxu/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Question about sharing spinlock_t among VMs in Xen 2016-06-13 20:46 ` Meng Xu @ 2016-06-13 21:17 ` Boris Ostrovsky 2016-06-14 1:50 ` Meng Xu 0 siblings, 1 reply; 9+ messages in thread From: Boris Ostrovsky @ 2016-06-13 21:17 UTC (permalink / raw) To: Meng Xu; +Cc: xen-devel On 06/13/2016 04:46 PM, Meng Xu wrote: > On Mon, Jun 13, 2016 at 2:28 PM, Boris Ostrovsky > <boris.ostrovsky@oracle.com> wrote: >> On 06/13/2016 01:43 PM, Meng Xu wrote: >>> Hi, >>> >>> I have a quick question about using the Linux spin_lock() in Xen >>> environment to protect some host-wide shared (memory) resource among >>> VMs. >>> >>> *** The question is as follows *** >>> Suppose I have two Linux VMs sharing the same spinlock_t lock (through >>> the sharing memory) on the same host. Suppose we have one process in >>> each VM. Each process uses the linux function spin_lock(&lock) [1] to >>> grab & release the lock. >>> Will these two processes in the two VMs have race on the shared lock? >> You can't do this: depending on which Linux version you use you will >> find that kernel uses ticketlocks or qlocks locks which keep track of >> who is holding the lock (obviously this information is internal to VM). > Yes, we are using the ticketlocks. We expose this information to other > VMs as well. That's why I'm guessing some data can be corrupted due to > races among VMs that share the same ticketlock information. I don't think it's a race that is the problem. It's that when holder releases a lock it wants to kick the waiter. It can't kick CPU in another guest and most likely kicks CPU from its own guest (because that's where it assumes the waiter is). -boris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Question about sharing spinlock_t among VMs in Xen 2016-06-13 21:17 ` Boris Ostrovsky @ 2016-06-14 1:50 ` Meng Xu 0 siblings, 0 replies; 9+ messages in thread From: Meng Xu @ 2016-06-14 1:50 UTC (permalink / raw) To: Boris Ostrovsky; +Cc: xen-devel On Mon, Jun 13, 2016 at 5:17 PM, Boris Ostrovsky <boris.ostrovsky@oracle.com> wrote: > On 06/13/2016 04:46 PM, Meng Xu wrote: >> On Mon, Jun 13, 2016 at 2:28 PM, Boris Ostrovsky >> <boris.ostrovsky@oracle.com> wrote: >>> On 06/13/2016 01:43 PM, Meng Xu wrote: >>>> Hi, >>>> >>>> I have a quick question about using the Linux spin_lock() in Xen >>>> environment to protect some host-wide shared (memory) resource among >>>> VMs. >>>> >>>> *** The question is as follows *** >>>> Suppose I have two Linux VMs sharing the same spinlock_t lock (through >>>> the sharing memory) on the same host. Suppose we have one process in >>>> each VM. Each process uses the linux function spin_lock(&lock) [1] to >>>> grab & release the lock. >>>> Will these two processes in the two VMs have race on the shared lock? >>> You can't do this: depending on which Linux version you use you will >>> find that kernel uses ticketlocks or qlocks locks which keep track of >>> who is holding the lock (obviously this information is internal to VM). >> Yes, we are using the ticketlocks. We expose this information to other >> VMs as well. That's why I'm guessing some data can be corrupted due to >> races among VMs that share the same ticketlock information. > > I don't think it's a race that is the problem. It's that when holder > releases a lock it wants to kick the waiter. It can't kick CPU in > another guest and most likely kicks CPU from its own guest (because > that's where it assumes the waiter is). I see. Thank you very much, Boris, for the explanation! I really appreciate it. :-) Best Regards, Meng ----------- Meng Xu PhD Student in Computer and Information Science University of Pennsylvania http://www.cis.upenn.edu/~mengxu/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Question about sharing spinlock_t among VMs in Xen 2016-06-13 17:43 Question about sharing spinlock_t among VMs in Xen Meng Xu 2016-06-13 18:28 ` Boris Ostrovsky @ 2016-06-13 22:54 ` Andrew Cooper 2016-06-14 2:13 ` Meng Xu 1 sibling, 1 reply; 9+ messages in thread From: Andrew Cooper @ 2016-06-13 22:54 UTC (permalink / raw) To: Meng Xu, xen-devel On 13/06/2016 18:43, Meng Xu wrote: > Hi, > > I have a quick question about using the Linux spin_lock() in Xen > environment to protect some host-wide shared (memory) resource among > VMs. > > *** The question is as follows *** > Suppose I have two Linux VMs sharing the same spinlock_t lock (through > the sharing memory) on the same host. Suppose we have one process in > each VM. Each process uses the linux function spin_lock(&lock) [1] to > grab & release the lock. > Will these two processes in the two VMs have race on the shared lock? "Race" is debatable. (After all, the point of a lock is to have serialise multiple accessors). But yes, this will be the same lock. The underlying cache coherency fabric will perform atomic locked operations on the same physical piece of RAM. The important question is whether the two difference VMs have an identical idea of what a spinlock_t is. If not, this will definitely fail. > My speculation is that it should have the race on the shard lock when > the spin_lock() function in *two VMs* operate on the same lock. > > We did some quick experiment on this and we found one VM sometimes see > the soft lockup on the lock. But we want to make sure our > understanding is correct. > > We are exploring if we can use the spin_lock to protect the shared > resources among VMs, instead of using the PV drivers. If the > spin_lock() in linux can provide the host-wide atomicity (which will > surprise me, though), that will be great. Otherwise, we probably have > to expose the spin_lock in Xen to the Linux? What are you attempting to protect like this? Anything which a guest can spin on like this is a recipe for disaster, as you observe, as the guest which holds the lock will get scheduled out in favour of the guest attempting to take the lock. Alternatively, two different guests with a different idea of how to manage the memory backing a spinlock_t. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Question about sharing spinlock_t among VMs in Xen 2016-06-13 22:54 ` Andrew Cooper @ 2016-06-14 2:13 ` Meng Xu 2016-06-14 16:01 ` Andrew Cooper 0 siblings, 1 reply; 9+ messages in thread From: Meng Xu @ 2016-06-14 2:13 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Mon, Jun 13, 2016 at 6:54 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote: > On 13/06/2016 18:43, Meng Xu wrote: >> Hi, >> >> I have a quick question about using the Linux spin_lock() in Xen >> environment to protect some host-wide shared (memory) resource among >> VMs. >> >> *** The question is as follows *** >> Suppose I have two Linux VMs sharing the same spinlock_t lock (through >> the sharing memory) on the same host. Suppose we have one process in >> each VM. Each process uses the linux function spin_lock(&lock) [1] to >> grab & release the lock. >> Will these two processes in the two VMs have race on the shared lock? > > "Race" is debatable. (After all, the point of a lock is to have > serialise multiple accessors). But yes, this will be the same lock. > > The underlying cache coherency fabric will perform atomic locked > operations on the same physical piece of RAM. The experiment we did is on a computer that is not NUMA. So it should not be caused by the sync. issue in hardware. > > The important question is whether the two difference VMs have an > identical idea of what a spinlock_t is. If not, this will definitely fail. I see the key point here now. However, I'm not that sure about if the two VMs have an *identical idea* of what a spinlock_t is. In other words, how to tell "if two VMs have an identical idea of what a spinlock_t is"? The current situation is as follows: Both VMs are using the same memory area for the spinlock_t variable. The spin_lock() in both VMs are operating on the same spinlock_t variable. So IMHO, the spinlock_t should be identical to these two VMs? Please correct me if I'm wrong. (I guess my understanding of the "identical idea of spinlock_t" may probably be incorrect. :-( ) > >> My speculation is that it should have the race on the shard lock when >> the spin_lock() function in *two VMs* operate on the same lock. >> >> We did some quick experiment on this and we found one VM sometimes see >> the soft lockup on the lock. But we want to make sure our >> understanding is correct. >> >> We are exploring if we can use the spin_lock to protect the shared >> resources among VMs, instead of using the PV drivers. If the >> spin_lock() in linux can provide the host-wide atomicity (which will >> surprise me, though), that will be great. Otherwise, we probably have >> to expose the spin_lock in Xen to the Linux? > > What are you attempting to protect like this? For example, if two VMs are sharing a chunk of memory with both read and write permissions, a VM has to grab the lock before it can operate on the shared memory. If we want a VM directly operate on the shared resource, instead of using the PV device model, we may need to use spinlock to protect the access to the shared resource. That's why we are looking at the spinlock. > > Anything which a guest can spin on like this is a recipe for disaster, > as you observe, as the guest which holds the lock will get scheduled out > in favour of the guest attempting to take the lock. It is true in general. The reason why we choose to let it spin is because some people in academia propose the protocols to access the shared resource through spinlock. In order to apply their theory, we may need to follow the system model they assumed. The theory did consider the situation when a guest/VCPU that is spinning on a lock is schedule out. The theory has to consider the extra delay caused by this situation. [OK. This is the reason why we did like this. But we are also thinking if we can do better in terms of the overall system performance.] BTW, I agree with you that letting guest spin like this could be a problem for the overall system performance. > Alternatively, two > different guests with a different idea of how to manage the memory > backing a spinlock_t. Just to confirm: Did you mean that different guests will use different policies to handle the same spinlock_t? This may mean that we need to have some special locking protocol, instead of the ticket_lock to handle the spin_lock? For example, a very simple and probably naive idea is that we may let a guest not be scheduled out before it releases the lock. I just want to use this simple example to make sure I understood the "alternative" idea here. :-) > ~Andrew Thank you very much for your insights and suggestions, Andrew! They are very helpful! Best Regards, Meng ----------- Meng Xu PhD Student in Computer and Information Science University of Pennsylvania http://www.cis.upenn.edu/~mengxu/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Question about sharing spinlock_t among VMs in Xen 2016-06-14 2:13 ` Meng Xu @ 2016-06-14 16:01 ` Andrew Cooper 2016-06-15 15:28 ` Meng Xu 0 siblings, 1 reply; 9+ messages in thread From: Andrew Cooper @ 2016-06-14 16:01 UTC (permalink / raw) To: Meng Xu; +Cc: xen-devel On 14/06/16 03:13, Meng Xu wrote: > On Mon, Jun 13, 2016 at 6:54 PM, Andrew Cooper > <andrew.cooper3@citrix.com> wrote: >> On 13/06/2016 18:43, Meng Xu wrote: >>> Hi, >>> >>> I have a quick question about using the Linux spin_lock() in Xen >>> environment to protect some host-wide shared (memory) resource among >>> VMs. >>> >>> *** The question is as follows *** >>> Suppose I have two Linux VMs sharing the same spinlock_t lock (through >>> the sharing memory) on the same host. Suppose we have one process in >>> each VM. Each process uses the linux function spin_lock(&lock) [1] to >>> grab & release the lock. >>> Will these two processes in the two VMs have race on the shared lock? >> "Race" is debatable. (After all, the point of a lock is to have >> serialise multiple accessors). But yes, this will be the same lock. >> >> The underlying cache coherency fabric will perform atomic locked >> operations on the same physical piece of RAM. > The experiment we did is on a computer that is not NUMA. Why do you think this makes any difference? Unless you have a uni-processor system from ages ago, there will be cache coherency being done in hardware. > So it should not be caused by the sync. issue in hardware. I do not understand what you are trying to say here. > >> The important question is whether the two difference VMs have an >> identical idea of what a spinlock_t is. If not, this will definitely fail. > I see the key point here now. However, I'm not that sure about if the > two VMs have an *identical idea* of what a spinlock_t is. If you are not sure, then the answer is almost certainly no. > In otherwords, how to tell "if two VMs have an identical idea of what a > spinlock_t is"? Is struct spinlock_t, and all functions which modify it, identical between all VMs trying to participate in the use of this shared memory spinlock? > > The current situation is as follows: > Both VMs are using the same memory area for the spinlock_t variable. > The spin_lock() in both VMs are operating on the same spinlock_t > variable. So IMHO, the spinlock_t should be identical to these two > VMs? > Please correct me if I'm wrong. (I guess my understanding of the > "identical idea of spinlock_t" may probably be incorrect. :-( ) > >>> My speculation is that it should have the race on the shard lock when >>> the spin_lock() function in *two VMs* operate on the same lock. >>> >>> We did some quick experiment on this and we found one VM sometimes see >>> the soft lockup on the lock. But we want to make sure our >>> understanding is correct. >>> >>> We are exploring if we can use the spin_lock to protect the shared >>> resources among VMs, instead of using the PV drivers. If the >>> spin_lock() in linux can provide the host-wide atomicity (which will >>> surprise me, though), that will be great. Otherwise, we probably have >>> to expose the spin_lock in Xen to the Linux? >> What are you attempting to protect like this? > For example, if two VMs are sharing a chunk of memory with both read > and write permissions, a VM has to grab the lock before it can operate > on the shared memory. > If we want a VM directly operate on the shared resource, instead of > using the PV device model, we may need to use spinlock to protect the > access to the shared resource. That's why we are looking at the > spinlock. > >> Anything which a guest can spin on like this is a recipe for disaster, >> as you observe, as the guest which holds the lock will get scheduled out >> in favour of the guest attempting to take the lock. > It is true in general. The reason why we choose to let it spin is > because some people in academia propose the protocols to access the > shared resource through spinlock. In order to apply their theory, we > may need to follow the system model they assumed. The theory did > consider the situation when a guest/VCPU that is spinning on a lock is > schedule out. The theory has to consider the extra delay caused by > this situation. [OK. This is the reason why we did like this. But we > are also thinking if we can do better in terms of the overall system > performance.] > > BTW, I agree with you that letting guest spin like this could be a > problem for the overall system performance. > >> Alternatively, two >> different guests with a different idea of how to manage the memory >> backing a spinlock_t. > Just to confirm: > Did you mean that different guests will use different policies to > handle the same spinlock_t? > This may mean that we need to have some special locking protocol, > instead of the ticket_lock to handle the spin_lock? > > For example, a very simple and probably naive idea is that we may let > a guest not be scheduled out before it releases the lock. I just want > to use this simple example to make sure I understood the "alternative" > idea here. :-) A guest is not in control of when it gets descheduled, and you cant yank a lock while the guest is in a critical region. If you want to proceed down this route, you will want to look at the PVspinlock impelementation where you block on an event channel while waiting for a lock held by a different vcpu, which frees up execution resource for the holder of the lock to complete. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Question about sharing spinlock_t among VMs in Xen 2016-06-14 16:01 ` Andrew Cooper @ 2016-06-15 15:28 ` Meng Xu 0 siblings, 0 replies; 9+ messages in thread From: Meng Xu @ 2016-06-15 15:28 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Tue, Jun 14, 2016 at 12:01 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote: > > On 14/06/16 03:13, Meng Xu wrote: > > On Mon, Jun 13, 2016 at 6:54 PM, Andrew Cooper > > <andrew.cooper3@citrix.com> wrote: > >> On 13/06/2016 18:43, Meng Xu wrote: > >>> Hi, > >>> > >>> I have a quick question about using the Linux spin_lock() in Xen > >>> environment to protect some host-wide shared (memory) resource among > >>> VMs. > >>> > >>> *** The question is as follows *** > >>> Suppose I have two Linux VMs sharing the same spinlock_t lock (through > >>> the sharing memory) on the same host. Suppose we have one process in > >>> each VM. Each process uses the linux function spin_lock(&lock) [1] to > >>> grab & release the lock. > >>> Will these two processes in the two VMs have race on the shared lock? > >> "Race" is debatable. (After all, the point of a lock is to have > >> serialise multiple accessors). But yes, this will be the same lock. > >> > >> The underlying cache coherency fabric will perform atomic locked > >> operations on the same physical piece of RAM. > > The experiment we did is on a computer that is not NUMA. > > Why do you think this makes any difference? Unless you have a > uni-processor system from ages ago, there will be cache coherency being > done in hardware. > > > So it should not be caused by the sync. issue in hardware. > > I do not understand what you are trying to say here. I was thinking if the x86 memory consistency model, i.e., TSO, will cause any issue? Should we use some memory barrier to sync. the memory operation? > > > > > >> The important question is whether the two difference VMs have an > >> identical idea of what a spinlock_t is. If not, this will definitely fail. > > I see the key point here now. However, I'm not that sure about if the > > two VMs have an *identical idea* of what a spinlock_t is. > > If you are not sure, then the answer is almost certainly no. Fair enough... > > > > In otherwords, how to tell "if two VMs have an identical idea of what a > > spinlock_t is"? > > Is struct spinlock_t, and all functions which modify it, identical > between all VMs trying to participate in the use of this shared memory > spinlock? Yes. The spinlock_t and all functions which modify it are identical between all VMs. Does this mean they have the identical idea of what a spinlock_t is? > > > > > > The current situation is as follows: > > Both VMs are using the same memory area for the spinlock_t variable. > > The spin_lock() in both VMs are operating on the same spinlock_t > > variable. So IMHO, the spinlock_t should be identical to these two > > VMs? > > Please correct me if I'm wrong. (I guess my understanding of the > > "identical idea of spinlock_t" may probably be incorrect. :-( ) > > > >>> My speculation is that it should have the race on the shard lock when > >>> the spin_lock() function in *two VMs* operate on the same lock. > >>> > >>> We did some quick experiment on this and we found one VM sometimes see > >>> the soft lockup on the lock. But we want to make sure our > >>> understanding is correct. > >>> > >>> We are exploring if we can use the spin_lock to protect the shared > >>> resources among VMs, instead of using the PV drivers. If the > >>> spin_lock() in linux can provide the host-wide atomicity (which will > >>> surprise me, though), that will be great. Otherwise, we probably have > >>> to expose the spin_lock in Xen to the Linux? > >> What are you attempting to protect like this? > > For example, if two VMs are sharing a chunk of memory with both read > > and write permissions, a VM has to grab the lock before it can operate > > on the shared memory. > > If we want a VM directly operate on the shared resource, instead of > > using the PV device model, we may need to use spinlock to protect the > > access to the shared resource. That's why we are looking at the > > spinlock. > > > >> Anything which a guest can spin on like this is a recipe for disaster, > >> as you observe, as the guest which holds the lock will get scheduled out > >> in favour of the guest attempting to take the lock. > > It is true in general. The reason why we choose to let it spin is > > because some people in academia propose the protocols to access the > > shared resource through spinlock. In order to apply their theory, we > > may need to follow the system model they assumed. The theory did > > consider the situation when a guest/VCPU that is spinning on a lock is > > schedule out. The theory has to consider the extra delay caused by > > this situation. [OK. This is the reason why we did like this. But we > > are also thinking if we can do better in terms of the overall system > > performance.] > > > > BTW, I agree with you that letting guest spin like this could be a > > problem for the overall system performance. > > > >> Alternatively, two > >> different guests with a different idea of how to manage the memory > >> backing a spinlock_t. > > Just to confirm: > > Did you mean that different guests will use different policies to > > handle the same spinlock_t? > > This may mean that we need to have some special locking protocol, > > instead of the ticket_lock to handle the spin_lock? > > > > For example, a very simple and probably naive idea is that we may let > > a guest not be scheduled out before it releases the lock. I just want > > to use this simple example to make sure I understood the "alternative" > > idea here. :-) > > A guest is not in control of when it gets descheduled, and you cant yank > a lock while the guest is in a critical region. Unless we don't commit the change until the end of the critical region. (But it will make this like a transaction. OK. Let's avoid this first.) > > If you want to proceed down this route, you will want to look at the > PVspinlock impelementation where you block on an event channel while > waiting for a lock held by a different vcpu, which frees up execution > resource for the holder of the lock to complete. I will have a look at the pvspinlock then. Thank you very much for your suggestions and advices, Andrew! :-) Best Regards, Meng ----------- Meng Xu PhD Student in Computer and Information Science University of Pennsylvania http://www.cis.upenn.edu/~mengxu/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-06-15 15:28 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-06-13 17:43 Question about sharing spinlock_t among VMs in Xen Meng Xu 2016-06-13 18:28 ` Boris Ostrovsky 2016-06-13 20:46 ` Meng Xu 2016-06-13 21:17 ` Boris Ostrovsky 2016-06-14 1:50 ` Meng Xu 2016-06-13 22:54 ` Andrew Cooper 2016-06-14 2:13 ` Meng Xu 2016-06-14 16:01 ` Andrew Cooper 2016-06-15 15:28 ` Meng Xu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).