kvm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jason Gunthorpe <jgg@ziepe.ca>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
	cohuck@redhat.com, peterx@redhat.com
Subject: Re: [PATCH 3/3] vfio-pci: Invalidate mmaps and block MMIO access on disabled memory
Date: Mon, 4 May 2020 15:44:36 -0300	[thread overview]
Message-ID: <20200504184436.GZ26002@ziepe.ca> (raw)
In-Reply-To: <20200504122643.52267e44@x1.home>

On Mon, May 04, 2020 at 12:26:43PM -0600, Alex Williamson wrote:
> On Fri, 1 May 2020 20:48:49 -0300
> Jason Gunthorpe <jgg@ziepe.ca> wrote:
> 
> > On Fri, May 01, 2020 at 03:39:30PM -0600, Alex Williamson wrote:
> > 
> > >  static int vfio_pci_add_vma(struct vfio_pci_device *vdev,
> > >  			    struct vm_area_struct *vma)
> > >  {
> > > @@ -1346,15 +1450,49 @@ static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf)
> > >  {
> > >  	struct vm_area_struct *vma = vmf->vma;
> > >  	struct vfio_pci_device *vdev = vma->vm_private_data;
> > > +	vm_fault_t ret = VM_FAULT_NOPAGE;
> > >  
> > > -	if (vfio_pci_add_vma(vdev, vma))
> > > -		return VM_FAULT_OOM;
> > > +	/*
> > > +	 * Zap callers hold memory_lock and acquire mmap_sem, we hold
> > > +	 * mmap_sem and need to acquire memory_lock to avoid races with
> > > +	 * memory bit settings.  Release mmap_sem, wait, and retry, or fail.
> > > +	 */
> > > +	if (unlikely(!down_read_trylock(&vdev->memory_lock))) {
> > > +		if (vmf->flags & FAULT_FLAG_ALLOW_RETRY) {
> > > +			if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
> > > +				return VM_FAULT_RETRY;
> > > +
> > > +			up_read(&vma->vm_mm->mmap_sem);
> > > +
> > > +			if (vmf->flags & FAULT_FLAG_KILLABLE) {
> > > +				if (!down_read_killable(&vdev->memory_lock))
> > > +					up_read(&vdev->memory_lock);
> > > +			} else {
> > > +				down_read(&vdev->memory_lock);
> > > +				up_read(&vdev->memory_lock);
> > > +			}
> > > +			return VM_FAULT_RETRY;
> > > +		}
> > > +		return VM_FAULT_SIGBUS;
> > > +	}  
> > 
> > So, why have the wait? It isn't reliable - if this gets faulted from a
> > call site that can't handle retry then it will SIGBUS anyhow?
> 
> Do such call sites exist?  My assumption was that half of the branch
> was unlikely to ever occur.

hmm_range_fault() for instance doesn't set ALLOW_RETRY, I assume there
are enough other case to care about, but am not so sure

> > The weird use of a rwsem as a completion suggest that perhaps using
> > wait_event might improve things:
> > 
> > disable:
> >   // Clean out the vma list with zap, then:
> > 
> >   down_read(mm->mmap_sem)
> 
> I assume this is simplifying the dance we do in zapping to first take
> vma_lock in order to walk vma_list, to find a vma from which we can
> acquire the mm, drop vma_lock, get mmap_sem, then re-get vma_lock
> below.  

No, that has to stay..

> Also accounting that vma_list might be empty and we might need
> to drop and re-acquire vma_lock to get to another mm, so we really
> probably want to set pause_faults at the start rather than at the end.

New vmas should not created/faulted while vma_lock is held, so the
order shouldn't matter..

> >   mutex_lock(vma_lock);
> >   list_for_each_entry_safe()
> >      // zap and remove all vmas
> > 
> >   pause_faults = true;
> >   mutex_write(vma_lock);
> > 
> > fault:
> >   // Already have down_read(mmap_sem)
> >   mutex_lock(vma_lock);
> >   while (pause_faults) {
> >      mutex_unlock(vma_lock)
> >      wait_event(..., !pause_faults)
> >      mutex_lock(vma_lock)
> >   }
> 
> Nit, we need to test the memory enable bit setting somewhere under this
> lock since it seems to be the only thing protecting it now.

I was thinking you'd keep the same locking for the memory enable bit,
the pause_faults is a shadow of that bit with locking connected to
vma_lock..

> >   list_add()
> >   remap_pfn()
> >   mutex_unlock(vma_lock)
> 
> The read and write file ops would need similar mechanisms.

Keep using the rwsem?

> > enable:
> >   pause_faults = false
> >   wake_event()
> 
> Hmm, vma_lock was dropped above and not re-acquired here.

I was thinking this would be under a continous rwlock

> I'm not sure if it was an oversight that pause_faults was not tested
> in the disable path, but this combination appears to lead to
> concurrent writers and serialized readers??

? pause_faults only exists to prevent the vm_ops fault callback from
progressing to a fault. I don't think any concurrancy is lost

> > The only requirement here is that while inside the write side of
> > memory_lock you cannot touch user pages (ie no copy_from_user/etc)
> 
> I'm lost at this statement, I can only figure the above works if we
> remove memory_lock.  Are you referring to a different lock?  Thanks,

No

This is just an approach to avoid the ABBA deadlock problem when using
a rwsem by using a looser form of lock combined witih the already
correctly nested vma_lock.

Stated another way, you can keep the existing memory_lock as is, if it
is structured like this:

disable:
 down_read(mmap_sem)
 mutex_lock(vma_lock)
 list_for_each_entry_safe()
      // zap and remove all vmas
 down_write(memory_lock)   // Now inside vma_lock!
 mutex_unlock(vma_lock)
 up_read(mmap_sem

 [ do the existing stuff under memory_lock ]


fault:
  mutex_lock(vma_lock)
  down_write(memory_lock)
  remap_pfn
  up_write(memory_lock)
  mutex_unlock(vma_lock)

enable:
  up_write(memory_lock)

Ie the key is to organize things to move the down_write(memory_lock)
to be under the mmap_sem/vma_lock

Jason

  reply	other threads:[~2020-05-04 18:44 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-01 21:38 [PATCH 0/3] vfio-pci: Block user access to disabled device MMIO Alex Williamson
2020-05-01 21:39 ` [PATCH 1/3] vfio/type1: Support faulting PFNMAP vmas Alex Williamson
2020-05-01 23:50   ` Jason Gunthorpe
2020-05-04 14:06     ` Alex Williamson
2020-05-04 15:02       ` Jason Gunthorpe
2020-05-01 21:39 ` [PATCH 2/3] vfio-pci: Fault mmaps to enable vma tracking Alex Williamson
2020-05-01 23:25   ` Jason Gunthorpe
2020-05-04 14:20     ` Alex Williamson
2020-05-04 15:05       ` Jason Gunthorpe
2020-05-04 15:53         ` Alex Williamson
2020-05-01 21:39 ` [PATCH 3/3] vfio-pci: Invalidate mmaps and block MMIO access on disabled memory Alex Williamson
2020-05-01 23:48   ` Jason Gunthorpe
2020-05-04 18:26     ` Alex Williamson
2020-05-04 18:44       ` Jason Gunthorpe [this message]
2020-05-04 19:35         ` Alex Williamson
2020-05-04 20:01           ` Jason Gunthorpe
2020-05-05 17:12             ` Alex Williamson
2020-05-05 18:36               ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200504184436.GZ26002@ziepe.ca \
    --to=jgg@ziepe.ca \
    --cc=alex.williamson@redhat.com \
    --cc=cohuck@redhat.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peterx@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).