Linux-PCI Archive on lore.kernel.org
 help / color / Atom feed
From: Alex Williamson <alex.williamson@redhat.com>
To: Andreas Hartmann <andihartmann@freenet.de>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
	linux-pci <linux-pci@vger.kernel.org>
Subject: Re: Hard and silent lock up since linux 3.14 with PCIe pass through (vfio)
Date: Wed, 29 Oct 2014 11:44:37 -0600
Message-ID: <1414604677.27420.263.camel@ul30vt.home> (raw)
In-Reply-To: <54511A16.30602@maya.org>

On Wed, 2014-10-29 at 17:47 +0100, Andreas Hartmann wrote:
> Alex Williamson wrote:
> > On Sat, 2014-10-25 at 08:03 +0200, Andreas Hartmann wrote:
> >>
> >> Out of interest:
> >> Bjorn's patch disables vc save/restore support - and the machine works
> >> fine again. Why is it needed at all if it seems to work perfectly w/o
> >> it? What's the additional benefit? Or in other words: What am I missing
> >> until today :-) ? What would be better? What could I do more?
> > 
> > 
> > You're right, in the configuration you have the endpoint device has a
> > Virtual Channel capability but the upstream root port does not.  The
> > spec is not at all clear about defining the endpoints for enabling
> > Virtual Channel in each type of configuration, but I think that if we
> > have an upstream port that does not support Virtual Channel, we can skip
> > the save/restore.  Please test the patch below.
> > 
> > I'm also still completely confused about whether this is a VC
> > save/restore issue or a bus reset issue.  You originally bisected this
> > back to the VC save/restore patch, but you also found that a manual,
> > setpci-based bus reset triggered a system hang.
> 
> With your additional patch posted here:
> http://article.gmane.org/gmane.linux.kernel.pci/36162

Right, a reset via sysfs also triggered it with that patch, but the
reset via setpci is independent of any VC save/restore and still hung
your box.

> 
> >  I believe that
> > re-ordering the kernel reset mechanisms also triggered this.  Since
> > recent versions of QEMU are going to favor a bus reset over PM reset, I
> > don't have a lot of confidence that we're actually solving the problem
> > for you.  Please make sure to test with a recent QEMU to be sure we'll
> > do a bus reset.
> 
> I'm running qemu 2.1.0 (newest is 2.1.2 - but this shouldn't be a
> problem) and tested w/ linux 3.17.

Yep, just want to make sure it's QEMU new enough to do a bus reset and
kernel with matching support.

> > diff --git a/drivers/pci/vc.c b/drivers/pci/vc.c
> > index 7e1304d..6d13d34 100644
> > --- a/drivers/pci/vc.c
> > +++ b/drivers/pci/vc.c
> > @@ -339,6 +339,25 @@ static int pci_vc_do_save_buffer(struct pci_dev *dev, int pos,
> >  	return buf ? 0 : len;
> >  }
> >  
> > +/**
> > + * pci_vc_needs_save - Determine whether a VC capability needs to be saved
> > + * @dev: device
> > + * @id: VC capability ID (VC/VC9/MFVC)
> > + *
> > + * In configurations where we have a VC or MFVC capability, but the upstream
> > + * device does not, we assume that VC save (and therefore restore) is not
> > + * necessary.  The intention is to only do VC save/restore in configuration
> > + * where it's necessary and hopefully avoid reset issues.
> > + */
> > +static bool pci_vc_needs_save(struct pci_dev *dev, u16 id)
> > +{
> > +	if (id == PCI_EXT_CAP_ID_VC9 || pci_is_root_bus(dev->bus) ||
> > +	    pci_find_ext_capability(dev->bus->self, PCI_EXT_CAP_ID_VC))
> > +		return true;
> > +
> > +	return false;
> > +}
> > +
> >  static struct {
> >  	u16 id;
> >  	const char *name;
> > @@ -362,7 +381,7 @@ int pci_save_vc_state(struct pci_dev *dev)
> >  		struct pci_cap_saved_state *save_state;
> >  
> >  		pos = pci_find_ext_capability(dev, vc_caps[i].id);
> > -		if (!pos)
> > +		if (!posi || !pci_vc_needs_save(dev, vc_caps[i].id))
>                         ^
> This should be most probably !pos (and not !posi - because !posi does
> through a compile error).

Oops, sorry.

> >  			continue;
> >  
> >  		save_state = pci_find_saved_ext_cap(dev, vc_caps[i].id);
> > @@ -422,7 +441,7 @@ void pci_allocate_vc_save_buffers(struct pci_dev *dev)
> >  	for (i = 0; i < ARRAY_SIZE(vc_caps); i++) {
> >  		int len, pos = pci_find_ext_capability(dev, vc_caps[i].id);
> >  
> > -		if (!pos)
> > +		if (!pos || !pci_vc_needs_save(dev, vc_caps[i].id))
> >  			continue;
> >  
> >  		len = pci_vc_do_save_buffer(dev, pos, NULL, false);
> 
> W/ the above patch, the machine hangs again (w/ qemu and setpci), but w/
> Bjorn's patch (and nothing more applied) which disables vc save/restore,
> the machine just works fine ... . I especially retested this case to be
> really sure. I'm so sorry. But that's how it behaves here :-(

Hmm, the intention was that this should effectively do the same thing as
Bjorn's patch.  The Atheros device (03:00.0) reports a VC capability but
the root port above it (00:05.0) does not.  The test in
pci_vc_needs_save() should therefore be false for all tests within the
if() block and the function should return false, causing us to neither
allocate a save buffer or perform a save.  The restore will
automatically be skipped since there's no save buffer.

This is what I was afraid of, on one hand you've bisected and proved via
patching that the problem is exclusively due to VC save/restore, but we
also have testing that indicates that we can't do a bus reset at all on
this port.  So I re-iterate my confusion that we don't seem to have a
good idea which is the problem.  Do you have any other devices that you
can install in the slot for testing?  Maybe if we knew that bus reset is
only a problem with the Atheros card then we could blacklist it.
Thanks,

Alex



  reply index

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-09-23 19:03 Andreas Hartmann
2014-09-23 20:07 ` Alex Williamson
2014-09-24 14:54   ` Andreas Hartmann
2014-09-24 17:16     ` Andreas Hartmann
2014-10-10  9:39   ` Andreas Hartmann
2014-10-10 14:37     ` Bjorn Helgaas
2014-10-10 14:49       ` Andreas Hartmann
2014-10-10 15:55         ` Bjorn Helgaas
2014-10-10 16:09           ` Andreas Hartmann
2014-10-10 16:41             ` Bjorn Helgaas
2014-10-10 22:32               ` Andreas Hartmann
2014-10-10 22:54                 ` Bjorn Helgaas
2014-10-11  6:20                   ` Andreas Hartmann
2014-10-15  8:04                     ` Alex Williamson
2014-10-17  1:04                       ` Andreas Hartmann
2014-10-21 21:06                         ` Alex Williamson
2014-10-21 21:32                           ` Alex Williamson
2014-10-22 16:22                             ` Andreas Hartmann
2014-10-22 20:36                               ` Alex Williamson
2014-10-23 16:00                                 ` Andreas Hartmann
2014-10-23 16:33                                   ` Alex Williamson
2014-10-23 17:12                                     ` Andreas Hartmann
2014-10-23 17:33                                     ` Andreas Hartmann
2014-10-23 19:37                                       ` Alex Williamson
2014-10-24 14:21                                         ` Andreas Hartmann
2014-10-25  6:03                                         ` Andreas Hartmann
2014-10-28 21:51                                           ` Alex Williamson
2014-10-29 16:47                                             ` Andreas Hartmann
2014-10-29 17:44                                               ` Alex Williamson [this message]
2014-10-29 17:57                                                 ` Andreas Hartmann
2014-10-29 18:16                                                   ` Alex Williamson
2014-10-29 19:43                                                     ` Andreas Hartmann
2014-10-29 20:50                                                       ` Alex Williamson
2014-10-29 21:35                                                         ` Andreas Hartmann
2014-10-30 16:35                                                         ` Andreas Hartmann
2014-10-30 16:58                                                           ` Alex Williamson
2014-10-30 19:09                                                             ` Andreas Hartmann
2014-10-30 19:45                                                               ` Alex Williamson
2014-10-30 20:21                                                                 ` Andreas Hartmann
2014-10-22 15:34                           ` Andreas Hartmann
2014-10-22 16:02                             ` Alex Williamson
2014-10-22 16:20                               ` Andreas Hartmann

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1414604677.27420.263.camel@ul30vt.home \
    --to=alex.williamson@redhat.com \
    --cc=andihartmann@freenet.de \
    --cc=bhelgaas@google.com \
    --cc=linux-pci@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-PCI Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-pci/0 linux-pci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-pci linux-pci/ https://lore.kernel.org/linux-pci \
		linux-pci@vger.kernel.org
	public-inbox-index linux-pci

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-pci


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git