From: Guenter Roeck <linux@roeck-us.net>
To: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Bjorn Helgaas <helgaas@kernel.org>,
Hari Vyas <hari.vyas@broadcom.com>,
bhelgaas@google.com, linux-pci@vger.kernel.org,
ray.jui@broadcom.com,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH] pci: Proof of concept at fixing pci_enable_device/bridge races
Date: Sun, 19 Aug 2018 18:31:01 -0700 [thread overview]
Message-ID: <20180820013101.GA2556@roeck-us.net> (raw)
In-Reply-To: <947d07eb6bc64eefe5cfd9a08420a33f855cbe2c.camel@kernel.crashing.org>
On Thu, Aug 16, 2018 at 09:38:41AM +1000, Benjamin Herrenschmidt wrote:
> On Wed, 2018-08-15 at 15:40 -0700, Guenter Roeck wrote:
> > On Thu, Aug 16, 2018 at 07:50:13AM +1000, Benjamin Herrenschmidt wrote:
> > > (Resent with lkml on copy)
> > >
> > > [Note: This isn't meant to be merged, it need splitting at the very
> > > least, see below]
> > >
> > > This is something I cooked up quickly today to test if that would fix
> > > my problems with large number of switch and NVME devices on POWER.
> > >
> >
> > Is that a problem that can be reproduced with a qemu setup ?
>
> With difficulty... mt-tcg might help, but you need a rather large
> systems to reproduce it.
>
> My repro-case is a 2 socket POWER9 system (about 40 cores off the top
> of my mind, so 160 threads) with 72 NVME devices underneath a tree of
> switches (I don't have the system at hand today to check how many).
>
> It's possible to observe it I suppose on a smaller system (in theory a
> single bridge with 2 devices is enough) but in practice the timing is
> extremely hard to hit.
>
> You need a combination of:
>
> - The bridges come up disabled (which is the case when Linux does the
> resource assignment, such as on POWER but not on x86 unless it's
> hotplug)
>
> - The nvme devices try to enable them simultaneously
>
> Also the resulting error is a UR, I don't know how well qemu models
> that.
>
Not well enough, apparently. I tried for a while, registering as many
nvme drives as the system would take, but I was not able to reproduce
the problem with qemu. It was worth a try, though.
Guenter
next prev parent reply other threads:[~2018-08-20 1:31 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-07-03 9:05 [PATCH v3] PCI: Data corruption happening due to race condition Hari Vyas
2018-07-03 9:05 ` Hari Vyas
2018-07-03 9:13 ` Lukas Wunner
2018-07-18 23:29 ` Bjorn Helgaas
2018-07-19 4:18 ` Benjamin Herrenschmidt
2018-07-19 14:04 ` Hari Vyas
2018-07-19 18:55 ` Lukas Wunner
2018-07-20 4:27 ` Benjamin Herrenschmidt
2018-07-27 22:25 ` Bjorn Helgaas
2018-07-28 0:45 ` Benjamin Herrenschmidt
2018-07-31 11:21 ` Michael Ellerman
2018-07-19 17:41 ` Bjorn Helgaas
2018-07-20 9:16 ` Hari Vyas
2018-07-20 12:20 ` Bjorn Helgaas
2018-07-31 16:37 ` Bjorn Helgaas
2018-08-15 3:35 ` PCIe enable device races (Was: [PATCH v3] PCI: Data corruption happening due to race condition) Benjamin Herrenschmidt
2018-08-15 4:16 ` Benjamin Herrenschmidt
2018-08-15 4:44 ` Benjamin Herrenschmidt
2018-08-15 5:21 ` [RFC PATCH] pci: Proof of concept at fixing pci_enable_device/bridge races Benjamin Herrenschmidt
2018-08-15 19:09 ` PCIe enable device races (Was: [PATCH v3] PCI: Data corruption happening due to race condition) Bjorn Helgaas
2018-08-15 21:50 ` [RFC PATCH] pci: Proof of concept at fixing pci_enable_device/bridge races Benjamin Herrenschmidt
2018-08-15 22:40 ` Guenter Roeck
2018-08-15 23:38 ` Benjamin Herrenschmidt
2018-08-20 1:31 ` Guenter Roeck [this message]
2018-08-17 3:07 ` Bjorn Helgaas
2018-08-17 3:42 ` Benjamin Herrenschmidt
2018-08-15 18:50 ` PCIe enable device races (Was: [PATCH v3] PCI: Data corruption happening due to race condition) Bjorn Helgaas
2018-08-15 21:52 ` Benjamin Herrenschmidt
2018-08-15 23:23 ` Benjamin Herrenschmidt
2018-08-16 7:58 ` Konstantin Khlebnikov
2018-08-16 8:02 ` Benjamin Herrenschmidt
2018-08-16 9:22 ` Hari Vyas
2018-08-16 10:10 ` Benjamin Herrenschmidt
2018-08-16 10:11 ` Benjamin Herrenschmidt
2018-08-16 10:26 ` Lukas Wunner
2018-08-16 10:47 ` Hari Vyas
2018-08-16 23:20 ` Benjamin Herrenschmidt
2018-08-16 23:17 ` Benjamin Herrenschmidt
2018-08-17 0:43 ` Benjamin Herrenschmidt
2018-08-16 19:43 ` Jens Axboe
2018-08-16 21:37 ` Benjamin Herrenschmidt
2018-08-16 21:56 ` Jens Axboe
2018-08-16 23:09 ` Benjamin Herrenschmidt
2018-08-17 0:14 ` Jens Axboe
2018-08-16 12:28 ` Lukas Wunner
2018-08-16 23:25 ` Benjamin Herrenschmidt
2018-08-17 1:12 ` Benjamin Herrenschmidt
2018-08-17 16:39 ` Lukas Wunner
2018-08-18 3:37 ` Benjamin Herrenschmidt
2018-08-18 9:22 ` Lukas Wunner
2018-08-18 13:11 ` Benjamin Herrenschmidt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180820013101.GA2556@roeck-us.net \
--to=linux@roeck-us.net \
--cc=benh@kernel.crashing.org \
--cc=bhelgaas@google.com \
--cc=hari.vyas@broadcom.com \
--cc=helgaas@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=ray.jui@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).