RE: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that behind VMD controller

[Xiaochun XC17 Li <lixc17@lenovo.com>]

On Wed, Nov 16, 2022 at 12:10 PM +0800, "Bjorn Helgaas" <helgaas@kernel.org> wrote:
> On Wed, Nov 16, 2022 at 03:34:34AM +0000, Xiaochun XC17 Li wrote:
> > On Sat, Nov 11, 2022 at 07:54:25, "Bjorn Helgaas" <helgaas@kernel.org>
> wrote:
> > > On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> > > > From: Xiaochun Lee <lixc17@lenovo.com>
> > > >
> > > > When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> > > > recognized by VMD driver and there are many failed messages of BAR
> > > > 13 when scan the bridges and assign IO resource behind it as
> > > > listed below, the bridge wants to get 0x6000 as its IO resource,
> > > > but there is no IO resources on the host bridge.
> > > >
> > > > VMD host bridge resources:
> > > > vmd 0000:64:00.5: PCI host bridge to bus 10000:80 pci_bus 10000:80:
> > > > root bus resource [bus 80-9f] pci_bus 10000:80: root bus resource
> > > > [mem 0xe0000000-0xe1ffffff] pci_bus 10000:80: root bus resource
> > > > [mem 0x24ffff02010-0x24fffffffff 64bit]
> > > >
> > > > Failed messages of BAR#13:
> > > > pci 10000:80:02.0: BAR 13: no space for [io  size 0x1000] pci
> > > > 10000:80:02.0: BAR 13: failed to assign [io  size 0x1000] pci
> > > > 10000:80:03.0: BAR 13: no space for [io  size 0x1000] pci
> > > > 10000:80:03.0: BAR 13: failed to assign [io  size 0x1000]
> > > >
> > > > VMD-enabled root ports use
> > > > Enhanced Configuration Access Mechanism (ECAM) access PCI
> Express
> > > > configuration space, and offer VMD_CFGBAR as base of PCI Express
> > > > configuration space for the bridges behind it. The configuration
> > > > space includes IO resources, but these IO resources are not
> > > > actually used on X86, especially the NVMes as device connected on
> > > > this hot plug bridges, and it can result in BAR#13 assign IO
> > > > resource failed. So we clear IO resources by setting an IO base
> > > > value greater than limit to these bridges. Hence, we can leverage
> > > > kernel parameter "pci=hpiosize=0KB" to avoid this failed messages
> show out.
> > > >
> > > > Signed-off-by: Xiaochun Lee <lixc17@lenovo.com>
> > >
> > > Some of the discussion here got lost because of email issues.  Lore
> > > has
> > > some:
> > >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> > > kernel.org%2Fall%2F1663075485-20591-1-git-send-email-
> > >
> lixiaochun.2888%40163.com%2FT%2F%23u&amp;data=05%7C01%7Clixc17
> > > %40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28
> bdf
> > >
> 8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknown
> > > %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
> ha
> > >
> WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JzWYkIsaEfINofiqo
> > > XyjEh43VjXO3HZw2JLSsmhpUiQ%3D&amp;reserved=0,
> > > and patchwork has a v2 with a little more discussion:
> > >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpa
> > > tc
> > > hwork.kernel.org%2Fproject%2Flinux-pci%2Fpatch%2F1664288166-
> 7432-1-
> > > git-send-email-
> > >
> lixiaochun.2888%40163.com%2F&amp;data=05%7C01%7Clixc17%40lenovo.
> > >
> com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf8410caa934
> > >
> df372b16203%7C0%7C0%7C638038076734438158%7CUnknown%7CTWFpb
> > >
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> > >
> 6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y%2BtIsepTpyLeoHW6CrgkZz2
> > > tiMjY0TONfK7zNCKXQ90%3D&amp;reserved=0
> > >
> > > But the v2 patch doesn't seem to have made it to the mailing lists
> > > or to lore
> > >
> (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
> > > or
> > >
> e.kernel.org%2Fall%2F%3Fq%3Df%253Alixc17&amp;data=05%7C01%7Clixc1
> > >
> 7%40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bd
> > >
> f8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknow
> > >
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> > >
> aWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=zxGz1hZOD2tvQP
> > > EsbxQTzjHwQvXvqeO%2FUd6I9S%2Fj314%3D&amp;reserved=0) and I
> don't
> > > apply things until they appear on the mailing list.
> > >
> > > I *would* like to get rid of those "no space" and "failed to assign"
> > > messages.  This is an issue for platforms other than VMD, too.  Just
> > > an FYI that you need to follow up on this if we want make progress.
> >
> > Thanks for your comments, so do you mean we'd better come up with a
> > solution to avoid messages like "no space" and "failed to assign"
> > using a common way, for both VMD and other platforms?
> 
> I tried to say two separate things:
> 
> 1) It doesn't seem like a VMD-specific thing, so it would be ideal if the
> solution were generic instead of being VMD-specific.

OK, I'll try to find out a generic approach.

> 
> 2) Some of your previous patches didn't make it to the mailing list, so we
> couldn't really do anything with them.
> 
> Bjorn

Actually, the V2 patch didn't following the mailing of V1 patch,
I sent it out separately like the first time for V1 patch, I don't know if there
is a mistake in the way I sent the V2 patch. I was wondering if I was going to
send out V2 patches based on the mailing list of V1 patch, if it is yes, I must
study how to summit the V2 patch, so would you please correct me?

Xiaochun