* Re: [PATCH] PCI: Warn about host bridge device when its numa node is NO_NODE
2019-10-23 17:10 ` Bjorn Helgaas
@ 2019-10-24 9:20 ` Michal Hocko
2019-10-24 17:40 ` Bjorn Helgaas
2019-10-24 9:39 ` Yunsheng Lin
2019-10-24 10:16 ` Robin Murphy
2 siblings, 1 reply; 15+ messages in thread
From: Michal Hocko @ 2019-10-24 9:20 UTC (permalink / raw)
To: Bjorn Helgaas, peterz
Cc: Yunsheng Lin, linux-pci, linux-kernel, robin.murphy, geert,
gregkh, paul.burton
On Wed 23-10-19 12:10:39, Bjorn Helgaas wrote:
> On Wed, Oct 23, 2019 at 04:22:43PM +0800, Yunsheng Lin wrote:
> > On 2019/10/23 5:04, Bjorn Helgaas wrote:
> > > On Sat, Oct 19, 2019 at 02:45:43PM +0800, Yunsheng Lin wrote:
>
> > > I think the underlying problem you're addressing is that:
> > >
> > > - NUMA_NO_NODE == -1,
> > > - dev_to_node(dev) may return NUMA_NO_NODE,
> > > - kmalloc(dev) relies on cpumask_of_node(dev_to_node(dev)), and
> > > - cpumask_of_node(NUMA_NO_NODE) makes an invalid array reference
> > >
> > > For example, on arm64, mips loongson, s390, and x86,
> > > cpumask_of_node(node) returns "node_to_cpumask_map[node]", and -1 is
> > > an invalid array index.
> >
> > The invalid array index of -1 is the underlying problem here when
> > cpumask_of_node(dev_to_node(dev)) is called and cpumask_of_node()
> > is not NUMA_NO_NODE aware yet.
> >
> > In the "numa: make node_to_cpumask_map() NUMA_NO_NODE aware" thread
> > disscusion, it is requested that it is better to warn about the pcie
> > device without a node assigned by the firmware before making the
> > cpumask_of_node() NUMA_NO_NODE aware, so that the system with pci
> > devices of "NUMA_NO_NODE" node can be fixed by their vendor.
> >
> > See: https://lore.kernel.org/lkml/20191011111539.GX2311@hirez.programming.kicks-ass.net/
>
> Right. We should warn if the NUMA node number would help us but DT or
> the firmware didn't give us one.
>
> But we can do that independently of any cpumask_of_node() changes.
> There's no need to do one patch before the other. Even if you make
> cpumask_of_node() tolerate NUMA_NO_NODE, we'll still get the warning
> because we're not actually changing any node assignments.
Agreed. And this has been proposed previously I believe but my
understanding was that Petr was against that.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] PCI: Warn about host bridge device when its numa node is NO_NODE
2019-10-24 9:20 ` Michal Hocko
@ 2019-10-24 17:40 ` Bjorn Helgaas
2019-10-25 8:16 ` Michal Hocko
0 siblings, 1 reply; 15+ messages in thread
From: Bjorn Helgaas @ 2019-10-24 17:40 UTC (permalink / raw)
To: Michal Hocko
Cc: peterz, Yunsheng Lin, linux-pci, linux-kernel, robin.murphy,
geert, gregkh, paul.burton
On Thu, Oct 24, 2019 at 11:20:13AM +0200, Michal Hocko wrote:
> On Wed 23-10-19 12:10:39, Bjorn Helgaas wrote:
> > On Wed, Oct 23, 2019 at 04:22:43PM +0800, Yunsheng Lin wrote:
> > > On 2019/10/23 5:04, Bjorn Helgaas wrote:
> > > > On Sat, Oct 19, 2019 at 02:45:43PM +0800, Yunsheng Lin wrote:
> >
> > > > I think the underlying problem you're addressing is that:
> > > >
> > > > - NUMA_NO_NODE == -1,
> > > > - dev_to_node(dev) may return NUMA_NO_NODE,
> > > > - kmalloc(dev) relies on cpumask_of_node(dev_to_node(dev)), and
> > > > - cpumask_of_node(NUMA_NO_NODE) makes an invalid array reference
> > > >
> > > > For example, on arm64, mips loongson, s390, and x86,
> > > > cpumask_of_node(node) returns "node_to_cpumask_map[node]", and -1 is
> > > > an invalid array index.
> > >
> > > The invalid array index of -1 is the underlying problem here when
> > > cpumask_of_node(dev_to_node(dev)) is called and cpumask_of_node()
> > > is not NUMA_NO_NODE aware yet.
> > >
> > > In the "numa: make node_to_cpumask_map() NUMA_NO_NODE aware" thread
> > > disscusion, it is requested that it is better to warn about the pcie
> > > device without a node assigned by the firmware before making the
> > > cpumask_of_node() NUMA_NO_NODE aware, so that the system with pci
> > > devices of "NUMA_NO_NODE" node can be fixed by their vendor.
> > >
> > > See: https://lore.kernel.org/lkml/20191011111539.GX2311@hirez.programming.kicks-ass.net/
> >
> > Right. We should warn if the NUMA node number would help us but DT or
> > the firmware didn't give us one.
> >
> > But we can do that independently of any cpumask_of_node() changes.
> > There's no need to do one patch before the other. Even if you make
> > cpumask_of_node() tolerate NUMA_NO_NODE, we'll still get the warning
> > because we're not actually changing any node assignments.
>
> Agreed. And this has been proposed previously I believe but my
> understanding was that Petr was against that.
I don't think Peter was opposed to a warning. I assume you're
referring to [1], which is about how cpumask_of_node() should work.
That's not my area, so I don't have a strong opinion. From that
discussion:
Yunsheng> From what I can see, the problem can be fixed in three
Yunsheng> place:
Yunsheng> 1. Make user dev_to_node return a valid node id
Yunsheng> even when proximity domain is not set by bios(or node id
Yunsheng> set by buggy bios is not valid), which may need info from
Yunsheng> the numa system to make sure it will return a valid node.
Yunsheng>
Yunsheng> 2. User that call cpumask_of_node should ensure the node
Yunsheng> id is valid before calling cpumask_of_node, and user also
Yunsheng> need some info to make ensure node id is valid.
Yunsheng>
Yunsheng> 3. Make sure cpumask_of_node deal with invalid node id as
Yunsheng> this patchset.
Peter> 1) because even it is not set, the device really does belong
Peter> to a node. It is impossible a device will have magic
Peter> uniform access to memory when CPUs cannot.
Peter>
Peter> 2) is already true today, cpumask_of_node() requires a valid
Peter> node_id.
Peter>
Peter> 3) is just wrong and increases overhead for everyone.
I think Peter is advocating (1), i.e., if firmware doesn't tell us a
node ID, we just pick one. We can certainly do that in *addition* to
adding a warning. I'd like it to be a separate patch because I think
we want the warning no matter what so users have some clue that
performance could be better.
If we pick one, I guess we either assign some default, like node 0, to
everything; or we somehow pick a random node to assign.
FWIW, (3) might be wrong, but it is what alpha, ia64, mips (ip27),
powerpc, sparc do already. It'd be nice if they were all the same one
way or the other.
[1] https://lore.kernel.org/linux-arm-kernel/20190831161247.GM2369@hirez.programming.kicks-ass.net/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] PCI: Warn about host bridge device when its numa node is NO_NODE
2019-10-24 17:40 ` Bjorn Helgaas
@ 2019-10-25 8:16 ` Michal Hocko
2019-10-25 8:51 ` Yunsheng Lin
0 siblings, 1 reply; 15+ messages in thread
From: Michal Hocko @ 2019-10-25 8:16 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: peterz, Yunsheng Lin, linux-pci, linux-kernel, robin.murphy,
geert, gregkh, paul.burton
On Thu 24-10-19 12:40:13, Bjorn Helgaas wrote:
> On Thu, Oct 24, 2019 at 11:20:13AM +0200, Michal Hocko wrote:
> > On Wed 23-10-19 12:10:39, Bjorn Helgaas wrote:
> > > On Wed, Oct 23, 2019 at 04:22:43PM +0800, Yunsheng Lin wrote:
> > > > On 2019/10/23 5:04, Bjorn Helgaas wrote:
> > > > > On Sat, Oct 19, 2019 at 02:45:43PM +0800, Yunsheng Lin wrote:
> > >
> > > > > I think the underlying problem you're addressing is that:
> > > > >
> > > > > - NUMA_NO_NODE == -1,
> > > > > - dev_to_node(dev) may return NUMA_NO_NODE,
> > > > > - kmalloc(dev) relies on cpumask_of_node(dev_to_node(dev)), and
> > > > > - cpumask_of_node(NUMA_NO_NODE) makes an invalid array reference
> > > > >
> > > > > For example, on arm64, mips loongson, s390, and x86,
> > > > > cpumask_of_node(node) returns "node_to_cpumask_map[node]", and -1 is
> > > > > an invalid array index.
> > > >
> > > > The invalid array index of -1 is the underlying problem here when
> > > > cpumask_of_node(dev_to_node(dev)) is called and cpumask_of_node()
> > > > is not NUMA_NO_NODE aware yet.
> > > >
> > > > In the "numa: make node_to_cpumask_map() NUMA_NO_NODE aware" thread
> > > > disscusion, it is requested that it is better to warn about the pcie
> > > > device without a node assigned by the firmware before making the
> > > > cpumask_of_node() NUMA_NO_NODE aware, so that the system with pci
> > > > devices of "NUMA_NO_NODE" node can be fixed by their vendor.
> > > >
> > > > See: https://lore.kernel.org/lkml/20191011111539.GX2311@hirez.programming.kicks-ass.net/
> > >
> > > Right. We should warn if the NUMA node number would help us but DT or
> > > the firmware didn't give us one.
> > >
> > > But we can do that independently of any cpumask_of_node() changes.
> > > There's no need to do one patch before the other. Even if you make
> > > cpumask_of_node() tolerate NUMA_NO_NODE, we'll still get the warning
> > > because we're not actually changing any node assignments.
> >
> > Agreed. And this has been proposed previously I believe but my
> > understanding was that Petr was against that.
>
> I don't think Peter was opposed to a warning.
Now, he was opposed to cpumask_of_node handling IIRC.
> I assume you're
> referring to [1], which is about how cpumask_of_node() should work.
> That's not my area, so I don't have a strong opinion. From that
> discussion:
>
> Yunsheng> From what I can see, the problem can be fixed in three
> Yunsheng> place:
> Yunsheng> 1. Make user dev_to_node return a valid node id
> Yunsheng> even when proximity domain is not set by bios(or node id
> Yunsheng> set by buggy bios is not valid), which may need info from
> Yunsheng> the numa system to make sure it will return a valid node.
> Yunsheng>
> Yunsheng> 2. User that call cpumask_of_node should ensure the node
> Yunsheng> id is valid before calling cpumask_of_node, and user also
> Yunsheng> need some info to make ensure node id is valid.
> Yunsheng>
> Yunsheng> 3. Make sure cpumask_of_node deal with invalid node id as
> Yunsheng> this patchset.
>
> Peter> 1) because even it is not set, the device really does belong
> Peter> to a node. It is impossible a device will have magic
> Peter> uniform access to memory when CPUs cannot.
> Peter>
> Peter> 2) is already true today, cpumask_of_node() requires a valid
> Peter> node_id.
> Peter>
> Peter> 3) is just wrong and increases overhead for everyone.
>
> I think Peter is advocating (1), i.e., if firmware doesn't tell us a
> node ID, we just pick one. We can certainly do that in *addition* to
> adding a warning. I'd like it to be a separate patch because I think
> we want the warning no matter what so users have some clue that
> performance could be better.
Yes, those should be two different patches.
> If we pick one, I guess we either assign some default, like node 0, to
> everything; or we somehow pick a random node to assign.
I have tried to explain that picking a default node is tricky because
node 0 is not generally available and you never know when a node might
just disappear if the device is not bound to it.
I really do not see why the proposed online_cpu_mask for that case is
such a big deal. It will likely lead to suboptimal performance but what
do you expect from a suboptimal HW description. There is no question
that the proper node affinity should be set and the warning might really
help here but trying to be clever and find a replacement sounds like
potential for more subtly broken results than doing a straightforward
thing.
But I will just go silent now because I am just repeating myself.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] PCI: Warn about host bridge device when its numa node is NO_NODE
2019-10-25 8:16 ` Michal Hocko
@ 2019-10-25 8:51 ` Yunsheng Lin
0 siblings, 0 replies; 15+ messages in thread
From: Yunsheng Lin @ 2019-10-25 8:51 UTC (permalink / raw)
To: Michal Hocko, Bjorn Helgaas
Cc: peterz, linux-pci, linux-kernel, robin.murphy, geert, gregkh,
paul.burton
On 2019/10/25 16:16, Michal Hocko wrote:
> On Thu 24-10-19 12:40:13, Bjorn Helgaas wrote:
>> On Thu, Oct 24, 2019 at 11:20:13AM +0200, Michal Hocko wrote:
>>> On Wed 23-10-19 12:10:39, Bjorn Helgaas wrote:
>>>> On Wed, Oct 23, 2019 at 04:22:43PM +0800, Yunsheng Lin wrote:
>>>>> On 2019/10/23 5:04, Bjorn Helgaas wrote:
>>>>>> On Sat, Oct 19, 2019 at 02:45:43PM +0800, Yunsheng Lin wrote:
>>>>
>>>>>> I think the underlying problem you're addressing is that:
>>>>>>
>>>>>> - NUMA_NO_NODE == -1,
>>>>>> - dev_to_node(dev) may return NUMA_NO_NODE,
>>>>>> - kmalloc(dev) relies on cpumask_of_node(dev_to_node(dev)), and
>>>>>> - cpumask_of_node(NUMA_NO_NODE) makes an invalid array reference
>>>>>>
>>>>>> For example, on arm64, mips loongson, s390, and x86,
>>>>>> cpumask_of_node(node) returns "node_to_cpumask_map[node]", and -1 is
>>>>>> an invalid array index.
>>>>>
>>>>> The invalid array index of -1 is the underlying problem here when
>>>>> cpumask_of_node(dev_to_node(dev)) is called and cpumask_of_node()
>>>>> is not NUMA_NO_NODE aware yet.
>>>>>
>>>>> In the "numa: make node_to_cpumask_map() NUMA_NO_NODE aware" thread
>>>>> disscusion, it is requested that it is better to warn about the pcie
>>>>> device without a node assigned by the firmware before making the
>>>>> cpumask_of_node() NUMA_NO_NODE aware, so that the system with pci
>>>>> devices of "NUMA_NO_NODE" node can be fixed by their vendor.
>>>>>
>>>>> See: https://lore.kernel.org/lkml/20191011111539.GX2311@hirez.programming.kicks-ass.net/
>>>>
>>>> Right. We should warn if the NUMA node number would help us but DT or
>>>> the firmware didn't give us one.
>>>>
>>>> But we can do that independently of any cpumask_of_node() changes.
>>>> There's no need to do one patch before the other. Even if you make
>>>> cpumask_of_node() tolerate NUMA_NO_NODE, we'll still get the warning
>>>> because we're not actually changing any node assignments.
>>>
>>> Agreed. And this has been proposed previously I believe but my
>>> understanding was that Petr was against that.
>>
>> I don't think Peter was opposed to a warning.
>
> Now, he was opposed to cpumask_of_node handling IIRC.
That was my understanding too.
But I am not sure if Peter is still opposed to cpumask_of_node() after
the pcie device without node affinity is warned in this patch.
From previous discussion [1]:
>Yunsheng> But I failed to see why the above is related to making node_to_cpumask_map()
>Yunsheng> NUMA_NO_NODE aware?
Peter> Your initial bug is for hns3, which is a PCI device, which really _MUST_
Peter> have a node assigned.
Peter> It not having one, is a straight up bug. We must not silently accept
Peter> NO_NODE there, ever."
I think Peter is not opposed to cpumask_of_node() after this patch if I
understand it correctly.
[1] https://lore.kernel.org/lkml/20191011111539.GX2311@hirez.programming.kicks-ass.net/
>
>> I assume you're
>> referring to [1], which is about how cpumask_of_node() should work.
>> That's not my area, so I don't have a strong opinion. From that
>> discussion:
>>
>> Yunsheng> From what I can see, the problem can be fixed in three
>> Yunsheng> place:
>> Yunsheng> 1. Make user dev_to_node return a valid node id
>> Yunsheng> even when proximity domain is not set by bios(or node id
>> Yunsheng> set by buggy bios is not valid), which may need info from
>> Yunsheng> the numa system to make sure it will return a valid node.
>> Yunsheng>
>> Yunsheng> 2. User that call cpumask_of_node should ensure the node
>> Yunsheng> id is valid before calling cpumask_of_node, and user also
>> Yunsheng> need some info to make ensure node id is valid.
>> Yunsheng>
>> Yunsheng> 3. Make sure cpumask_of_node deal with invalid node id as
>> Yunsheng> this patchset.
>>
>> Peter> 1) because even it is not set, the device really does belong
>> Peter> to a node. It is impossible a device will have magic
>> Peter> uniform access to memory when CPUs cannot.
>> Peter>
>> Peter> 2) is already true today, cpumask_of_node() requires a valid
>> Peter> node_id.
>> Peter>
>> Peter> 3) is just wrong and increases overhead for everyone.
>>
>> I think Peter is advocating (1), i.e., if firmware doesn't tell us a
>> node ID, we just pick one. We can certainly do that in *addition* to
>> adding a warning. I'd like it to be a separate patch because I think
>> we want the warning no matter what so users have some clue that
>> performance could be better.
>
> Yes, those should be two different patches.
>
>> If we pick one, I guess we either assign some default, like node 0, to
>> everything; or we somehow pick a random node to assign.
>
> I have tried to explain that picking a default node is tricky because
> node 0 is not generally available and you never know when a node might
> just disappear if the device is not bound to it.
Maybe the example you already mentioned in previous discussion may help
here:
3e8589963773 ("memcg: make it work on sparse non-0-node systems")
>
> I really do not see why the proposed online_cpu_mask for that case is
> such a big deal. It will likely lead to suboptimal performance but what
> do you expect from a suboptimal HW description. There is no question
> that the proper node affinity should be set and the warning might really
> help here but trying to be clever and find a replacement sounds like
> potential for more subtly broken results than doing a straightforward
> thing.
>
> But I will just go silent now because I am just repeating myself.
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] PCI: Warn about host bridge device when its numa node is NO_NODE
2019-10-23 17:10 ` Bjorn Helgaas
2019-10-24 9:20 ` Michal Hocko
@ 2019-10-24 9:39 ` Yunsheng Lin
2019-10-24 10:16 ` Robin Murphy
2 siblings, 0 replies; 15+ messages in thread
From: Yunsheng Lin @ 2019-10-24 9:39 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: linux-pci, linux-kernel, mhocko, peterz, robin.murphy, geert,
gregkh, paul.burton
On 2019/10/24 1:10, Bjorn Helgaas wrote:
> On Wed, Oct 23, 2019 at 04:22:43PM +0800, Yunsheng Lin wrote:
>> On 2019/10/23 5:04, Bjorn Helgaas wrote:
>>> On Sat, Oct 19, 2019 at 02:45:43PM +0800, Yunsheng Lin wrote:
>> And it seems a pci device's parent will always set to the bridge
>> device in pci_setup_device(), and device_add() which will set the
>> node to its parent's when the child device' node is NUMA_NO_NODE,
>> maybe we can add the bridge device' node checking to make sure
>> the pci device really does not have a node assigned, as below:
>>
>> if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE &&
>> dev_to_node(bus->bridge) == NUMA_NO_NODE)
>> dev_err(&bus->dev, FW_BUG "No node assigned on NUMA capable HW. Please contact your vendor for updates.\n");
>
> Anyway, would the attached patch work for you? I have it tentatively
> queued up on pci/enumeration for v5.5.
>
Yes, thanks for making the change when applying.
>>>
>>>
>>>> [1] https://lore.kernel.org/lkml/1568724534-146242-1-git-send-email-linyunsheng@huawei.com/
>>>>
>>>> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
>>>> ---
>>>> drivers/pci/probe.c | 3 +++
>>>> 1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>>>> index 3d5271a..22be96a 100644
>>>> --- a/drivers/pci/probe.c
>>>> +++ b/drivers/pci/probe.c
>>>> @@ -927,6 +927,9 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
>>>> list_add_tail(&bus->node, &pci_root_buses);
>>>> up_write(&pci_bus_sem);
>>>>
>>>> + if (nr_node_ids > 1 && dev_to_node(bus->bridge) == NUMA_NO_NODE)
>>>> + dev_err(bus->bridge, FW_BUG "No node assigned on NUMA capable HW by BIOS. Please contact your vendor for updates.\n");
>>>> +
>>>> return 0;
>>>>
>>>> unregister:
>
> commit 8f8cf239c4f1
> Author: Yunsheng Lin <linyunsheng@huawei.com>
> Date: Sat Oct 19 14:45:43 2019 +0800
>
> PCI: Warn if no host bridge NUMA node info
>
> In pci_call_probe(), we try to run driver probe functions on the node where
> the device is attached. If we don't know which node the device is attached
> to, the driver will likely run on the wrong node. This will still work,
> but performance will not be as good as it could be.
>
> On NUMA systems, warn if we don't know which node a PCI host bridge is
> attached to. This is likely an indication that ACPI didn't supply a _PXM
> method or the DT didn't supply a "numa-node-id" property.
>
> [bhelgaas: commit log, check bus node]
> Link: https://lore.kernel.org/r/1571467543-26125-1-git-send-email-linyunsheng@huawei.com
> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index 3d5271a7a849..40259c38d66a 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -897,6 +897,9 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
> else
> pr_info("PCI host bridge to bus %s\n", name);
>
> + if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE)
> + dev_warn(&bus->dev, "Unknown NUMA node; performance will be reduced\n");
> +
> /* Add initial resources to the bus */
> resource_list_for_each_entry_safe(window, n, &resources) {
> list_move_tail(&window->node, &bridge->windows);
>
> .
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] PCI: Warn about host bridge device when its numa node is NO_NODE
2019-10-23 17:10 ` Bjorn Helgaas
2019-10-24 9:20 ` Michal Hocko
2019-10-24 9:39 ` Yunsheng Lin
@ 2019-10-24 10:16 ` Robin Murphy
2019-10-25 12:51 ` Bjorn Helgaas
2 siblings, 1 reply; 15+ messages in thread
From: Robin Murphy @ 2019-10-24 10:16 UTC (permalink / raw)
To: Bjorn Helgaas, Yunsheng Lin
Cc: linux-pci, linux-kernel, mhocko, peterz, geert, gregkh, paul.burton
On 2019-10-23 6:10 pm, Bjorn Helgaas wrote:
> On Wed, Oct 23, 2019 at 04:22:43PM +0800, Yunsheng Lin wrote:
>> On 2019/10/23 5:04, Bjorn Helgaas wrote:
>>> On Sat, Oct 19, 2019 at 02:45:43PM +0800, Yunsheng Lin wrote:
>
>>> I think the underlying problem you're addressing is that:
>>>
>>> - NUMA_NO_NODE == -1,
>>> - dev_to_node(dev) may return NUMA_NO_NODE,
>>> - kmalloc(dev) relies on cpumask_of_node(dev_to_node(dev)), and
>>> - cpumask_of_node(NUMA_NO_NODE) makes an invalid array reference
>>>
>>> For example, on arm64, mips loongson, s390, and x86,
>>> cpumask_of_node(node) returns "node_to_cpumask_map[node]", and -1 is
>>> an invalid array index.
>>
>> The invalid array index of -1 is the underlying problem here when
>> cpumask_of_node(dev_to_node(dev)) is called and cpumask_of_node()
>> is not NUMA_NO_NODE aware yet.
>>
>> In the "numa: make node_to_cpumask_map() NUMA_NO_NODE aware" thread
>> disscusion, it is requested that it is better to warn about the pcie
>> device without a node assigned by the firmware before making the
>> cpumask_of_node() NUMA_NO_NODE aware, so that the system with pci
>> devices of "NUMA_NO_NODE" node can be fixed by their vendor.
>>
>> See: https://lore.kernel.org/lkml/20191011111539.GX2311@hirez.programming.kicks-ass.net/
>
> Right. We should warn if the NUMA node number would help us but DT or
> the firmware didn't give us one.
>
> But we can do that independently of any cpumask_of_node() changes.
> There's no need to do one patch before the other. Even if you make
> cpumask_of_node() tolerate NUMA_NO_NODE, we'll still get the warning
> because we're not actually changing any node assignments.
>
>> So maybe change the warning to below:
>>
>> if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE)
>> dev_err(&bus->dev, FW_BUG "No node assigned on NUMA capable HW. Please contact your vendor for updates.\n");
>
> I think this is perfect and I don't see the need for the refinement
> below:
>
>> And it seems a pci device's parent will always set to the bridge
>> device in pci_setup_device(), and device_add() which will set the
>> node to its parent's when the child device' node is NUMA_NO_NODE,
>> maybe we can add the bridge device' node checking to make sure
>> the pci device really does not have a node assigned, as below:
>>
>> if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE &&
>> dev_to_node(bus->bridge) == NUMA_NO_NODE)
>> dev_err(&bus->dev, FW_BUG "No node assigned on NUMA capable HW. Please contact your vendor for updates.\n");
>
> Anyway, would the attached patch work for you? I have it tentatively
> queued up on pci/enumeration for v5.5.
>
>>>> It is possible to
>>>> have a PCI bridge shared between two nodes, such that the PCI
>>>> devices have equidistance. But the moment you scale this out, you
>>>> either get devices that are 'local' to a package while having
>>>> multiple packages, or if you maintain a single bridge in a big
>>>> system, things become so slow it all doesn't matter anyway.
>>>> Assigning a node (one of the shared) is, in the generic ase of
>>>> multiple packages, the better solution over assigning all nodes.
>>>>
>>>> As pci_device_add() will assign the pci device' node according to
>>>> the bus the device is on, which is decided by pcibus_to_node().
>>>> Currently different arch may implement the pcibus_to_node() based
>>>> on bus->sysdata or bus device' node, which has the same node as
>>>> the bridge device.
>>>>
>>>> And for devices behind another bridge case, the child bus device
>>>> is setup with proper parent bus device and inherit its parent'
>>>> sysdata in pci_alloc_child_bus(), so the pcie device under the
>>>> child bus should have the same node as the parent bridge when
>>>> device_add() is called, which will set the node to its parent's
>>>> node when the child device' node is NUMA_NO_NODE.
>>>>
>>>> So this patch only warns about the case when a host bridge device
>>>> is registered with a node of NO_NODE in pci_register_host_bridge().
>>>> And it only warns about that when there are more than one numa
>>>> nodes in the system.
>>>
>>>
>>>> [1] https://lore.kernel.org/lkml/1568724534-146242-1-git-send-email-linyunsheng@huawei.com/
>>>>
>>>> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
>>>> ---
>>>> drivers/pci/probe.c | 3 +++
>>>> 1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>>>> index 3d5271a..22be96a 100644
>>>> --- a/drivers/pci/probe.c
>>>> +++ b/drivers/pci/probe.c
>>>> @@ -927,6 +927,9 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
>>>> list_add_tail(&bus->node, &pci_root_buses);
>>>> up_write(&pci_bus_sem);
>>>>
>>>> + if (nr_node_ids > 1 && dev_to_node(bus->bridge) == NUMA_NO_NODE)
>>>> + dev_err(bus->bridge, FW_BUG "No node assigned on NUMA capable HW by BIOS. Please contact your vendor for updates.\n");
>>>> +
>>>> return 0;
>>>>
>>>> unregister:
>
> commit 8f8cf239c4f1
> Author: Yunsheng Lin <linyunsheng@huawei.com>
> Date: Sat Oct 19 14:45:43 2019 +0800
>
> PCI: Warn if no host bridge NUMA node info
>
> In pci_call_probe(), we try to run driver probe functions on the node where
> the device is attached. If we don't know which node the device is attached
> to, the driver will likely run on the wrong node. This will still work,
> but performance will not be as good as it could be.
Is it guaranteed to be purely a performance issue? In other words, is
there definitely no way a physical node could be disabled via
idle/hotplug/etc. such that unattributed devices can silently disappear
while still in use?
>
> On NUMA systems, warn if we don't know which node a PCI host bridge is
> attached to. This is likely an indication that ACPI didn't supply a _PXM
> method or the DT didn't supply a "numa-node-id" property.
>
> [bhelgaas: commit log, check bus node]
> Link: https://lore.kernel.org/r/1571467543-26125-1-git-send-email-linyunsheng@huawei.com
> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index 3d5271a7a849..40259c38d66a 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -897,6 +897,9 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
> else
> pr_info("PCI host bridge to bus %s\n", name);
>
> + if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE)
> + dev_warn(&bus->dev, "Unknown NUMA node; performance will be reduced\n");
I think this still deserves the FW_BUG prefix.
Robin.
> +
> /* Add initial resources to the bus */
> resource_list_for_each_entry_safe(window, n, &resources) {
> list_move_tail(&window->node, &bridge->windows);
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] PCI: Warn about host bridge device when its numa node is NO_NODE
2019-10-24 10:16 ` Robin Murphy
@ 2019-10-25 12:51 ` Bjorn Helgaas
0 siblings, 0 replies; 15+ messages in thread
From: Bjorn Helgaas @ 2019-10-25 12:51 UTC (permalink / raw)
To: Robin Murphy
Cc: Yunsheng Lin, linux-pci, linux-kernel, mhocko, peterz, geert,
gregkh, paul.burton
On Thu, Oct 24, 2019 at 11:16:41AM +0100, Robin Murphy wrote:
> On 2019-10-23 6:10 pm, Bjorn Helgaas wrote:
> > PCI: Warn if no host bridge NUMA node info
> > In pci_call_probe(), we try to run driver probe functions on the node where
> > the device is attached. If we don't know which node the device is attached
> > to, the driver will likely run on the wrong node. This will still work,
> > but performance will not be as good as it could be.
>
> Is it guaranteed to be purely a performance issue? In other words, is there
> definitely no way a physical node could be disabled via idle/hotplug/etc.
> such that unattributed devices can silently disappear while still in use?
I think so. At least, if it's more than a performance issue, I have
no idea what sort of problem might happen or how to deal with it.
> > @@ -897,6 +897,9 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
> > else
> > pr_info("PCI host bridge to bus %s\n", name);
> > + if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE)
> > + dev_warn(&bus->dev, "Unknown NUMA node; performance will be reduced\n");
>
> I think this still deserves the FW_BUG prefix.
Putting the warning here in pci_register_host_bridge() is convenient
for now but doesn't seem like the ideal place.
I'd rather have the warning at the point where we get the node number,
e.g., in pci_acpi_root_get_node() or of_node_to_nid(), where we would
know what's actually required by spec and we could point to the
specific ACPI device or DT device node that's broken. Then I think
we'd have a better case for using FW_BUG.
I'm a little hesitant to use FW_BUG here in pci_register_host_bridge()
because we don't know where the node number was supposed to come from,
so we can't reliably determine that the lack of one is a bug.
Bjorn
^ permalink raw reply [flat|nested] 15+ messages in thread