[Issue] mlx5_core probe failed with error -17 when rescanning devices

[Issue] mlx5_core probe failed with error -17 when rescanning devices

[Yicong Yang]

Hi,

I met an issue of mlx5_core on v5.14-rc6. After performing hot reset of root port
where mellanox connectX-4 locates, I remove the root port and rescan the pci bus,
but the driver cannot probe the device. The hirarchy of the devices:

# lspci -tv
[...]
 +-[0000:80]-+-00.0-[81]--+-00.0  Intel Corporation I350 Gigabit Network Connection
 |           |            +-00.1  Intel Corporation I350 Gigabit Network Connection
 |           |            +-00.2  Intel Corporation I350 Gigabit Network Connection
 |           |            \-00.3  Intel Corporation I350 Gigabit Network Connection
 |           +-04.0-[82]--+-00.0  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
 |           |            \-00.1  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
[...]
# lspci -vx -s 82:00.0
82:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Subsystem: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Flags: fast devsel, IRQ 42, NUMA node 2, IOMMU group 13
	Memory at 280000000000 (64-bit, prefetchable) [size=32M]
	Expansion ROM at f0700000 [disabled] [size=1M]
	Capabilities: [60] Express Endpoint, MSI 00
	Capabilities: [48] Vital Product Data
	Capabilities: [9c] MSI-X: Enable- Count=64 Masked-
	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
	Capabilities: [40] Power Management version 3
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
	Capabilities: [1c0] Secondary PCI Express
	Capabilities: [230] Access Control Services
00: b3 15 15 10 42 01 10 00 00 00 00 02 08 00 80 00
10: 0c 00 00 00 00 28 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 9c 00
30: 00 00 10 f0 60 00 00 00 00 00 00 00 ff 01 00 00

Here is the error log:

[...]
[13904.335445] mlx5_core 0000:82:00.1: Adding to iommu group 29
[13904.341460] mlx5_core 0000:82:00.1: firmware version: 14.22.1002
[13904.347479] mlx5_core 0000:82:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[13904.880505] mlx5_core 0000:82:00.1: E-Switch: Total vports 6, per vport: max uc(1024) max mc(16384)
[13904.893763] mlx5_core 0000:82:00.1: Port module event: module 1, Cable plugged
[13904.917246] sysfs: cannot create duplicate filename '/bus/auxiliary/devices/mlx5_core.eth.2'
[13904.925656] CPU: 64 PID: 957 Comm: kworker/64:2 Tainted: G        W         5.13.0-rc3-bisect+ #37
[13904.934573] Hardware name: Huawei TaiShan 200 (Model 2280)/BC82AMDD, BIOS 2280-V2 CS V5.B170.01 06/30/2021
[13904.944181] Workqueue: events work_for_cpu_fn
[13904.948522] Call trace:
[13904.950949]  dump_backtrace+0x0/0x19c
[13904.954598]  show_stack+0x24/0x30
[13904.957899]  dump_stack+0xc8/0x104
[13904.961287]  sysfs_warn_dup+0x70/0x90
[13904.964936]  sysfs_do_create_link_sd+0xf8/0x100
[13904.969448]  sysfs_create_link+0x2c/0x50
[13904.973355]  bus_add_device+0x74/0x120
[13904.977088]  device_add+0x2f4/0x840
[13904.980562]  __auxiliary_device_add+0x4c/0xb0
[13904.984901]  add_adev+0x9c/0xf0
[13904.988030]  mlx5_rescan_drivers_locked.part.0+0x154/0x1c0
[13904.993491]  mlx5_register_device+0x80/0xe0
[13904.997656]  mlx5_init_one+0x224/0x4dc
[13905.001389]  probe_one+0x1dc/0x4cc
[13905.004776]  local_pci_probe+0x4c/0xc0
[13905.008500]  work_for_cpu_fn+0x28/0x40
[13905.012232]  process_one_work+0x1dc/0x48c
[13905.016223]  worker_thread+0x2e8/0x464
[13905.019955]  kthread+0x168/0x16c
[13905.023171]  ret_from_fork+0x10/0x18
[13905.026764] auxiliary mlx5_core.eth.2: adding auxiliary device failed!: -17
[13905.033698] mlx5_core 0000:82:00.1: add_drivers:424:(pid 957): Device[0] (eth) failed to load
[13907.274212] mlx5_core 0000:82:00.1: E-Switch: cleanup
[13907.896435] mlx5_core 0000:82:00.1: probe_one:1484:(pid 957): mlx5_init_one failed with error code -17
[13907.905855] mlx5_core: probe of 0000:82:00.1 failed with error -17

The test script I used:

#!/bin/bash
setpci -s 80:04.0 0x3e.b=0x43 # perform hot reset by setting secondary bus reset bit
setpci -s 80:04.0 0x3e.b=0x3 # clear secondary bus reset bit
echo 1 > /sys/bus/pci/devices/0000:80:04.0/remove
echo 1 > /sys/bus/pci/rescan

There is no such issue on v5.13-rc4. So I did a git bisect and here is log:

[...]
# good: [c7d2ef5dd4b03ed0ee1d13bc0c55f9cf62d49bd6] net/packet: annotate accesses to po->bind
git bisect good c7d2ef5dd4b03ed0ee1d13bc0c55f9cf62d49bd6
# bad: [bc39f6792ede3a830b1893c9133636b9f6991e59] Merge tag 'mlx5-fixes-2021-06-16' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux
git bisect bad bc39f6792ede3a830b1893c9133636b9f6991e59
# good: [94a4b8414d3e91104873007b659252f855ee344a] net/mlx5: Fix error path for set HCA defaults
git bisect good 94a4b8414d3e91104873007b659252f855ee344a
# good: [65fb7d109abe3a1a9f1c2d3ba7e1249bc978d5f0] net/mlx5: DR, Fix STEv1 incorrect L3 decapsulation padding
git bisect good 65fb7d109abe3a1a9f1c2d3ba7e1249bc978d5f0
# bad: [0232fc2ddcf4ffe01069fd1aa07922652120f44a] net/mlx5: Reset mkey index on creation
git bisect bad 0232fc2ddcf4ffe01069fd1aa07922652120f44a
# bad: [a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6] net/mlx5e: Don't create devices during unload flow
git bisect bad a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6
# first bad commit: [a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6] net/mlx5e: Don't create devices during unload flow

So after reverting a5ae8fc9058e ("net/mlx5e: Don't create devices during unload flow") the issue
is resolved. Seems the devlink file is not removed properly and the unremoved file fails the
probe process. I don't know about the driver so I hope somebody can get this fixed.

Thanks,
Yicong

Re: [Issue] mlx5_core probe failed with error -17 when rescanning devices

[Leon Romanovsky]

On Thu, Aug 19, 2021 at 08:46:32PM +0800, Yicong Yang wrote:
> Hi,
> 
> I met an issue of mlx5_core on v5.14-rc6. After performing hot reset of root port
> where mellanox connectX-4 locates, I remove the root port and rescan the pci bus,
> but the driver cannot probe the device. The hirarchy of the devices:
> 
> # lspci -tv
> [...]
>  +-[0000:80]-+-00.0-[81]--+-00.0  Intel Corporation I350 Gigabit Network Connection
>  |           |            +-00.1  Intel Corporation I350 Gigabit Network Connection
>  |           |            +-00.2  Intel Corporation I350 Gigabit Network Connection
>  |           |            \-00.3  Intel Corporation I350 Gigabit Network Connection
>  |           +-04.0-[82]--+-00.0  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
>  |           |            \-00.1  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
> [...]
> # lspci -vx -s 82:00.0
> 82:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
> 	Subsystem: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
> 	Flags: fast devsel, IRQ 42, NUMA node 2, IOMMU group 13
> 	Memory at 280000000000 (64-bit, prefetchable) [size=32M]
> 	Expansion ROM at f0700000 [disabled] [size=1M]
> 	Capabilities: [60] Express Endpoint, MSI 00
> 	Capabilities: [48] Vital Product Data
> 	Capabilities: [9c] MSI-X: Enable- Count=64 Masked-
> 	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
> 	Capabilities: [40] Power Management version 3
> 	Capabilities: [100] Advanced Error Reporting
> 	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
> 	Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
> 	Capabilities: [1c0] Secondary PCI Express
> 	Capabilities: [230] Access Control Services
> 00: b3 15 15 10 42 01 10 00 00 00 00 02 08 00 80 00
> 10: 0c 00 00 00 00 28 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 9c 00
> 30: 00 00 10 f0 60 00 00 00 00 00 00 00 ff 01 00 00
> 
> Here is the error log:
> 
> [...]
> [13904.335445] mlx5_core 0000:82:00.1: Adding to iommu group 29
> [13904.341460] mlx5_core 0000:82:00.1: firmware version: 14.22.1002
> [13904.347479] mlx5_core 0000:82:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
> [13904.880505] mlx5_core 0000:82:00.1: E-Switch: Total vports 6, per vport: max uc(1024) max mc(16384)
> [13904.893763] mlx5_core 0000:82:00.1: Port module event: module 1, Cable plugged
> [13904.917246] sysfs: cannot create duplicate filename '/bus/auxiliary/devices/mlx5_core.eth.2'
> [13904.925656] CPU: 64 PID: 957 Comm: kworker/64:2 Tainted: G        W         5.13.0-rc3-bisect+ #37
> [13904.934573] Hardware name: Huawei TaiShan 200 (Model 2280)/BC82AMDD, BIOS 2280-V2 CS V5.B170.01 06/30/2021
> [13904.944181] Workqueue: events work_for_cpu_fn
> [13904.948522] Call trace:
> [13904.950949]  dump_backtrace+0x0/0x19c
> [13904.954598]  show_stack+0x24/0x30
> [13904.957899]  dump_stack+0xc8/0x104
> [13904.961287]  sysfs_warn_dup+0x70/0x90
> [13904.964936]  sysfs_do_create_link_sd+0xf8/0x100
> [13904.969448]  sysfs_create_link+0x2c/0x50
> [13904.973355]  bus_add_device+0x74/0x120
> [13904.977088]  device_add+0x2f4/0x840
> [13904.980562]  __auxiliary_device_add+0x4c/0xb0
> [13904.984901]  add_adev+0x9c/0xf0
> [13904.988030]  mlx5_rescan_drivers_locked.part.0+0x154/0x1c0
> [13904.993491]  mlx5_register_device+0x80/0xe0
> [13904.997656]  mlx5_init_one+0x224/0x4dc
> [13905.001389]  probe_one+0x1dc/0x4cc
> [13905.004776]  local_pci_probe+0x4c/0xc0
> [13905.008500]  work_for_cpu_fn+0x28/0x40
> [13905.012232]  process_one_work+0x1dc/0x48c
> [13905.016223]  worker_thread+0x2e8/0x464
> [13905.019955]  kthread+0x168/0x16c
> [13905.023171]  ret_from_fork+0x10/0x18
> [13905.026764] auxiliary mlx5_core.eth.2: adding auxiliary device failed!: -17
> [13905.033698] mlx5_core 0000:82:00.1: add_drivers:424:(pid 957): Device[0] (eth) failed to load
> [13907.274212] mlx5_core 0000:82:00.1: E-Switch: cleanup
> [13907.896435] mlx5_core 0000:82:00.1: probe_one:1484:(pid 957): mlx5_init_one failed with error code -17
> [13907.905855] mlx5_core: probe of 0000:82:00.1 failed with error -17
> 
> The test script I used:
> 
> #!/bin/bash
> setpci -s 80:04.0 0x3e.b=0x43 # perform hot reset by setting secondary bus reset bit
> setpci -s 80:04.0 0x3e.b=0x3 # clear secondary bus reset bit
> echo 1 > /sys/bus/pci/devices/0000:80:04.0/remove
> echo 1 > /sys/bus/pci/rescan
> 
> There is no such issue on v5.13-rc4. So I did a git bisect and here is log:
> 
> [...]
> # good: [c7d2ef5dd4b03ed0ee1d13bc0c55f9cf62d49bd6] net/packet: annotate accesses to po->bind
> git bisect good c7d2ef5dd4b03ed0ee1d13bc0c55f9cf62d49bd6
> # bad: [bc39f6792ede3a830b1893c9133636b9f6991e59] Merge tag 'mlx5-fixes-2021-06-16' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux
> git bisect bad bc39f6792ede3a830b1893c9133636b9f6991e59
> # good: [94a4b8414d3e91104873007b659252f855ee344a] net/mlx5: Fix error path for set HCA defaults
> git bisect good 94a4b8414d3e91104873007b659252f855ee344a
> # good: [65fb7d109abe3a1a9f1c2d3ba7e1249bc978d5f0] net/mlx5: DR, Fix STEv1 incorrect L3 decapsulation padding
> git bisect good 65fb7d109abe3a1a9f1c2d3ba7e1249bc978d5f0
> # bad: [0232fc2ddcf4ffe01069fd1aa07922652120f44a] net/mlx5: Reset mkey index on creation
> git bisect bad 0232fc2ddcf4ffe01069fd1aa07922652120f44a
> # bad: [a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6] net/mlx5e: Don't create devices during unload flow
> git bisect bad a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6
> # first bad commit: [a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6] net/mlx5e: Don't create devices during unload flow
> 
> So after reverting a5ae8fc9058e ("net/mlx5e: Don't create devices during unload flow") the issue
> is resolved. Seems the devlink file is not removed properly and the unremoved file fails the
> probe process. I don't know about the driver so I hope somebody can get this fixed.

Thank you for your report,

Your command "echo 1 > /sys/bus/pci/devices/0000:80:04.0/remove" was
supposed to remove all auxiliary devices, but because of commit
a5ae8fc9058e ("net/mlx5e: Don't create devices during unload flow"),
the deletion was skipped.

Can you please try this patch instead of revert?

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
index ff6b03dc7e32..e8093c4e09d4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/dev.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
@@ -450,7 +450,7 @@ int mlx5_register_device(struct mlx5_core_dev *dev)
 void mlx5_unregister_device(struct mlx5_core_dev *dev)
 {
        mutex_lock(&mlx5_intf_mutex);
-       dev->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV;
+       dev->priv.flags = MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV;
        mlx5_rescan_drivers_locked(dev);
        mutex_unlock(&mlx5_intf_mutex);
 }


> 
> Thanks,
> Yicong
> 
> 

Re: [Issue] mlx5_core probe failed with error -17 when rescanning devices

[Yicong Yang]

On 2021/8/20 21:53, Leon Romanovsky wrote:
> On Thu, Aug 19, 2021 at 08:46:32PM +0800, Yicong Yang wrote:
>> Hi,
>>
>> I met an issue of mlx5_core on v5.14-rc6. After performing hot reset of root port
>> where mellanox connectX-4 locates, I remove the root port and rescan the pci bus,
>> but the driver cannot probe the device. The hirarchy of the devices:
>>
>> # lspci -tv
>> [...]
>>  +-[0000:80]-+-00.0-[81]--+-00.0  Intel Corporation I350 Gigabit Network Connection
>>  |           |            +-00.1  Intel Corporation I350 Gigabit Network Connection
>>  |           |            +-00.2  Intel Corporation I350 Gigabit Network Connection
>>  |           |            \-00.3  Intel Corporation I350 Gigabit Network Connection
>>  |           +-04.0-[82]--+-00.0  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
>>  |           |            \-00.1  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
>> [...]
>> # lspci -vx -s 82:00.0
>> 82:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
>> 	Subsystem: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
>> 	Flags: fast devsel, IRQ 42, NUMA node 2, IOMMU group 13
>> 	Memory at 280000000000 (64-bit, prefetchable) [size=32M]
>> 	Expansion ROM at f0700000 [disabled] [size=1M]
>> 	Capabilities: [60] Express Endpoint, MSI 00
>> 	Capabilities: [48] Vital Product Data
>> 	Capabilities: [9c] MSI-X: Enable- Count=64 Masked-
>> 	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
>> 	Capabilities: [40] Power Management version 3
>> 	Capabilities: [100] Advanced Error Reporting
>> 	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
>> 	Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
>> 	Capabilities: [1c0] Secondary PCI Express
>> 	Capabilities: [230] Access Control Services
>> 00: b3 15 15 10 42 01 10 00 00 00 00 02 08 00 80 00
>> 10: 0c 00 00 00 00 28 00 00 00 00 00 00 00 00 00 00
>> 20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 9c 00
>> 30: 00 00 10 f0 60 00 00 00 00 00 00 00 ff 01 00 00
>>
>> Here is the error log:
>>
>> [...]
>> [13904.335445] mlx5_core 0000:82:00.1: Adding to iommu group 29
>> [13904.341460] mlx5_core 0000:82:00.1: firmware version: 14.22.1002
>> [13904.347479] mlx5_core 0000:82:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
>> [13904.880505] mlx5_core 0000:82:00.1: E-Switch: Total vports 6, per vport: max uc(1024) max mc(16384)
>> [13904.893763] mlx5_core 0000:82:00.1: Port module event: module 1, Cable plugged
>> [13904.917246] sysfs: cannot create duplicate filename '/bus/auxiliary/devices/mlx5_core.eth.2'
>> [13904.925656] CPU: 64 PID: 957 Comm: kworker/64:2 Tainted: G        W         5.13.0-rc3-bisect+ #37
>> [13904.934573] Hardware name: Huawei TaiShan 200 (Model 2280)/BC82AMDD, BIOS 2280-V2 CS V5.B170.01 06/30/2021
>> [13904.944181] Workqueue: events work_for_cpu_fn
>> [13904.948522] Call trace:
>> [13904.950949]  dump_backtrace+0x0/0x19c
>> [13904.954598]  show_stack+0x24/0x30
>> [13904.957899]  dump_stack+0xc8/0x104
>> [13904.961287]  sysfs_warn_dup+0x70/0x90
>> [13904.964936]  sysfs_do_create_link_sd+0xf8/0x100
>> [13904.969448]  sysfs_create_link+0x2c/0x50
>> [13904.973355]  bus_add_device+0x74/0x120
>> [13904.977088]  device_add+0x2f4/0x840
>> [13904.980562]  __auxiliary_device_add+0x4c/0xb0
>> [13904.984901]  add_adev+0x9c/0xf0
>> [13904.988030]  mlx5_rescan_drivers_locked.part.0+0x154/0x1c0
>> [13904.993491]  mlx5_register_device+0x80/0xe0
>> [13904.997656]  mlx5_init_one+0x224/0x4dc
>> [13905.001389]  probe_one+0x1dc/0x4cc
>> [13905.004776]  local_pci_probe+0x4c/0xc0
>> [13905.008500]  work_for_cpu_fn+0x28/0x40
>> [13905.012232]  process_one_work+0x1dc/0x48c
>> [13905.016223]  worker_thread+0x2e8/0x464
>> [13905.019955]  kthread+0x168/0x16c
>> [13905.023171]  ret_from_fork+0x10/0x18
>> [13905.026764] auxiliary mlx5_core.eth.2: adding auxiliary device failed!: -17
>> [13905.033698] mlx5_core 0000:82:00.1: add_drivers:424:(pid 957): Device[0] (eth) failed to load
>> [13907.274212] mlx5_core 0000:82:00.1: E-Switch: cleanup
>> [13907.896435] mlx5_core 0000:82:00.1: probe_one:1484:(pid 957): mlx5_init_one failed with error code -17
>> [13907.905855] mlx5_core: probe of 0000:82:00.1 failed with error -17
>>
>> The test script I used:
>>
>> #!/bin/bash
>> setpci -s 80:04.0 0x3e.b=0x43 # perform hot reset by setting secondary bus reset bit
>> setpci -s 80:04.0 0x3e.b=0x3 # clear secondary bus reset bit
>> echo 1 > /sys/bus/pci/devices/0000:80:04.0/remove
>> echo 1 > /sys/bus/pci/rescan
>>
>> There is no such issue on v5.13-rc4. So I did a git bisect and here is log:
>>
>> [...]
>> # good: [c7d2ef5dd4b03ed0ee1d13bc0c55f9cf62d49bd6] net/packet: annotate accesses to po->bind
>> git bisect good c7d2ef5dd4b03ed0ee1d13bc0c55f9cf62d49bd6
>> # bad: [bc39f6792ede3a830b1893c9133636b9f6991e59] Merge tag 'mlx5-fixes-2021-06-16' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux
>> git bisect bad bc39f6792ede3a830b1893c9133636b9f6991e59
>> # good: [94a4b8414d3e91104873007b659252f855ee344a] net/mlx5: Fix error path for set HCA defaults
>> git bisect good 94a4b8414d3e91104873007b659252f855ee344a
>> # good: [65fb7d109abe3a1a9f1c2d3ba7e1249bc978d5f0] net/mlx5: DR, Fix STEv1 incorrect L3 decapsulation padding
>> git bisect good 65fb7d109abe3a1a9f1c2d3ba7e1249bc978d5f0
>> # bad: [0232fc2ddcf4ffe01069fd1aa07922652120f44a] net/mlx5: Reset mkey index on creation
>> git bisect bad 0232fc2ddcf4ffe01069fd1aa07922652120f44a
>> # bad: [a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6] net/mlx5e: Don't create devices during unload flow
>> git bisect bad a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6
>> # first bad commit: [a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6] net/mlx5e: Don't create devices during unload flow
>>
>> So after reverting a5ae8fc9058e ("net/mlx5e: Don't create devices during unload flow") the issue
>> is resolved. Seems the devlink file is not removed properly and the unremoved file fails the
>> probe process. I don't know about the driver so I hope somebody can get this fixed.
> 
> Thank you for your report,
> 
> Your command "echo 1 > /sys/bus/pci/devices/0000:80:04.0/remove" was
> supposed to remove all auxiliary devices, but because of commit
> a5ae8fc9058e ("net/mlx5e: Don't create devices during unload flow"),
> the deletion was skipped.
> 
> Can you please try this patch instead of revert?
> 
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
> index ff6b03dc7e32..e8093c4e09d4 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/dev.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
> @@ -450,7 +450,7 @@ int mlx5_register_device(struct mlx5_core_dev *dev)
>  void mlx5_unregister_device(struct mlx5_core_dev *dev)
>  {
>         mutex_lock(&mlx5_intf_mutex);
> -       dev->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV;
> +       dev->priv.flags = MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV;
>         mlx5_rescan_drivers_locked(dev);
>         mutex_unlock(&mlx5_intf_mutex);
>  }
> 

Thanks for the patch. I manually applied the change on v5.14-rc6 and the driver
can probe the devices after rescannig.

Tested-by: Yicong Yang <yangyicong@hisilicon.com>

> 
>>
>> Thanks,
>> Yicong
>>
>>
> 
> .
>