* mlx5 endpoint driver problem
@ 2017-05-09 16:25 Joao Pinto
2017-05-09 17:35 ` Saeed Mahameed
0 siblings, 1 reply; 5+ messages in thread
From: Joao Pinto @ 2017-05-09 16:25 UTC (permalink / raw)
To: saeedm; +Cc: netdev
Hello,
I am making tests with a Mellanox MLX5 Endpoint, and I am getting kernel hangs
when trying to enable the hca:
mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit PCI DMA mask
mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit consistent PCI DMA mask
mlx5_core 0000:01:00.0: firmware version: 16.19.21102
INFO: task swapper:1 blocked for more than 10 seconds.
Not tainted 4.11.0-BETAMSIX1 #51
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
swapper D 0 1 0 0x00000000
Stack Trace:
__switch_to+0x0/0x94
__schedule+0x1da/0x8b0
schedule+0x26/0x6c
schedule_timeout+0x2da/0x380
wait_for_completion+0x92/0x104
mlx5_cmd_exec+0x70e/0xd60
mlx5_load_one+0x1b4/0xad8
init_one+0x404/0x600
pci_device_probe+0x122/0x1f0
really_probe+0x1ac/0x348
__driver_attach+0xa8/0xd0
bus_for_each_dev+0x3c/0x74
bus_add_driver+0xc2/0x184
driver_register+0x50/0xec
init+0x40/0x60
(...)
Stack Trace:
__switch_to+0x0/0x94
__schedule+0x1da/0x8b0
schedule+0x26/0x6c
schedule_timeout+0x2da/0x380
wait_for_completion+0x92/0x104
mlx5_cmd_exec+0x70e/0xd60
mlx5_load_one+0x1b4/0xad8
init_one+0x404/0x600
pci_device_probe+0x122/0x1f0
really_probe+0x1ac/0x348
__driver_attach+0xa8/0xd0
bus_for_each_dev+0x3c/0x74
bus_add_driver+0xc2/0x184
driver_register+0x50/0xec
init+0x40/0x60
mlx5_core 0000:01:00.0: wait_func:882:(pid 1): ENABLE_HCA(0x104) timeout. Will
cause a leak of a command resource
mlx5_core 0000:01:00.0: enable hca failed
mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -110
mlx5_core: probe of 0000:01:00.0 failed with error -110
Could you give me a clue of what might be happennig?
Thanks,
Joao
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mlx5 endpoint driver problem
2017-05-09 16:25 mlx5 endpoint driver problem Joao Pinto
@ 2017-05-09 17:35 ` Saeed Mahameed
2017-05-09 17:38 ` Joao Pinto
0 siblings, 1 reply; 5+ messages in thread
From: Saeed Mahameed @ 2017-05-09 17:35 UTC (permalink / raw)
To: Joao Pinto; +Cc: Saeed Mahameed, netdev
On Tue, May 9, 2017 at 7:25 PM, Joao Pinto <Joao.Pinto@synopsys.com> wrote:
> Hello,
>
> I am making tests with a Mellanox MLX5 Endpoint, and I am getting kernel hangs
> when trying to enable the hca:
>
> mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
> mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit PCI DMA mask
> mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit consistent PCI DMA mask
> mlx5_core 0000:01:00.0: firmware version: 16.19.21102
> INFO: task swapper:1 blocked for more than 10 seconds.
> Not tainted 4.11.0-BETAMSIX1 #51
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> swapper D 0 1 0 0x00000000
>
> Stack Trace:
> __switch_to+0x0/0x94
> __schedule+0x1da/0x8b0
> schedule+0x26/0x6c
> schedule_timeout+0x2da/0x380
> wait_for_completion+0x92/0x104
> mlx5_cmd_exec+0x70e/0xd60
> mlx5_load_one+0x1b4/0xad8
> init_one+0x404/0x600
> pci_device_probe+0x122/0x1f0
> really_probe+0x1ac/0x348
> __driver_attach+0xa8/0xd0
> bus_for_each_dev+0x3c/0x74
> bus_add_driver+0xc2/0x184
> driver_register+0x50/0xec
> init+0x40/0x60
>
> (...)
>
> Stack Trace:
> __switch_to+0x0/0x94
> __schedule+0x1da/0x8b0
> schedule+0x26/0x6c
> schedule_timeout+0x2da/0x380
> wait_for_completion+0x92/0x104
> mlx5_cmd_exec+0x70e/0xd60
> mlx5_load_one+0x1b4/0xad8
> init_one+0x404/0x600
> pci_device_probe+0x122/0x1f0
> really_probe+0x1ac/0x348
> __driver_attach+0xa8/0xd0
> bus_for_each_dev+0x3c/0x74
> bus_add_driver+0xc2/0x184
> driver_register+0x50/0xec
> init+0x40/0x60
> mlx5_core 0000:01:00.0: wait_func:882:(pid 1): ENABLE_HCA(0x104) timeout. Will
> cause a leak of a command resource
> mlx5_core 0000:01:00.0: enable hca failed
> mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -110
> mlx5_core: probe of 0000:01:00.0 failed with error -110
>
> Could you give me a clue of what might be happennig?
>
Hi Joao,
Looks like FW is not responding, most likely due to the DMA mask
setting warnings, which architecture is this ?
> Thanks,
> Joao
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mlx5 endpoint driver problem
2017-05-09 17:35 ` Saeed Mahameed
@ 2017-05-09 17:38 ` Joao Pinto
2017-05-09 17:44 ` Saeed Mahameed
0 siblings, 1 reply; 5+ messages in thread
From: Joao Pinto @ 2017-05-09 17:38 UTC (permalink / raw)
To: Saeed Mahameed, Joao Pinto; +Cc: Saeed Mahameed, netdev
Hi Saeed,
Às 6:35 PM de 5/9/2017, Saeed Mahameed escreveu:
> On Tue, May 9, 2017 at 7:25 PM, Joao Pinto <Joao.Pinto@synopsys.com> wrote:
>> Hello,
>>
>> I am making tests with a Mellanox MLX5 Endpoint, and I am getting kernel hangs
>> when trying to enable the hca:
>>
>> mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
>> mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit PCI DMA mask
>> mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit consistent PCI DMA mask
>> mlx5_core 0000:01:00.0: firmware version: 16.19.21102
>> INFO: task swapper:1 blocked for more than 10 seconds.
>> Not tainted 4.11.0-BETAMSIX1 #51
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> swapper D 0 1 0 0x00000000
>>
>> Stack Trace:
>> __switch_to+0x0/0x94
>> __schedule+0x1da/0x8b0
>> schedule+0x26/0x6c
>> schedule_timeout+0x2da/0x380
>> wait_for_completion+0x92/0x104
>> mlx5_cmd_exec+0x70e/0xd60
>> mlx5_load_one+0x1b4/0xad8
>> init_one+0x404/0x600
>> pci_device_probe+0x122/0x1f0
>> really_probe+0x1ac/0x348
>> __driver_attach+0xa8/0xd0
>> bus_for_each_dev+0x3c/0x74
>> bus_add_driver+0xc2/0x184
>> driver_register+0x50/0xec
>> init+0x40/0x60
>>
>> (...)
>>
>> Stack Trace:
>> __switch_to+0x0/0x94
>> __schedule+0x1da/0x8b0
>> schedule+0x26/0x6c
>> schedule_timeout+0x2da/0x380
>> wait_for_completion+0x92/0x104
>> mlx5_cmd_exec+0x70e/0xd60
>> mlx5_load_one+0x1b4/0xad8
>> init_one+0x404/0x600
>> pci_device_probe+0x122/0x1f0
>> really_probe+0x1ac/0x348
>> __driver_attach+0xa8/0xd0
>> bus_for_each_dev+0x3c/0x74
>> bus_add_driver+0xc2/0x184
>> driver_register+0x50/0xec
>> init+0x40/0x60
>> mlx5_core 0000:01:00.0: wait_func:882:(pid 1): ENABLE_HCA(0x104) timeout. Will
>> cause a leak of a command resource
>> mlx5_core 0000:01:00.0: enable hca failed
>> mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -110
>> mlx5_core: probe of 0000:01:00.0 failed with error -110
>>
>> Could you give me a clue of what might be happennig?
>>
>
> Hi Joao,
>
> Looks like FW is not responding, most likely due to the DMA mask
> setting warnings, which architecture is this ?
>
>> Thanks,
>> Joao
I am working with a 32-bit ARC processor based board, connected to a prototyped
Gen4 PCI RC.
Thanks,
Joao
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mlx5 endpoint driver problem
2017-05-09 17:38 ` Joao Pinto
@ 2017-05-09 17:44 ` Saeed Mahameed
2017-05-09 17:57 ` Joao Pinto
0 siblings, 1 reply; 5+ messages in thread
From: Saeed Mahameed @ 2017-05-09 17:44 UTC (permalink / raw)
To: Joao Pinto; +Cc: Saeed Mahameed, netdev
On Tue, May 9, 2017 at 8:38 PM, Joao Pinto <Joao.Pinto@synopsys.com> wrote:
> Hi Saeed,
>
> Às 6:35 PM de 5/9/2017, Saeed Mahameed escreveu:
>> On Tue, May 9, 2017 at 7:25 PM, Joao Pinto <Joao.Pinto@synopsys.com> wrote:
>>> Hello,
>>>
>>> I am making tests with a Mellanox MLX5 Endpoint, and I am getting kernel hangs
>>> when trying to enable the hca:
>>>
>>> mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
>>> mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit PCI DMA mask
>>> mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit consistent PCI DMA mask
>>> mlx5_core 0000:01:00.0: firmware version: 16.19.21102
>>> INFO: task swapper:1 blocked for more than 10 seconds.
>>> Not tainted 4.11.0-BETAMSIX1 #51
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> swapper D 0 1 0 0x00000000
>>>
>>> Stack Trace:
>>> __switch_to+0x0/0x94
>>> __schedule+0x1da/0x8b0
>>> schedule+0x26/0x6c
>>> schedule_timeout+0x2da/0x380
>>> wait_for_completion+0x92/0x104
>>> mlx5_cmd_exec+0x70e/0xd60
>>> mlx5_load_one+0x1b4/0xad8
>>> init_one+0x404/0x600
>>> pci_device_probe+0x122/0x1f0
>>> really_probe+0x1ac/0x348
>>> __driver_attach+0xa8/0xd0
>>> bus_for_each_dev+0x3c/0x74
>>> bus_add_driver+0xc2/0x184
>>> driver_register+0x50/0xec
>>> init+0x40/0x60
>>>
>>> (...)
>>>
>>> Stack Trace:
>>> __switch_to+0x0/0x94
>>> __schedule+0x1da/0x8b0
>>> schedule+0x26/0x6c
>>> schedule_timeout+0x2da/0x380
>>> wait_for_completion+0x92/0x104
>>> mlx5_cmd_exec+0x70e/0xd60
>>> mlx5_load_one+0x1b4/0xad8
>>> init_one+0x404/0x600
>>> pci_device_probe+0x122/0x1f0
>>> really_probe+0x1ac/0x348
>>> __driver_attach+0xa8/0xd0
>>> bus_for_each_dev+0x3c/0x74
>>> bus_add_driver+0xc2/0x184
>>> driver_register+0x50/0xec
>>> init+0x40/0x60
>>> mlx5_core 0000:01:00.0: wait_func:882:(pid 1): ENABLE_HCA(0x104) timeout. Will
>>> cause a leak of a command resource
>>> mlx5_core 0000:01:00.0: enable hca failed
>>> mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -110
>>> mlx5_core: probe of 0000:01:00.0 failed with error -110
>>>
>>> Could you give me a clue of what might be happennig?
>>>
>>
>> Hi Joao,
>>
>> Looks like FW is not responding, most likely due to the DMA mask
>> setting warnings, which architecture is this ?
>>
>>> Thanks,
>>> Joao
>
> I am working with a 32-bit ARC processor based board, connected to a prototyped
> Gen4 PCI RC.
>
Ok, i will consult with our PCI and FW experts and get back to you.
please note that the current mlx5 driver was never tested on 32-bit
architecture and might not work properly for you.
> Thanks,
> Joao
>
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mlx5 endpoint driver problem
2017-05-09 17:44 ` Saeed Mahameed
@ 2017-05-09 17:57 ` Joao Pinto
0 siblings, 0 replies; 5+ messages in thread
From: Joao Pinto @ 2017-05-09 17:57 UTC (permalink / raw)
To: Saeed Mahameed, Joao Pinto; +Cc: Saeed Mahameed, netdev
Hi again Saeed,
Às 6:44 PM de 5/9/2017, Saeed Mahameed escreveu:
> On Tue, May 9, 2017 at 8:38 PM, Joao Pinto <Joao.Pinto@synopsys.com> wrote:
>> Hi Saeed,
>>
>> Às 6:35 PM de 5/9/2017, Saeed Mahameed escreveu:
>>> On Tue, May 9, 2017 at 7:25 PM, Joao Pinto <Joao.Pinto@synopsys.com> wrote:
>>>> Hello,
>>>>
>>>> I am making tests with a Mellanox MLX5 Endpoint, and I am getting kernel hangs
>>>> when trying to enable the hca:
>>>>
>>>> mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
>>>> mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit PCI DMA mask
>>>> mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit consistent PCI DMA mask
>>>> mlx5_core 0000:01:00.0: firmware version: 16.19.21102
>>>> INFO: task swapper:1 blocked for more than 10 seconds.
>>>> Not tainted 4.11.0-BETAMSIX1 #51
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> swapper D 0 1 0 0x00000000
>>>>
>>>> Stack Trace:
>>>> __switch_to+0x0/0x94
>>>> __schedule+0x1da/0x8b0
>>>> schedule+0x26/0x6c
>>>> schedule_timeout+0x2da/0x380
>>>> wait_for_completion+0x92/0x104
>>>> mlx5_cmd_exec+0x70e/0xd60
>>>> mlx5_load_one+0x1b4/0xad8
>>>> init_one+0x404/0x600
>>>> pci_device_probe+0x122/0x1f0
>>>> really_probe+0x1ac/0x348
>>>> __driver_attach+0xa8/0xd0
>>>> bus_for_each_dev+0x3c/0x74
>>>> bus_add_driver+0xc2/0x184
>>>> driver_register+0x50/0xec
>>>> init+0x40/0x60
>>>>
>>>> (...)
>>>>
>>>> Stack Trace:
>>>> __switch_to+0x0/0x94
>>>> __schedule+0x1da/0x8b0
>>>> schedule+0x26/0x6c
>>>> schedule_timeout+0x2da/0x380
>>>> wait_for_completion+0x92/0x104
>>>> mlx5_cmd_exec+0x70e/0xd60
>>>> mlx5_load_one+0x1b4/0xad8
>>>> init_one+0x404/0x600
>>>> pci_device_probe+0x122/0x1f0
>>>> really_probe+0x1ac/0x348
>>>> __driver_attach+0xa8/0xd0
>>>> bus_for_each_dev+0x3c/0x74
>>>> bus_add_driver+0xc2/0x184
>>>> driver_register+0x50/0xec
>>>> init+0x40/0x60
>>>> mlx5_core 0000:01:00.0: wait_func:882:(pid 1): ENABLE_HCA(0x104) timeout. Will
>>>> cause a leak of a command resource
>>>> mlx5_core 0000:01:00.0: enable hca failed
>>>> mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -110
>>>> mlx5_core: probe of 0000:01:00.0 failed with error -110
>>>>
>>>> Could you give me a clue of what might be happennig?
>>>>
>>>
>>> Hi Joao,
>>>
>>> Looks like FW is not responding, most likely due to the DMA mask
>>> setting warnings, which architecture is this ?
>>>
>>>> Thanks,
>>>> Joao
>>
>> I am working with a 32-bit ARC processor based board, connected to a prototyped
>> Gen4 PCI RC.
>>
>
> Ok, i will consult with our PCI and FW experts and get back to you.
>
> please note that the current mlx5 driver was never tested on 32-bit
> architecture and might not work properly for you.
I have new data for you. My colleague is using a Mellanox MT27800 Family
(ConnectX-5) with Firmware version 16.19.148 and it does have hangs, but it
fails in the CPU mask:
mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit PCI DMA mask
mlx5_core 0000:01:00.0: Warning: couldn't set 64-bit consistent PCI DMA mask
mlx5_core 0000:01:00.0: firmware version: 16.19.148
mlx5_core 0000:01:00.0: Port module event: module 0, Cable unplugged
mlx5_core 0000:01:00.0: mlx5_irq_set_affinity_hint:628:(pid 1):
irq_set_affinity_hint failed,irq 0x0032
mlx5_core 0000:01:00.0: Failed to alloc affinity hint cpumask
mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -22
mlx5_core: probe of 0000:01:00.0 failed with error -22
Mine is a Mellanox MT28800 Family (ConnectX-5) with Firmware Version 16.19.21102.
Hopes it gives more data for analysis.
Thanks,
Joao
>
>> Thanks,
>> Joao
>>
>>
>>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-05-09 17:57 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-09 16:25 mlx5 endpoint driver problem Joao Pinto
2017-05-09 17:35 ` Saeed Mahameed
2017-05-09 17:38 ` Joao Pinto
2017-05-09 17:44 ` Saeed Mahameed
2017-05-09 17:57 ` Joao Pinto
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).