* mlx5 sub function issue
@ 2021-03-08 3:27 avis.wang
0 siblings, 0 replies; 8+ messages in thread
From: avis.wang @ 2021-03-08 3:27 UTC (permalink / raw)
To: parav, saeedm; +Cc: netdev
Hi all,
I have some confusion with mlx5 sub function feature. I tested it
with Mellanox ConnectX-6 Dx recently,
then I found the maximum amount of SF in each PF is just 128, I‘m not
sure if there is a problem with
my configuration, or there are some limits of SF in ConnectX-6.
Here is the environment:
system: CentOS Linux release 7.9.2009
NIC: Mellanox Technologies MT2892 Family
[ConnectX-6 Dx]
firmware-version: 22.29.2002 (MT_0000000430)
kernel: net-next master branch
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/
mlxconfig:
PF_BAR2_ENABLE True(1)
PF_BAR2_SIZE 8
PER_PF_NUM_SF True(1)
PF_TOTAL_SF 65535
PF_SF_BAR_SIZE 0
When I tried to create more SF:
~# devlink port add pci/0000:b3:00.0 flavour pcisf pfnum 0 sfnum 128
devlink answers: No space left on device
Is there anything missed in configuration? Can I create more sub
functions in ConnectX-6?
Trully yours,
avis
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: mlx5 sub function issue
2021-03-10 9:28 ` ze wang
@ 2021-03-11 6:13 ` Parav Pandit
0 siblings, 0 replies; 8+ messages in thread
From: Parav Pandit @ 2021-03-11 6:13 UTC (permalink / raw)
To: ze wang; +Cc: Saeed Mahameed, netdev
Hi Ze wang,
> From: ze wang <wangze712@gmail.com>
> Sent: Wednesday, March 10, 2021 2:59 PM
>
> Hi Parav,
> Thanks for your help. I did enabled VFs, and after turning off SR-IOV, it
> works.
Happy to hear that.
> Now the each PF can create 255 SFs, which is probably enough for us.
> Is it convenient
> to reveal how many SFs can be created at most?
>
Yes, it is. We will be doing this, mostly as devlink resource as discussed last time with Jacob, David, Samudrala in past.
It is in my todo list already. Until its available, please use the existing tooling.
> Ze Wang
>
> Parav Pandit <parav@nvidia.com> 于2021年3月9日周二 下午8:36写道:
> >
> > Hi Ze Wang,
> >
> > > From: ze wang <wangze712@gmail.com>
> > > Sent: Tuesday, March 9, 2021 8:34 AM
> > >
> > > Hi David,
> > > I can see that the variable settings are in effect:
> > > # mlxconfig -d b3:00.0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1
> > > PF_SF_BAR_SIZE=8 # mlxconfig -d b3:00.0 s PER_PF_NUM_SF=1
> > > PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8 # mlxconfig -d b3:00.1 s
> > > PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> > >
> > > after cold reboot:
> > > # mlxconfig -d b3:00.0 q|grep BAR
> > > PF_BAR2_ENABLE False(0)
> > > # mlxconfig -d b3:00.0 q|grep SF
> > > Description: ConnectX-6 Dx EN adapter card; 25GbE; Dual-port SFP28;
> > > PCIe 4.0 x8; Crypto and Secure Boot
> > > PER_PF_NUM_SF True(1)
> > > PF_TOTAL_SF 192
> > > PF_SF_BAR_SIZE 8
> > > # mlxconfig -d b3:00.1 q|grep SF
> > > Description: ConnectX-6 Dx EN adapter card; 25GbE; Dual-port SFP28;
> > > PCIe 4.0 x8; Crypto and Secure Boot
> > > PER_PF_NUM_SF True(1)
> > > PF_TOTAL_SF 192
> > > PF_SF_BAR_SIZE 8
> > >
> > > I tried to create as many SF as possible, then I found each PF can
> > > create up to
> > > 132 SFs. I want to confirm the maximum number of SFs that
> > > CX6 can create. If the mft version is correct, can I think that CX6
> > > can create up to 132 SFs per PF?
> > Do you have VFs enabled on the system? mlxconfig -d b3:00.0 q | grep
> > VF If so, please disable SRIOV.
> >
> > >
> > > David Ahern <dsahern@gmail.com> 于2021年3月8日周一 下午11:48写
> 道
> > > :
> > > >
> > > > On 3/8/21 12:21 AM, ze wang wrote:
> > > > > mlxconfig tool from mft tools version 4.16.52 or higher to set
> > > > > number of
> > > SF.
> > > > >
> > > > > mlxconfig -d b3:00.0 PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1
> > > > > PF_SF_BAR_SIZE=8 mlxconfig -d b3:00.0 PER_PF_NUM_SF=1
> > > > > PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8 mlxconfig -d b3:00.1
> > > > > PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> > > > >
> > > > > Cold reboot power cycle of the system as this changes the BAR
> > > > > size in device
> > > > >
> > > >
> > > > Is that capability going to be added to devlink?
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mlx5 sub function issue
2021-03-09 12:36 ` Parav Pandit
@ 2021-03-10 9:28 ` ze wang
2021-03-11 6:13 ` Parav Pandit
0 siblings, 1 reply; 8+ messages in thread
From: ze wang @ 2021-03-10 9:28 UTC (permalink / raw)
To: Parav Pandit; +Cc: Saeed Mahameed, netdev
Hi Parav,
Thanks for your help. I did enabled VFs, and after turning off
SR-IOV, it works.
Now the each PF can create 255 SFs, which is probably enough for us.
Is it convenient
to reveal how many SFs can be created at most?
Ze Wang
Parav Pandit <parav@nvidia.com> 于2021年3月9日周二 下午8:36写道:
>
> Hi Ze Wang,
>
> > From: ze wang <wangze712@gmail.com>
> > Sent: Tuesday, March 9, 2021 8:34 AM
> >
> > Hi David,
> > I can see that the variable settings are in effect:
> > # mlxconfig -d b3:00.0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1
> > PF_SF_BAR_SIZE=8 # mlxconfig -d b3:00.0 s PER_PF_NUM_SF=1
> > PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8 # mlxconfig -d b3:00.1 s
> > PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> >
> > after cold reboot:
> > # mlxconfig -d b3:00.0 q|grep BAR
> > PF_BAR2_ENABLE False(0)
> > # mlxconfig -d b3:00.0 q|grep SF
> > Description: ConnectX-6 Dx EN adapter card; 25GbE; Dual-port SFP28;
> > PCIe 4.0 x8; Crypto and Secure Boot
> > PER_PF_NUM_SF True(1)
> > PF_TOTAL_SF 192
> > PF_SF_BAR_SIZE 8
> > # mlxconfig -d b3:00.1 q|grep SF
> > Description: ConnectX-6 Dx EN adapter card; 25GbE; Dual-port SFP28;
> > PCIe 4.0 x8; Crypto and Secure Boot
> > PER_PF_NUM_SF True(1)
> > PF_TOTAL_SF 192
> > PF_SF_BAR_SIZE 8
> >
> > I tried to create as many SF as possible, then I found each PF can create up to
> > 132 SFs. I want to confirm the maximum number of SFs that
> > CX6 can create. If the mft version is correct, can I think that CX6 can create up
> > to 132 SFs per PF?
> Do you have VFs enabled on the system? mlxconfig -d b3:00.0 q | grep VF
> If so, please disable SRIOV.
>
> >
> > David Ahern <dsahern@gmail.com> 于2021年3月8日周一 下午11:48写道
> > :
> > >
> > > On 3/8/21 12:21 AM, ze wang wrote:
> > > > mlxconfig tool from mft tools version 4.16.52 or higher to set number of
> > SF.
> > > >
> > > > mlxconfig -d b3:00.0 PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1
> > > > PF_SF_BAR_SIZE=8 mlxconfig -d b3:00.0 PER_PF_NUM_SF=1
> > > > PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8 mlxconfig -d b3:00.1
> > > > PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> > > >
> > > > Cold reboot power cycle of the system as this changes the BAR size
> > > > in device
> > > >
> > >
> > > Is that capability going to be added to devlink?
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: mlx5 sub function issue
2021-03-09 3:04 ` ze wang
@ 2021-03-09 12:36 ` Parav Pandit
2021-03-10 9:28 ` ze wang
0 siblings, 1 reply; 8+ messages in thread
From: Parav Pandit @ 2021-03-09 12:36 UTC (permalink / raw)
To: ze wang, David Ahern; +Cc: Saeed Mahameed, netdev
Hi Ze Wang,
> From: ze wang <wangze712@gmail.com>
> Sent: Tuesday, March 9, 2021 8:34 AM
>
> Hi David,
> I can see that the variable settings are in effect:
> # mlxconfig -d b3:00.0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1
> PF_SF_BAR_SIZE=8 # mlxconfig -d b3:00.0 s PER_PF_NUM_SF=1
> PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8 # mlxconfig -d b3:00.1 s
> PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
>
> after cold reboot:
> # mlxconfig -d b3:00.0 q|grep BAR
> PF_BAR2_ENABLE False(0)
> # mlxconfig -d b3:00.0 q|grep SF
> Description: ConnectX-6 Dx EN adapter card; 25GbE; Dual-port SFP28;
> PCIe 4.0 x8; Crypto and Secure Boot
> PER_PF_NUM_SF True(1)
> PF_TOTAL_SF 192
> PF_SF_BAR_SIZE 8
> # mlxconfig -d b3:00.1 q|grep SF
> Description: ConnectX-6 Dx EN adapter card; 25GbE; Dual-port SFP28;
> PCIe 4.0 x8; Crypto and Secure Boot
> PER_PF_NUM_SF True(1)
> PF_TOTAL_SF 192
> PF_SF_BAR_SIZE 8
>
> I tried to create as many SF as possible, then I found each PF can create up to
> 132 SFs. I want to confirm the maximum number of SFs that
> CX6 can create. If the mft version is correct, can I think that CX6 can create up
> to 132 SFs per PF?
Do you have VFs enabled on the system? mlxconfig -d b3:00.0 q | grep VF
If so, please disable SRIOV.
>
> David Ahern <dsahern@gmail.com> 于2021年3月8日周一 下午11:48写道
> :
> >
> > On 3/8/21 12:21 AM, ze wang wrote:
> > > mlxconfig tool from mft tools version 4.16.52 or higher to set number of
> SF.
> > >
> > > mlxconfig -d b3:00.0 PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1
> > > PF_SF_BAR_SIZE=8 mlxconfig -d b3:00.0 PER_PF_NUM_SF=1
> > > PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8 mlxconfig -d b3:00.1
> > > PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> > >
> > > Cold reboot power cycle of the system as this changes the BAR size
> > > in device
> > >
> >
> > Is that capability going to be added to devlink?
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: mlx5 sub function issue
2021-03-08 15:48 ` David Ahern
2021-03-09 3:04 ` ze wang
@ 2021-03-09 12:33 ` Parav Pandit
1 sibling, 0 replies; 8+ messages in thread
From: Parav Pandit @ 2021-03-09 12:33 UTC (permalink / raw)
To: David Ahern, ze wang; +Cc: Saeed Mahameed, netdev
Hi David,
> From: David Ahern <dsahern@gmail.com>
> Sent: Monday, March 8, 2021 9:18 PM
>
> On 3/8/21 12:21 AM, ze wang wrote:
> > mlxconfig tool from mft tools version 4.16.52 or higher to set number of SF.
> >
> > mlxconfig -d b3:00.0 PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1
> > PF_SF_BAR_SIZE=8 mlxconfig -d b3:00.0 PER_PF_NUM_SF=1
> PF_TOTAL_SF=192
> > PF_SF_BAR_SIZE=8 mlxconfig -d b3:00.1 PER_PF_NUM_SF=1
> PF_TOTAL_SF=192
> > PF_SF_BAR_SIZE=8
> >
> > Cold reboot power cycle of the system as this changes the BAR size in
> > device
> >
>
> Is that capability going to be added to devlink?
Yes, want to expose as devlink resource at per controller level.
However devlink resource doesn’t have enough plumbing available to define controller, pf annotation which was the primary blocker for lacking this info right now.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mlx5 sub function issue
2021-03-08 15:48 ` David Ahern
@ 2021-03-09 3:04 ` ze wang
2021-03-09 12:36 ` Parav Pandit
2021-03-09 12:33 ` Parav Pandit
1 sibling, 1 reply; 8+ messages in thread
From: ze wang @ 2021-03-09 3:04 UTC (permalink / raw)
To: David Ahern; +Cc: Parav Pandit, Saeed Mahameed, netdev
Hi David,
I can see that the variable settings are in effect:
# mlxconfig -d b3:00.0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_SF_BAR_SIZE=8
# mlxconfig -d b3:00.0 s PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
# mlxconfig -d b3:00.1 s PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
after cold reboot:
# mlxconfig -d b3:00.0 q|grep BAR
PF_BAR2_ENABLE False(0)
# mlxconfig -d b3:00.0 q|grep SF
Description: ConnectX-6 Dx EN adapter card; 25GbE; Dual-port SFP28;
PCIe 4.0 x8; Crypto and Secure Boot
PER_PF_NUM_SF True(1)
PF_TOTAL_SF 192
PF_SF_BAR_SIZE 8
# mlxconfig -d b3:00.1 q|grep SF
Description: ConnectX-6 Dx EN adapter card; 25GbE; Dual-port SFP28;
PCIe 4.0 x8; Crypto and Secure Boot
PER_PF_NUM_SF True(1)
PF_TOTAL_SF 192
PF_SF_BAR_SIZE 8
I tried to create as many SF as possible, then I found each PF can
create up to 132 SFs. I want to confirm the maximum number of SFs that
CX6 can create. If the mft version is correct, can I think that CX6
can create up to 132 SFs per PF?
David Ahern <dsahern@gmail.com> 于2021年3月8日周一 下午11:48写道:
>
> On 3/8/21 12:21 AM, ze wang wrote:
> > mlxconfig tool from mft tools version 4.16.52 or higher to set number of SF.
> >
> > mlxconfig -d b3:00.0 PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_SF_BAR_SIZE=8
> > mlxconfig -d b3:00.0 PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> > mlxconfig -d b3:00.1 PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> >
> > Cold reboot power cycle of the system as this changes the BAR size in device
> >
>
> Is that capability going to be added to devlink?
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mlx5 sub function issue
2021-03-08 7:21 ` ze wang
@ 2021-03-08 15:48 ` David Ahern
2021-03-09 3:04 ` ze wang
2021-03-09 12:33 ` Parav Pandit
0 siblings, 2 replies; 8+ messages in thread
From: David Ahern @ 2021-03-08 15:48 UTC (permalink / raw)
To: ze wang, Parav Pandit; +Cc: Saeed Mahameed, netdev
On 3/8/21 12:21 AM, ze wang wrote:
> mlxconfig tool from mft tools version 4.16.52 or higher to set number of SF.
>
> mlxconfig -d b3:00.0 PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_SF_BAR_SIZE=8
> mlxconfig -d b3:00.0 PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> mlxconfig -d b3:00.1 PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
>
> Cold reboot power cycle of the system as this changes the BAR size in device
>
Is that capability going to be added to devlink?
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mlx5 sub function issue
[not found] ` <BY5PR12MB43226FF17791F6365812D028DC939@BY5PR12MB4322.namprd12.prod.outlook.com>
@ 2021-03-08 7:21 ` ze wang
2021-03-08 15:48 ` David Ahern
0 siblings, 1 reply; 8+ messages in thread
From: ze wang @ 2021-03-08 7:21 UTC (permalink / raw)
To: Parav Pandit; +Cc: Saeed Mahameed, netdev
Hi Parav,
Thanks for the answer.
There are more than two hundred CPUs on our machine, we're planning to
put SFs into the containers, This may requires allocate one SF per
CPU,so we want to know the upper limit of creating SFs.
By the way, the latest version of the MFT download center is 4.16.1-9,
does this meet the 4.16.52 version requirements? If not satisfied,
where can I get mft-4.16.52?
Ze Wang
Parav Pandit <parav@nvidia.com> 於 2021年3月8日週一 下午1:17寫道:
>
> Hi Ze Wang,
>
>
>
> How many SFs do you prefer to create?
> What applications would you like to run on more than 128 sfs?
> How many CPUs do you have in system which will use this many sfs?
>
>
>
> You need to use following.
>
> mlxconfig tool from mft tools version 4.16.52 or higher to set number of SF.
>
> mlxconfig -d b3:00.0 PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_SF_BAR_SIZE=8
> mlxconfig -d b3:00.0 PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
> mlxconfig -d b3:00.1 PER_PF_NUM_SF=1 PF_TOTAL_SF=192 PF_SF_BAR_SIZE=8
>
> Cold reboot power cycle of the system as this changes the BAR size in device
>
>
>
> Parav
>
>
>
> From: ze wang <wangze712@gmail.com>
> Sent: Monday, March 8, 2021 7:22 AM
> To: Saeed Mahameed <saeedm@nvidia.com>; Parav Pandit <parav@nvidia.com>
> Subject: mlx5 sub function issue
>
>
>
> Hi all,
> I have some confusion with mlx5 sub function feature. I tested it with Mellanox ConnectX-6 Dx recently, then I
>
> found the maximum amount of SF in each PF is just 128,I‘m not sure if there is a problem with my configuration,
>
> or there are some limits of SF in ConnectX-6.
> Here is the environment:
> system: CentOS Linux release 7.9.2009
> NIC: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
> firmware-version: 22.29.2002 (MT_0000000430)
> kernel: net-next master branch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/
> mlxconfig:
> PF_BAR2_ENABLE True(1)
> PF_BAR2_SIZE 8
> PER_PF_NUM_SF True(1)
> PF_TOTAL_SF 65535
> PF_SF_BAR_SIZE 0
>
> When I tried to create more SF:
>
> ~# devlink port add pci/0000:b3:00.0 flavour pcisf pfnum 0 sfnum 128
> devlink answers: No space left on device
>
> Is there anything missed in configuration? Can I create more sub functions in ConnectX-6?
>
> Trully yours,
> wangze
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-03-11 6:14 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-08 3:27 mlx5 sub function issue avis.wang
[not found] <CANS1P8H8sDGUzQEh_LEFVi=6tUZzVxAty9_OKWAs4CU67wdLeg@mail.gmail.com>
[not found] ` <BY5PR12MB43226FF17791F6365812D028DC939@BY5PR12MB4322.namprd12.prod.outlook.com>
2021-03-08 7:21 ` ze wang
2021-03-08 15:48 ` David Ahern
2021-03-09 3:04 ` ze wang
2021-03-09 12:36 ` Parav Pandit
2021-03-10 9:28 ` ze wang
2021-03-11 6:13 ` Parav Pandit
2021-03-09 12:33 ` Parav Pandit
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).