Vmxnet3 activation of device fails in DPDK1.7

Vmxnet3 activation of device fails in DPDK1.7

[Dey, Souvik]

Hi,
                In DPDK 1.7 , while using the vmxnet3 pmd on vmware Esxi 5.5 update 3 we are seeing that activation of the device fails.

status = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD); return a non zero status. Though the normal vmxnet3.ko works fine in the same system. Any idea if anyone has faced this type of issue.

--
Regards,
Souvik

Re: Vmxnet3 activation of device fails in DPDK1.7

[Stephen Hemminger]

On Thu, 10 Dec 2015 10:22:30 +0000
"Dey, Souvik" <sodey@sonusnet.com> wrote:

> Hi,
>                 In DPDK 1.7 , while using the vmxnet3 pmd on vmware Esxi 5.5 update 3 we are seeing that activation of the device fails.

Please try something more recent. DPDK 1.7 had a version of vmxnet3
which was brand new at the time, and had lots of bugs. A lot of collabrative
changes went into later versions. I would recommend 2.0 or later
if possible.

Re: Vmxnet3 activation of device fails in DPDK1.7

[Dey, Souvik]

I have also tried with DPDK 2.1 but the result is same. 

-----Original Message-----
From: Stephen Hemminger [mailto:stephen@networkplumber.org] 
Sent: Thursday, December 10, 2015 10:54 PM
To: Dey, Souvik <sodey@sonusnet.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Vmxnet3 activation of device fails in DPDK1.7

On Thu, 10 Dec 2015 10:22:30 +0000
"Dey, Souvik" <sodey@sonusnet.com> wrote:

> Hi,
>                 In DPDK 1.7 , while using the vmxnet3 pmd on vmware Esxi 5.5 update 3 we are seeing that activation of the device fails.

Please try something more recent. DPDK 1.7 had a version of vmxnet3 which was brand new at the time, and had lots of bugs. A lot of collabrative changes went into later versions. I would recommend 2.0 or later if possible.

Re: Vmxnet3 activation of device fails in DPDK1.7

[Yong Wang]

I checked with our engineers internally and it turns out that the changes to lower the max ring2 size never got checked-in in 6.0. This should explain why it works on ESXi6.0.  And for 5.5, the plan is to revert the changes in 55p08 patch to bring it back to 4096.




On 12/13/15, 11:06 PM, "Dey, Souvik" <sodey@sonusnet.com> wrote:
>Thanks for the update. Yes I tried with both 1024 and 2048 it worked fine. But interestingly 4096 is working fine in ESXi6.0. Are you aware of some configuration in ESXi which can solve it issue or bring back nb_rx_desc to max of 4096 ?. Or any idea of any fixes from vmware which fixes this issue ?
>
>-----Original Message-----
>From: Yong Wang [mailto:yongwang@vmware.com] 
>Sent: Monday, December 14, 2015 11:57 AM
>To: Dey, Souvik <sodey@sonusnet.com>; dev@dpdk.org
>Subject: Re: [dpdk-dev] Vmxnet3 activation of device fails in DPDK1.7
>
>nb_rx_desc should be less or equal to 2048 in update 3 due to a change in vmxnet3 backend. More details can be found at http://kb.vmware.com/kb/2136932.
>
>
>On 12/13/15, 9:01 PM, "Dey, Souvik" <sodey@sonusnet.com> wrote:
>
>>Not sure about but it definitely worked in ESXi5.5 Update1, and also prior versions. Also on ESXi6.0 it works fine. 
>>We are using 1rx queue and 8 tx queues per port. The nb_rx_desc  is set to 4096  and the each mbuf size is set to 2048. The rx_conf struct has the following values during the init time .
>>
>>static const struct rte_eth_rxconf rx_conf = {
>>   .rx_thresh = {
>>      .pthresh = RX_PTHRESH,
>>      .hthresh = RX_HTHRESH,
>>      .wthresh = RX_WTHRESH,
>>   },
>>};
>>
>>Do you suspect anything wrong in this ?
>>
>>-----Original Message-----
>>From: Yong Wang [mailto:yongwang@vmware.com] 
>>Sent: Friday, December 11, 2015 12:07 AM
>>To: Dey, Souvik <sodey@sonusnet.com>; dev@dpdk.org
>>Subject: Re: [dpdk-dev] Vmxnet3 activation of device fails in DPDK1.7
>>
>>On 12/10/15, 2:22 AM, "dev on behalf of Dey, Souvik" <dev-bounces@dpdk.org on behalf of sodey@sonusnet.com> wrote:
>>
>>
>>
>>>Hi,
>>>                In DPDK 1.7 , while using the vmxnet3 pmd on vmware Esxi 5.5 update 3 we are seeing that activation of the device fails.
>>>
>>>status = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD); return a non zero status. Though the normal vmxnet3.ko works fine in the same system. Any idea if anyone has faced this type of issue.
>>>
>>>--
>>>Regards,
>>>Souvik
>>
>>Did it work for ESXi 5.5 update 2 or some earlier version?
>>
>>Can you also post the rx ring size you used to config the rx queue?
>>In update 3, there are some changes to the max allowed rx ring size for ring1. Even ring1 is not used yet in the pmd, the setup code sets ring1’s size the same as ring0’s.

Re: Vmxnet3 activation of device fails in DPDK1.7

[Dey, Souvik]

Thanks for the update. Yes I tried with both 1024 and 2048 it worked fine. But interestingly 4096 is working fine in ESXi6.0. Are you aware of some configuration in ESXi which can solve it issue or bring back nb_rx_desc to max of 4096 ?. Or any idea of any fixes from vmware which fixes this issue ?

-----Original Message-----
From: Yong Wang [mailto:yongwang@vmware.com] 
Sent: Monday, December 14, 2015 11:57 AM
To: Dey, Souvik <sodey@sonusnet.com>; dev@dpdk.org
Subject: Re: [dpdk-dev] Vmxnet3 activation of device fails in DPDK1.7

nb_rx_desc should be less or equal to 2048 in update 3 due to a change in vmxnet3 backend. More details can be found at http://kb.vmware.com/kb/2136932.


On 12/13/15, 9:01 PM, "Dey, Souvik" <sodey@sonusnet.com> wrote:

>Not sure about but it definitely worked in ESXi5.5 Update1, and also prior versions. Also on ESXi6.0 it works fine. 
>We are using 1rx queue and 8 tx queues per port. The nb_rx_desc  is set to 4096  and the each mbuf size is set to 2048. The rx_conf struct has the following values during the init time .
>
>static const struct rte_eth_rxconf rx_conf = {
>   .rx_thresh = {
>      .pthresh = RX_PTHRESH,
>      .hthresh = RX_HTHRESH,
>      .wthresh = RX_WTHRESH,
>   },
>};
>
>Do you suspect anything wrong in this ?
>
>-----Original Message-----
>From: Yong Wang [mailto:yongwang@vmware.com] 
>Sent: Friday, December 11, 2015 12:07 AM
>To: Dey, Souvik <sodey@sonusnet.com>; dev@dpdk.org
>Subject: Re: [dpdk-dev] Vmxnet3 activation of device fails in DPDK1.7
>
>On 12/10/15, 2:22 AM, "dev on behalf of Dey, Souvik" <dev-bounces@dpdk.org on behalf of sodey@sonusnet.com> wrote:
>
>
>
>>Hi,
>>                In DPDK 1.7 , while using the vmxnet3 pmd on vmware Esxi 5.5 update 3 we are seeing that activation of the device fails.
>>
>>status = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD); return a non zero status. Though the normal vmxnet3.ko works fine in the same system. Any idea if anyone has faced this type of issue.
>>
>>--
>>Regards,
>>Souvik
>
>Did it work for ESXi 5.5 update 2 or some earlier version?
>
>Can you also post the rx ring size you used to config the rx queue?
>In update 3, there are some changes to the max allowed rx ring size for ring1. Even ring1 is not used yet in the pmd, the setup code sets ring1’s size the same as ring0’s.

Re: Vmxnet3 activation of device fails in DPDK1.7

[Yong Wang]

nb_rx_desc should be less or equal to 2048 in update 3 due to a change in vmxnet3 backend. More details can be found at http://kb.vmware.com/kb/2136932.


On 12/13/15, 9:01 PM, "Dey, Souvik" <sodey@sonusnet.com> wrote:

>Not sure about but it definitely worked in ESXi5.5 Update1, and also prior versions. Also on ESXi6.0 it works fine. 
>We are using 1rx queue and 8 tx queues per port. The nb_rx_desc  is set to 4096  and the each mbuf size is set to 2048. The rx_conf struct has the following values during the init time .
>
>static const struct rte_eth_rxconf rx_conf = {
>   .rx_thresh = {
>      .pthresh = RX_PTHRESH,
>      .hthresh = RX_HTHRESH,
>      .wthresh = RX_WTHRESH,
>   },
>};
>
>Do you suspect anything wrong in this ?
>
>-----Original Message-----
>From: Yong Wang [mailto:yongwang@vmware.com] 
>Sent: Friday, December 11, 2015 12:07 AM
>To: Dey, Souvik <sodey@sonusnet.com>; dev@dpdk.org
>Subject: Re: [dpdk-dev] Vmxnet3 activation of device fails in DPDK1.7
>
>On 12/10/15, 2:22 AM, "dev on behalf of Dey, Souvik" <dev-bounces@dpdk.org on behalf of sodey@sonusnet.com> wrote:
>
>
>
>>Hi,
>>                In DPDK 1.7 , while using the vmxnet3 pmd on vmware Esxi 5.5 update 3 we are seeing that activation of the device fails.
>>
>>status = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD); return a non zero status. Though the normal vmxnet3.ko works fine in the same system. Any idea if anyone has faced this type of issue.
>>
>>--
>>Regards,
>>Souvik
>
>Did it work for ESXi 5.5 update 2 or some earlier version?
>
>Can you also post the rx ring size you used to config the rx queue?
>In update 3, there are some changes to the max allowed rx ring size for ring1. Even ring1 is not used yet in the pmd, the setup code sets ring1’s size the same as ring0’s.

Re: Vmxnet3 activation of device fails in DPDK1.7

[Dey, Souvik]

Not sure about but it definitely worked in ESXi5.5 Update1, and also prior versions. Also on ESXi6.0 it works fine. 
We are using 1rx queue and 8 tx queues per port. The nb_rx_desc  is set to 4096  and the each mbuf size is set to 2048. The rx_conf struct has the following values during the init time .

static const struct rte_eth_rxconf rx_conf = {
   .rx_thresh = {
      .pthresh = RX_PTHRESH,
      .hthresh = RX_HTHRESH,
      .wthresh = RX_WTHRESH,
   },
};

Do you suspect anything wrong in this ?

-----Original Message-----
From: Yong Wang [mailto:yongwang@vmware.com] 
Sent: Friday, December 11, 2015 12:07 AM
To: Dey, Souvik <sodey@sonusnet.com>; dev@dpdk.org
Subject: Re: [dpdk-dev] Vmxnet3 activation of device fails in DPDK1.7

On 12/10/15, 2:22 AM, "dev on behalf of Dey, Souvik" <dev-bounces@dpdk.org on behalf of sodey@sonusnet.com> wrote:



>Hi,
>                In DPDK 1.7 , while using the vmxnet3 pmd on vmware Esxi 5.5 update 3 we are seeing that activation of the device fails.
>
>status = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD); return a non zero status. Though the normal vmxnet3.ko works fine in the same system. Any idea if anyone has faced this type of issue.
>
>--
>Regards,
>Souvik

Did it work for ESXi 5.5 update 2 or some earlier version?

Can you also post the rx ring size you used to config the rx queue?
In update 3, there are some changes to the max allowed rx ring size for ring1. Even ring1 is not used yet in the pmd, the setup code sets ring1’s size the same as ring0’s.

Re: Vmxnet3 activation of device fails in DPDK1.7

[Yong Wang]

On 12/10/15, 2:22 AM, "dev on behalf of Dey, Souvik" <dev-bounces@dpdk.org on behalf of sodey@sonusnet.com> wrote:



>Hi,
>                In DPDK 1.7 , while using the vmxnet3 pmd on vmware Esxi 5.5 update 3 we are seeing that activation of the device fails.
>
>status = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD); return a non zero status. Though the normal vmxnet3.ko works fine in the same system. Any idea if anyone has faced this type of issue.
>
>--
>Regards,
>Souvik

Did it work for ESXi 5.5 update 2 or some earlier version?

Can you also post the rx ring size you used to config the rx queue?
In update 3, there are some changes to the max allowed rx ring size
for ring1. Even ring1 is not used yet in the pmd, the setup code
sets ring1’s size the same as ring0’s.