[PATCH] net/vmxnet3: fix queue size changes

[PATCH] net/vmxnet3: fix queue size changes

[Charles (Chas) Williams]

If the user reconfigures the queues size, then the previosly allocated
memzone may potentially be too small.  Instead, always free the old
memzone and allocate a new one.

Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver implementation")

Signed-off-by: Chas Williams <ciwillia@brocade.com>
---
 drivers/net/vmxnet3/vmxnet3_rxtx.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c b/drivers/net/vmxnet3/vmxnet3_rxtx.c
index 6649c3f..104e040 100644
--- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
+++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
@@ -893,8 +893,8 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 
 /*
  * Create memzone for device rings. malloc can't be used as the physical address is
- * needed. If the memzone is already created, then this function returns a ptr
- * to the old one.
+ * needed. If the memzone already exists, we free it since it may have been created
+ * with a different size.
  */
 static const struct rte_memzone *
 ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
@@ -909,7 +909,7 @@ ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
 
 	mz = rte_memzone_lookup(z_name);
 	if (mz)
-		return mz;
+		rte_memzone_free(mz);
 
 	return rte_memzone_reserve_aligned(z_name, ring_size,
 					   socket_id, 0, VMXNET3_RING_BA_ALIGN);
-- 
2.1.4

Re: [PATCH] net/vmxnet3: fix queue size changes

[Jan Blunck]

On Mon, Mar 13, 2017 at 11:41 PM, Charles (Chas) Williams
<ciwillia@brocade.com> wrote:
> If the user reconfigures the queues size, then the previosly allocated
> memzone may potentially be too small.  Instead, always free the old
> memzone and allocate a new one.
>
> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver implementation")
>
> Signed-off-by: Chas Williams <ciwillia@brocade.com>
> ---
>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c b/drivers/net/vmxnet3/vmxnet3_rxtx.c
> index 6649c3f..104e040 100644
> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
> @@ -893,8 +893,8 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
>
>  /*
>   * Create memzone for device rings. malloc can't be used as the physical address is
> - * needed. If the memzone is already created, then this function returns a ptr
> - * to the old one.
> + * needed. If the memzone already exists, we free it since it may have been created
> + * with a different size.
>   */
>  static const struct rte_memzone *
>  ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
> @@ -909,7 +909,7 @@ ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
>
>         mz = rte_memzone_lookup(z_name);
>         if (mz)
> -               return mz;
> +               rte_memzone_free(mz);
>
>         return rte_memzone_reserve_aligned(z_name, ring_size,
>                                            socket_id, 0, VMXNET3_RING_BA_ALIGN);

Chas,

Thanks for hunting this one down. Wouldn't the rte_memzone_free()
better fit into vmxnet3_cmd_ring_release() ?

Also the ring_dma_zone_reserve() could get replaced by
rte_eth_dma_zone_reserve() (see also
http://dpdk.org/dev/patchwork/patch/21432/).

Re: [PATCH] net/vmxnet3: fix queue size changes

[Charles (Chas) Williams]

On 03/14/2017 12:11 PM, Jan Blunck wrote:
> On Mon, Mar 13, 2017 at 11:41 PM, Charles (Chas) Williams
> <ciwillia@brocade.com> wrote:
>> If the user reconfigures the queues size, then the previosly allocated
>> memzone may potentially be too small.  Instead, always free the old
>> memzone and allocate a new one.
>>
>> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver implementation")
>>
>> Signed-off-by: Chas Williams <ciwillia@brocade.com>
>> ---
>>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>> index 6649c3f..104e040 100644
>> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>> @@ -893,8 +893,8 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
>>
>>  /*
>>   * Create memzone for device rings. malloc can't be used as the physical address is
>> - * needed. If the memzone is already created, then this function returns a ptr
>> - * to the old one.
>> + * needed. If the memzone already exists, we free it since it may have been created
>> + * with a different size.
>>   */
>>  static const struct rte_memzone *
>>  ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
>> @@ -909,7 +909,7 @@ ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
>>
>>         mz = rte_memzone_lookup(z_name);
>>         if (mz)
>> -               return mz;
>> +               rte_memzone_free(mz);
>>
>>         return rte_memzone_reserve_aligned(z_name, ring_size,
>>                                            socket_id, 0, VMXNET3_RING_BA_ALIGN);
>
> Chas,
>
> Thanks for hunting this one down. Wouldn't the rte_memzone_free()
> better fit into vmxnet3_cmd_ring_release() ?

I don't care which way it goes.  I just did what is basically done in
gpa_zone_reserve() to match the "style".  Tracking the current ring size
and avoiding reallocating a potentially large chunk of memory seems like
a better idea.

> Also the ring_dma_zone_reserve() could get replaced by
> rte_eth_dma_zone_reserve() (see also

Yes, it probably should get changed to that along with tracking the size.

Re: [PATCH] net/vmxnet3: fix queue size changes

[Jan Blunck]

On Tue, Mar 14, 2017 at 5:38 PM, Charles (Chas) Williams
<ciwillia@brocade.com> wrote:
>
>
> On 03/14/2017 12:11 PM, Jan Blunck wrote:
>>
>> On Mon, Mar 13, 2017 at 11:41 PM, Charles (Chas) Williams
>> <ciwillia@brocade.com> wrote:
>>>
>>> If the user reconfigures the queues size, then the previosly allocated
>>> memzone may potentially be too small.  Instead, always free the old
>>> memzone and allocate a new one.
>>>
>>> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver
>>> implementation")
>>>
>>> Signed-off-by: Chas Williams <ciwillia@brocade.com>
>>> ---
>>>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 6 +++---
>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>> b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>> index 6649c3f..104e040 100644
>>> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>> @@ -893,8 +893,8 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf
>>> **rx_pkts, uint16_t nb_pkts)
>>>
>>>  /*
>>>   * Create memzone for device rings. malloc can't be used as the physical
>>> address is
>>> - * needed. If the memzone is already created, then this function returns
>>> a ptr
>>> - * to the old one.
>>> + * needed. If the memzone already exists, we free it since it may have
>>> been created
>>> + * with a different size.
>>>   */
>>>  static const struct rte_memzone *
>>>  ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
>>> @@ -909,7 +909,7 @@ ring_dma_zone_reserve(struct rte_eth_dev *dev, const
>>> char *ring_name,
>>>
>>>         mz = rte_memzone_lookup(z_name);
>>>         if (mz)
>>> -               return mz;
>>> +               rte_memzone_free(mz);
>>>
>>>         return rte_memzone_reserve_aligned(z_name, ring_size,
>>>                                            socket_id, 0,
>>> VMXNET3_RING_BA_ALIGN);
>>
>>
>> Chas,
>>
>> Thanks for hunting this one down. Wouldn't the rte_memzone_free()
>> better fit into vmxnet3_cmd_ring_release() ?
>
>
> I don't care which way it goes.  I just did what is basically done in
> gpa_zone_reserve() to match the "style".  Tracking the current ring size
> and avoiding reallocating a potentially large chunk of memory seems like
> a better idea.
>
>> Also the ring_dma_zone_reserve() could get replaced by
>> rte_eth_dma_zone_reserve() (see also
>
>
> Yes, it probably should get changed to that along with tracking the size.

Why don't we always allocate VMXNET3_RX_RING_MAX_SIZE entries? That
way we don't need to reallocate on a later queue setup change?

Re: [PATCH] net/vmxnet3: fix queue size changes

[Charles (Chas) Williams]

On 03/15/2017 04:18 AM, Jan Blunck wrote:
> On Tue, Mar 14, 2017 at 5:38 PM, Charles (Chas) Williams
> <ciwillia@brocade.com> wrote:
>>
>>
>> On 03/14/2017 12:11 PM, Jan Blunck wrote:
>>>
>>> On Mon, Mar 13, 2017 at 11:41 PM, Charles (Chas) Williams
>>> <ciwillia@brocade.com> wrote:
>>>>
>>>> If the user reconfigures the queues size, then the previosly allocated
>>>> memzone may potentially be too small.  Instead, always free the old
>>>> memzone and allocate a new one.
>>>>
>>>> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver
>>>> implementation")
>>>>
>>>> Signed-off-by: Chas Williams <ciwillia@brocade.com>
>>>> ---
>>>>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 6 +++---
>>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>> b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>> index 6649c3f..104e040 100644
>>>> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>> @@ -893,8 +893,8 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf
>>>> **rx_pkts, uint16_t nb_pkts)
>>>>
>>>>  /*
>>>>   * Create memzone for device rings. malloc can't be used as the physical
>>>> address is
>>>> - * needed. If the memzone is already created, then this function returns
>>>> a ptr
>>>> - * to the old one.
>>>> + * needed. If the memzone already exists, we free it since it may have
>>>> been created
>>>> + * with a different size.
>>>>   */
>>>>  static const struct rte_memzone *
>>>>  ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
>>>> @@ -909,7 +909,7 @@ ring_dma_zone_reserve(struct rte_eth_dev *dev, const
>>>> char *ring_name,
>>>>
>>>>         mz = rte_memzone_lookup(z_name);
>>>>         if (mz)
>>>> -               return mz;
>>>> +               rte_memzone_free(mz);
>>>>
>>>>         return rte_memzone_reserve_aligned(z_name, ring_size,
>>>>                                            socket_id, 0,
>>>> VMXNET3_RING_BA_ALIGN);
>>>
>>>
>>> Chas,
>>>
>>> Thanks for hunting this one down. Wouldn't the rte_memzone_free()
>>> better fit into vmxnet3_cmd_ring_release() ?
>>
>>
>> I don't care which way it goes.  I just did what is basically done in
>> gpa_zone_reserve() to match the "style".  Tracking the current ring size
>> and avoiding reallocating a potentially large chunk of memory seems like
>> a better idea.
>>
>>> Also the ring_dma_zone_reserve() could get replaced by
>>> rte_eth_dma_zone_reserve() (see also
>>
>>
>> Yes, it probably should get changed to that along with tracking the size.
>
> Why don't we always allocate VMXNET3_RX_RING_MAX_SIZE entries? That
> way we don't need to reallocate on a later queue setup change?

That's using more memory than it needs to use.  It might break someone's application
that already runs in some tightly constrained machine.  Failing to shrink the memzone
isn't likely to break anything since they have (apparently) already dealt with having
less memory available before switching to a smaller queue size.

Still, it can be a matter for another day.

Re: [PATCH] net/vmxnet3: fix queue size changes

[Jan Blunck]

On Wed, Mar 15, 2017 at 10:45 AM, Charles (Chas) Williams
<ciwillia@brocade.com> wrote:
>
>
> On 03/15/2017 04:18 AM, Jan Blunck wrote:
>>
>> On Tue, Mar 14, 2017 at 5:38 PM, Charles (Chas) Williams
>> <ciwillia@brocade.com> wrote:
>>>
>>>
>>>
>>> On 03/14/2017 12:11 PM, Jan Blunck wrote:
>>>>
>>>>
>>>> On Mon, Mar 13, 2017 at 11:41 PM, Charles (Chas) Williams
>>>> <ciwillia@brocade.com> wrote:
>>>>>
>>>>>
>>>>> If the user reconfigures the queues size, then the previosly allocated
>>>>> memzone may potentially be too small.  Instead, always free the old
>>>>> memzone and allocate a new one.
>>>>>
>>>>> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver
>>>>> implementation")
>>>>>
>>>>> Signed-off-by: Chas Williams <ciwillia@brocade.com>
>>>>> ---
>>>>>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 6 +++---
>>>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>> b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>> index 6649c3f..104e040 100644
>>>>> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>> @@ -893,8 +893,8 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf
>>>>> **rx_pkts, uint16_t nb_pkts)
>>>>>
>>>>>  /*
>>>>>   * Create memzone for device rings. malloc can't be used as the
>>>>> physical
>>>>> address is
>>>>> - * needed. If the memzone is already created, then this function
>>>>> returns
>>>>> a ptr
>>>>> - * to the old one.
>>>>> + * needed. If the memzone already exists, we free it since it may have
>>>>> been created
>>>>> + * with a different size.
>>>>>   */
>>>>>  static const struct rte_memzone *
>>>>>  ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
>>>>> @@ -909,7 +909,7 @@ ring_dma_zone_reserve(struct rte_eth_dev *dev,
>>>>> const
>>>>> char *ring_name,
>>>>>
>>>>>         mz = rte_memzone_lookup(z_name);
>>>>>         if (mz)
>>>>> -               return mz;
>>>>> +               rte_memzone_free(mz);
>>>>>
>>>>>         return rte_memzone_reserve_aligned(z_name, ring_size,
>>>>>                                            socket_id, 0,
>>>>> VMXNET3_RING_BA_ALIGN);
>>>>
>>>>
>>>>
>>>> Chas,
>>>>
>>>> Thanks for hunting this one down. Wouldn't the rte_memzone_free()
>>>> better fit into vmxnet3_cmd_ring_release() ?
>>>
>>>
>>>
>>> I don't care which way it goes.  I just did what is basically done in
>>> gpa_zone_reserve() to match the "style".  Tracking the current ring size
>>> and avoiding reallocating a potentially large chunk of memory seems like
>>> a better idea.
>>>
>>>> Also the ring_dma_zone_reserve() could get replaced by
>>>> rte_eth_dma_zone_reserve() (see also
>>>
>>>
>>>
>>> Yes, it probably should get changed to that along with tracking the size.
>>
>>
>> Why don't we always allocate VMXNET3_RX_RING_MAX_SIZE entries? That
>> way we don't need to reallocate on a later queue setup change?
>
>
> That's using more memory than it needs to use.  It might break someone's
> application
> that already runs in some tightly constrained machine.  Failing to shrink
> the memzone
> isn't likely to break anything since they have (apparently) already dealt
> with having
> less memory available before switching to a smaller queue size.
>
> Still, it can be a matter for another day.
>

Other drivers (ixgbe, e1000, ...) always allocate based on the max
ring size too. Since the VMXNET3_RX_RING_MAX_SIZE is 4096 I don't
think it is a huge waste of memory.

Re: [PATCH] net/vmxnet3: fix queue size changes

[Charles (Chas) Williams]

On 03/15/2017 06:05 AM, Jan Blunck wrote:
> On Wed, Mar 15, 2017 at 10:45 AM, Charles (Chas) Williams
> <ciwillia@brocade.com> wrote:
>>
>>
>> On 03/15/2017 04:18 AM, Jan Blunck wrote:
>>>
>>> On Tue, Mar 14, 2017 at 5:38 PM, Charles (Chas) Williams
>>> <ciwillia@brocade.com> wrote:
>>>>
>>>>
>>>>
>>>> On 03/14/2017 12:11 PM, Jan Blunck wrote:
>>>>>
>>>>>
>>>>> On Mon, Mar 13, 2017 at 11:41 PM, Charles (Chas) Williams
>>>>> <ciwillia@brocade.com> wrote:
>>>>>>
>>>>>>
>>>>>> If the user reconfigures the queues size, then the previosly allocated
>>>>>> memzone may potentially be too small.  Instead, always free the old
>>>>>> memzone and allocate a new one.
>>>>>>
>>>>>> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver
>>>>>> implementation")
>>>>>>
>>>>>> Signed-off-by: Chas Williams <ciwillia@brocade.com>
>>>>>> ---
>>>>>>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 6 +++---
>>>>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>>> b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>>> index 6649c3f..104e040 100644
>>>>>> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>>> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>>> @@ -893,8 +893,8 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf
>>>>>> **rx_pkts, uint16_t nb_pkts)
>>>>>>
>>>>>>  /*
>>>>>>   * Create memzone for device rings. malloc can't be used as the
>>>>>> physical
>>>>>> address is
>>>>>> - * needed. If the memzone is already created, then this function
>>>>>> returns
>>>>>> a ptr
>>>>>> - * to the old one.
>>>>>> + * needed. If the memzone already exists, we free it since it may have
>>>>>> been created
>>>>>> + * with a different size.
>>>>>>   */
>>>>>>  static const struct rte_memzone *
>>>>>>  ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
>>>>>> @@ -909,7 +909,7 @@ ring_dma_zone_reserve(struct rte_eth_dev *dev,
>>>>>> const
>>>>>> char *ring_name,
>>>>>>
>>>>>>         mz = rte_memzone_lookup(z_name);
>>>>>>         if (mz)
>>>>>> -               return mz;
>>>>>> +               rte_memzone_free(mz);
>>>>>>
>>>>>>         return rte_memzone_reserve_aligned(z_name, ring_size,
>>>>>>                                            socket_id, 0,
>>>>>> VMXNET3_RING_BA_ALIGN);
>>>>>
>>>>>
>>>>>
>>>>> Chas,
>>>>>
>>>>> Thanks for hunting this one down. Wouldn't the rte_memzone_free()
>>>>> better fit into vmxnet3_cmd_ring_release() ?
>>>>
>>>>
>>>>
>>>> I don't care which way it goes.  I just did what is basically done in
>>>> gpa_zone_reserve() to match the "style".  Tracking the current ring size
>>>> and avoiding reallocating a potentially large chunk of memory seems like
>>>> a better idea.
>>>>
>>>>> Also the ring_dma_zone_reserve() could get replaced by
>>>>> rte_eth_dma_zone_reserve() (see also
>>>>
>>>>
>>>>
>>>> Yes, it probably should get changed to that along with tracking the size.
>>>
>>>
>>> Why don't we always allocate VMXNET3_RX_RING_MAX_SIZE entries? That
>>> way we don't need to reallocate on a later queue setup change?
>>
>>
>> That's using more memory than it needs to use.  It might break someone's
>> application
>> that already runs in some tightly constrained machine.  Failing to shrink
>> the memzone
>> isn't likely to break anything since they have (apparently) already dealt
>> with having
>> less memory available before switching to a smaller queue size.
>>
>> Still, it can be a matter for another day.
>>
>
> Other drivers (ixgbe, e1000, ...) always allocate based on the max
> ring size too. Since the VMXNET3_RX_RING_MAX_SIZE is 4096 I don't
> think it is a huge waste of memory.

OK.  BTW, the RX queues have the same issue.

Re: [PATCH] net/vmxnet3: fix queue size changes

[Charles (Chas) Williams]

On 03/15/2017 06:05 AM, Jan Blunck wrote:
> On Wed, Mar 15, 2017 at 10:45 AM, Charles (Chas) Williams
> <ciwillia@brocade.com> wrote:
>>
>>
>> On 03/15/2017 04:18 AM, Jan Blunck wrote:
>>>
>>> On Tue, Mar 14, 2017 at 5:38 PM, Charles (Chas) Williams
>>> <ciwillia@brocade.com> wrote:
>>>>
>>>>
>>>>
>>>> On 03/14/2017 12:11 PM, Jan Blunck wrote:
>>>>>
>>>>>
>>>>> On Mon, Mar 13, 2017 at 11:41 PM, Charles (Chas) Williams
>>>>> <ciwillia@brocade.com> wrote:
>>>>>>
>>>>>>
>>>>>> If the user reconfigures the queues size, then the previosly allocated
>>>>>> memzone may potentially be too small.  Instead, always free the old
>>>>>> memzone and allocate a new one.
>>>>>>
>>>>>> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver
>>>>>> implementation")
>>>>>>
>>>>>> Signed-off-by: Chas Williams <ciwillia@brocade.com>
>>>>>> ---
>>>>>>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 6 +++---
>>>>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>>> b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>>> index 6649c3f..104e040 100644
>>>>>> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>>> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
>>>>>> @@ -893,8 +893,8 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf
>>>>>> **rx_pkts, uint16_t nb_pkts)
>>>>>>
>>>>>>  /*
>>>>>>   * Create memzone for device rings. malloc can't be used as the
>>>>>> physical
>>>>>> address is
>>>>>> - * needed. If the memzone is already created, then this function
>>>>>> returns
>>>>>> a ptr
>>>>>> - * to the old one.
>>>>>> + * needed. If the memzone already exists, we free it since it may have
>>>>>> been created
>>>>>> + * with a different size.
>>>>>>   */
>>>>>>  static const struct rte_memzone *
>>>>>>  ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
>>>>>> @@ -909,7 +909,7 @@ ring_dma_zone_reserve(struct rte_eth_dev *dev,
>>>>>> const
>>>>>> char *ring_name,
>>>>>>
>>>>>>         mz = rte_memzone_lookup(z_name);
>>>>>>         if (mz)
>>>>>> -               return mz;
>>>>>> +               rte_memzone_free(mz);
>>>>>>
>>>>>>         return rte_memzone_reserve_aligned(z_name, ring_size,
>>>>>>                                            socket_id, 0,
>>>>>> VMXNET3_RING_BA_ALIGN);
>>>>>
>>>>>
>>>>>
>>>>> Chas,
>>>>>
>>>>> Thanks for hunting this one down. Wouldn't the rte_memzone_free()
>>>>> better fit into vmxnet3_cmd_ring_release() ?
>>>>
>>>>
>>>>
>>>> I don't care which way it goes.  I just did what is basically done in
>>>> gpa_zone_reserve() to match the "style".  Tracking the current ring size
>>>> and avoiding reallocating a potentially large chunk of memory seems like
>>>> a better idea.
>>>>
>>>>> Also the ring_dma_zone_reserve() could get replaced by
>>>>> rte_eth_dma_zone_reserve() (see also
>>>>
>>>>
>>>>
>>>> Yes, it probably should get changed to that along with tracking the size.
>>>
>>>
>>> Why don't we always allocate VMXNET3_RX_RING_MAX_SIZE entries? That
>>> way we don't need to reallocate on a later queue setup change?
>>
>>
>> That's using more memory than it needs to use.  It might break someone's
>> application
>> that already runs in some tightly constrained machine.  Failing to shrink
>> the memzone
>> isn't likely to break anything since they have (apparently) already dealt
>> with having
>> less memory available before switching to a smaller queue size.
>>
>> Still, it can be a matter for another day.
>>
>
> Other drivers (ixgbe, e1000, ...) always allocate based on the max
> ring size too. Since the VMXNET3_RX_RING_MAX_SIZE is 4096 I don't
> think it is a huge waste of memory.

I changed my mind.  Our typical application only uses a queue length of
512 for the TX side which would mean we don't use more than half of the
allocated elements.  It's easy enough to just allocate what we need and
release in the queue releases as you suggested earlier.

[PATCH] net/vmxnet3: fix queue size changes

[Charles (Chas) Williams]

If the user reconfigures the queue size, then the previously allocated
memzone may potentially be too small.  Release the memzone when a queue
is released and allocate a new one each time a queue is setup.

While here convert to rte_eth_dma_zone_reserve() which does basically
the same things as the private function.

Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver implementation")

Signed-off-by: Chas Williams <ciwillia@brocade.com>
---
 drivers/net/vmxnet3/vmxnet3_ring.h |  2 ++
 drivers/net/vmxnet3/vmxnet3_rxtx.c | 37 +++++++++++--------------------------
 2 files changed, 13 insertions(+), 26 deletions(-)

diff --git a/drivers/net/vmxnet3/vmxnet3_ring.h b/drivers/net/vmxnet3/vmxnet3_ring.h
index 0ce46c5..d2e8323 100644
--- a/drivers/net/vmxnet3/vmxnet3_ring.h
+++ b/drivers/net/vmxnet3/vmxnet3_ring.h
@@ -141,6 +141,7 @@ typedef struct vmxnet3_tx_queue {
 	uint32_t                     qid;
 	struct Vmxnet3_TxQueueDesc   *shared;
 	struct vmxnet3_txq_stats     stats;
+	const struct rte_memzone     *mz;
 	bool                         stopped;
 	uint16_t                     queue_id;      /**< Device TX queue index. */
 	uint8_t                      port_id;       /**< Device port identifier. */
@@ -175,6 +176,7 @@ typedef struct vmxnet3_rx_queue {
 	struct rte_mbuf             *start_seg;
 	struct rte_mbuf             *last_seg;
 	struct vmxnet3_rxq_stats    stats;
+	const struct rte_memzone    *mz;
 	bool                        stopped;
 	uint16_t                    queue_id;      /**< Device RX queue index. */
 	uint8_t                     port_id;       /**< Device port identifier. */
diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c b/drivers/net/vmxnet3/vmxnet3_rxtx.c
index 6649c3f..e865c67 100644
--- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
+++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
@@ -201,6 +201,8 @@ vmxnet3_dev_tx_queue_release(void *txq)
 		vmxnet3_tx_cmd_ring_release_mbufs(&tq->cmd_ring);
 		/* Release the cmd_ring */
 		vmxnet3_cmd_ring_release(&tq->cmd_ring);
+		/* Release the memzone */
+		rte_memzone_free(tq->mz);
 	}
 }
 
@@ -218,6 +220,9 @@ vmxnet3_dev_rx_queue_release(void *rxq)
 		/* Release both the cmd_rings */
 		for (i = 0; i < VMXNET3_RX_CMDRING_SIZE; i++)
 			vmxnet3_cmd_ring_release(&rq->cmd_ring[i]);
+
+		/* Release the memzone */
+		rte_memzone_free(rq->mz);
 	}
 }
 
@@ -891,30 +896,6 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	return nb_rx;
 }
 
-/*
- * Create memzone for device rings. malloc can't be used as the physical address is
- * needed. If the memzone is already created, then this function returns a ptr
- * to the old one.
- */
-static const struct rte_memzone *
-ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
-		      uint16_t queue_id, uint32_t ring_size, int socket_id)
-{
-	char z_name[RTE_MEMZONE_NAMESIZE];
-	const struct rte_memzone *mz;
-
-	snprintf(z_name, sizeof(z_name), "%s_%s_%d_%d",
-		 dev->driver->pci_drv.driver.name, ring_name,
-		 dev->data->port_id, queue_id);
-
-	mz = rte_memzone_lookup(z_name);
-	if (mz)
-		return mz;
-
-	return rte_memzone_reserve_aligned(z_name, ring_size,
-					   socket_id, 0, VMXNET3_RING_BA_ALIGN);
-}
-
 int
 vmxnet3_dev_tx_queue_setup(struct rte_eth_dev *dev,
 			   uint16_t queue_idx,
@@ -983,11 +964,13 @@ vmxnet3_dev_tx_queue_setup(struct rte_eth_dev *dev,
 	size += sizeof(struct Vmxnet3_TxCompDesc) * comp_ring->size;
 	size += txq->txdata_desc_size * data_ring->size;
 
-	mz = ring_dma_zone_reserve(dev, "txdesc", queue_idx, size, socket_id);
+	mz = rte_eth_dma_zone_reserve(dev, "txdesc", queue_idx, size,
+				      VMXNET3_RING_BA_ALIGN, socket_id);
 	if (mz == NULL) {
 		PMD_INIT_LOG(ERR, "ERROR: Creating queue descriptors zone");
 		return -ENOMEM;
 	}
+	txq->mz = mz;
 	memset(mz->addr, 0, mz->len);
 
 	/* cmd_ring initialization */
@@ -1092,11 +1075,13 @@ vmxnet3_dev_rx_queue_setup(struct rte_eth_dev *dev,
 	if (VMXNET3_VERSION_GE_3(hw) && rxq->data_desc_size)
 		size += rxq->data_desc_size * data_ring->size;
 
-	mz = ring_dma_zone_reserve(dev, "rxdesc", queue_idx, size, socket_id);
+	mz = rte_eth_dma_zone_reserve(dev, "rxdesc", queue_idx, size,
+				      VMXNET3_RING_BA_ALIGN, socket_id);
 	if (mz == NULL) {
 		PMD_INIT_LOG(ERR, "ERROR: Creating queue descriptors zone");
 		return -ENOMEM;
 	}
+	rxq->mz = mz;
 	memset(mz->addr, 0, mz->len);
 
 	/* cmd_ring0 initialization */
-- 
2.1.4

Re: [PATCH] net/vmxnet3: fix queue size changes

[Yong Wang]

> -----Original Message-----
> From: Charles (Chas) Williams [mailto:ciwillia@brocade.com]
> Sent: Wednesday, March 15, 2017 5:35 AM
> To: dev@dpdk.org
> Cc: Yong Wang <yongwang@vmware.com>; Charles (Chas) Williams
> <ciwillia@brocade.com>
> Subject: [PATCH] net/vmxnet3: fix queue size changes
> 
> If the user reconfigures the queue size, then the previously allocated
> memzone may potentially be too small.  Release the memzone when a
> queue
> is released and allocate a new one each time a queue is setup.
> 
> While here convert to rte_eth_dma_zone_reserve() which does basically
> the same things as the private function.
> 
> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver
> implementation")
> 
> Signed-off-by: Chas Williams <ciwillia@brocade.com>
> ---

Looks good to me and cc-ing Shrikrishna, the new vmxnet3 pmd maintainer.

>  drivers/net/vmxnet3/vmxnet3_ring.h |  2 ++
>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 37 +++++++++++-----------------------
> ---
>  2 files changed, 13 insertions(+), 26 deletions(-)
> 
> diff --git a/drivers/net/vmxnet3/vmxnet3_ring.h
> b/drivers/net/vmxnet3/vmxnet3_ring.h
> index 0ce46c5..d2e8323 100644
> --- a/drivers/net/vmxnet3/vmxnet3_ring.h
> +++ b/drivers/net/vmxnet3/vmxnet3_ring.h
> @@ -141,6 +141,7 @@ typedef struct vmxnet3_tx_queue {
>  	uint32_t                     qid;
>  	struct Vmxnet3_TxQueueDesc   *shared;
>  	struct vmxnet3_txq_stats     stats;
> +	const struct rte_memzone     *mz;
>  	bool                         stopped;
>  	uint16_t                     queue_id;      /**< Device TX queue index. */
>  	uint8_t                      port_id;       /**< Device port identifier. */
> @@ -175,6 +176,7 @@ typedef struct vmxnet3_rx_queue {
>  	struct rte_mbuf             *start_seg;
>  	struct rte_mbuf             *last_seg;
>  	struct vmxnet3_rxq_stats    stats;
> +	const struct rte_memzone    *mz;
>  	bool                        stopped;
>  	uint16_t                    queue_id;      /**< Device RX queue index. */
>  	uint8_t                     port_id;       /**< Device port identifier. */
> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c
> b/drivers/net/vmxnet3/vmxnet3_rxtx.c
> index 6649c3f..e865c67 100644
> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
> @@ -201,6 +201,8 @@ vmxnet3_dev_tx_queue_release(void *txq)
>  		vmxnet3_tx_cmd_ring_release_mbufs(&tq->cmd_ring);
>  		/* Release the cmd_ring */
>  		vmxnet3_cmd_ring_release(&tq->cmd_ring);
> +		/* Release the memzone */
> +		rte_memzone_free(tq->mz);
>  	}
>  }
> 
> @@ -218,6 +220,9 @@ vmxnet3_dev_rx_queue_release(void *rxq)
>  		/* Release both the cmd_rings */
>  		for (i = 0; i < VMXNET3_RX_CMDRING_SIZE; i++)
>  			vmxnet3_cmd_ring_release(&rq->cmd_ring[i]);
> +
> +		/* Release the memzone */
> +		rte_memzone_free(rq->mz);
>  	}
>  }
> 
> @@ -891,30 +896,6 @@ vmxnet3_recv_pkts(void *rx_queue, struct
> rte_mbuf **rx_pkts, uint16_t nb_pkts)
>  	return nb_rx;
>  }
> 
> -/*
> - * Create memzone for device rings. malloc can't be used as the physical
> address is
> - * needed. If the memzone is already created, then this function returns a
> ptr
> - * to the old one.
> - */
> -static const struct rte_memzone *
> -ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
> -		      uint16_t queue_id, uint32_t ring_size, int socket_id)
> -{
> -	char z_name[RTE_MEMZONE_NAMESIZE];
> -	const struct rte_memzone *mz;
> -
> -	snprintf(z_name, sizeof(z_name), "%s_%s_%d_%d",
> -		 dev->driver->pci_drv.driver.name, ring_name,
> -		 dev->data->port_id, queue_id);
> -
> -	mz = rte_memzone_lookup(z_name);
> -	if (mz)
> -		return mz;
> -
> -	return rte_memzone_reserve_aligned(z_name, ring_size,
> -					   socket_id, 0,
> VMXNET3_RING_BA_ALIGN);
> -}
> -
>  int
>  vmxnet3_dev_tx_queue_setup(struct rte_eth_dev *dev,
>  			   uint16_t queue_idx,
> @@ -983,11 +964,13 @@ vmxnet3_dev_tx_queue_setup(struct
> rte_eth_dev *dev,
>  	size += sizeof(struct Vmxnet3_TxCompDesc) * comp_ring->size;
>  	size += txq->txdata_desc_size * data_ring->size;
> 
> -	mz = ring_dma_zone_reserve(dev, "txdesc", queue_idx, size,
> socket_id);
> +	mz = rte_eth_dma_zone_reserve(dev, "txdesc", queue_idx, size,
> +				      VMXNET3_RING_BA_ALIGN, socket_id);
>  	if (mz == NULL) {
>  		PMD_INIT_LOG(ERR, "ERROR: Creating queue descriptors
> zone");
>  		return -ENOMEM;
>  	}
> +	txq->mz = mz;
>  	memset(mz->addr, 0, mz->len);
> 
>  	/* cmd_ring initialization */
> @@ -1092,11 +1075,13 @@ vmxnet3_dev_rx_queue_setup(struct
> rte_eth_dev *dev,
>  	if (VMXNET3_VERSION_GE_3(hw) && rxq->data_desc_size)
>  		size += rxq->data_desc_size * data_ring->size;
> 
> -	mz = ring_dma_zone_reserve(dev, "rxdesc", queue_idx, size,
> socket_id);
> +	mz = rte_eth_dma_zone_reserve(dev, "rxdesc", queue_idx, size,
> +				      VMXNET3_RING_BA_ALIGN, socket_id);
>  	if (mz == NULL) {
>  		PMD_INIT_LOG(ERR, "ERROR: Creating queue descriptors
> zone");
>  		return -ENOMEM;
>  	}
> +	rxq->mz = mz;
>  	memset(mz->addr, 0, mz->len);
> 
>  	/* cmd_ring0 initialization */
> --
> 2.1.4

Re: [PATCH] net/vmxnet3: fix queue size changes

[Jan Blunck]

On Wed, Mar 15, 2017 at 1:35 PM, Charles (Chas) Williams
<ciwillia@brocade.com> wrote:
> If the user reconfigures the queue size, then the previously allocated
> memzone may potentially be too small.  Release the memzone when a queue
> is released and allocate a new one each time a queue is setup.
>
> While here convert to rte_eth_dma_zone_reserve() which does basically
> the same things as the private function.
>
> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver implementation")
>
> Signed-off-by: Chas Williams <ciwillia@brocade.com>
> ---
>  drivers/net/vmxnet3/vmxnet3_ring.h |  2 ++
>  drivers/net/vmxnet3/vmxnet3_rxtx.c | 37 +++++++++++--------------------------
>  2 files changed, 13 insertions(+), 26 deletions(-)
>
> diff --git a/drivers/net/vmxnet3/vmxnet3_ring.h b/drivers/net/vmxnet3/vmxnet3_ring.h
> index 0ce46c5..d2e8323 100644
> --- a/drivers/net/vmxnet3/vmxnet3_ring.h
> +++ b/drivers/net/vmxnet3/vmxnet3_ring.h
> @@ -141,6 +141,7 @@ typedef struct vmxnet3_tx_queue {
>         uint32_t                     qid;
>         struct Vmxnet3_TxQueueDesc   *shared;
>         struct vmxnet3_txq_stats     stats;
> +       const struct rte_memzone     *mz;
>         bool                         stopped;
>         uint16_t                     queue_id;      /**< Device TX queue index. */
>         uint8_t                      port_id;       /**< Device port identifier. */
> @@ -175,6 +176,7 @@ typedef struct vmxnet3_rx_queue {
>         struct rte_mbuf             *start_seg;
>         struct rte_mbuf             *last_seg;
>         struct vmxnet3_rxq_stats    stats;
> +       const struct rte_memzone    *mz;
>         bool                        stopped;
>         uint16_t                    queue_id;      /**< Device RX queue index. */
>         uint8_t                     port_id;       /**< Device port identifier. */
> diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c b/drivers/net/vmxnet3/vmxnet3_rxtx.c
> index 6649c3f..e865c67 100644
> --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c
> +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c
> @@ -201,6 +201,8 @@ vmxnet3_dev_tx_queue_release(void *txq)
>                 vmxnet3_tx_cmd_ring_release_mbufs(&tq->cmd_ring);
>                 /* Release the cmd_ring */
>                 vmxnet3_cmd_ring_release(&tq->cmd_ring);
> +               /* Release the memzone */
> +               rte_memzone_free(tq->mz);
>         }
>  }
>
> @@ -218,6 +220,9 @@ vmxnet3_dev_rx_queue_release(void *rxq)
>                 /* Release both the cmd_rings */
>                 for (i = 0; i < VMXNET3_RX_CMDRING_SIZE; i++)
>                         vmxnet3_cmd_ring_release(&rq->cmd_ring[i]);
> +
> +               /* Release the memzone */
> +               rte_memzone_free(rq->mz);
>         }
>  }
>
> @@ -891,30 +896,6 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
>         return nb_rx;
>  }
>
> -/*
> - * Create memzone for device rings. malloc can't be used as the physical address is
> - * needed. If the memzone is already created, then this function returns a ptr
> - * to the old one.
> - */
> -static const struct rte_memzone *
> -ring_dma_zone_reserve(struct rte_eth_dev *dev, const char *ring_name,
> -                     uint16_t queue_id, uint32_t ring_size, int socket_id)
> -{
> -       char z_name[RTE_MEMZONE_NAMESIZE];
> -       const struct rte_memzone *mz;
> -
> -       snprintf(z_name, sizeof(z_name), "%s_%s_%d_%d",
> -                dev->driver->pci_drv.driver.name, ring_name,
> -                dev->data->port_id, queue_id);
> -
> -       mz = rte_memzone_lookup(z_name);
> -       if (mz)
> -               return mz;
> -
> -       return rte_memzone_reserve_aligned(z_name, ring_size,
> -                                          socket_id, 0, VMXNET3_RING_BA_ALIGN);
> -}
> -
>  int
>  vmxnet3_dev_tx_queue_setup(struct rte_eth_dev *dev,
>                            uint16_t queue_idx,
> @@ -983,11 +964,13 @@ vmxnet3_dev_tx_queue_setup(struct rte_eth_dev *dev,
>         size += sizeof(struct Vmxnet3_TxCompDesc) * comp_ring->size;
>         size += txq->txdata_desc_size * data_ring->size;
>
> -       mz = ring_dma_zone_reserve(dev, "txdesc", queue_idx, size, socket_id);
> +       mz = rte_eth_dma_zone_reserve(dev, "txdesc", queue_idx, size,
> +                                     VMXNET3_RING_BA_ALIGN, socket_id);
>         if (mz == NULL) {
>                 PMD_INIT_LOG(ERR, "ERROR: Creating queue descriptors zone");
>                 return -ENOMEM;
>         }
> +       txq->mz = mz;
>         memset(mz->addr, 0, mz->len);
>
>         /* cmd_ring initialization */
> @@ -1092,11 +1075,13 @@ vmxnet3_dev_rx_queue_setup(struct rte_eth_dev *dev,
>         if (VMXNET3_VERSION_GE_3(hw) && rxq->data_desc_size)
>                 size += rxq->data_desc_size * data_ring->size;
>
> -       mz = ring_dma_zone_reserve(dev, "rxdesc", queue_idx, size, socket_id);
> +       mz = rte_eth_dma_zone_reserve(dev, "rxdesc", queue_idx, size,
> +                                     VMXNET3_RING_BA_ALIGN, socket_id);
>         if (mz == NULL) {
>                 PMD_INIT_LOG(ERR, "ERROR: Creating queue descriptors zone");
>                 return -ENOMEM;
>         }
> +       rxq->mz = mz;
>         memset(mz->addr, 0, mz->len);
>
>         /* cmd_ring0 initialization */
> --
> 2.1.4
>

Acked-by: Jan Blunck <jblunck@infradead.org>

Re: [PATCH] net/vmxnet3: fix queue size changes

[Shrikrishna Khare]

On Wed, 15 Mar 2017, Yong Wang wrote:

> > -----Original Message-----
> > From: Charles (Chas) Williams [mailto:ciwillia@brocade.com]
> > Sent: Wednesday, March 15, 2017 5:35 AM
> > To: dev@dpdk.org
> > Cc: Yong Wang <yongwang@vmware.com>; Charles (Chas) Williams
> > <ciwillia@brocade.com>
> > Subject: [PATCH] net/vmxnet3: fix queue size changes
> >
> > If the user reconfigures the queue size, then the previously allocated
> > memzone may potentially be too small.  Release the memzone when a
> > queue
> > is released and allocate a new one each time a queue is setup.
> >
> > While here convert to rte_eth_dma_zone_reserve() which does basically
> > the same things as the private function.
> >
> > Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver
> > implementation")
> >
> > Signed-off-by: Chas Williams <ciwillia@brocade.com>
> > ---
> 
> Looks good to me and cc-ing Shrikrishna, the new vmxnet3 pmd maintainer.

Looks good to me too.

Acked-by: Shrikrishna Khare <skhare@vmware.com>

Re: [PATCH] net/vmxnet3: fix queue size changes

[Ferruh Yigit]

On 3/15/2017 6:19 PM, Jan Blunck wrote:
> On Wed, Mar 15, 2017 at 1:35 PM, Charles (Chas) Williams
> <ciwillia@brocade.com> wrote:
>> If the user reconfigures the queue size, then the previously allocated
>> memzone may potentially be too small.  Release the memzone when a queue
>> is released and allocate a new one each time a queue is setup.
>>
>> While here convert to rte_eth_dma_zone_reserve() which does basically
>> the same things as the private function.
>>
>> Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode driver implementation")
   Cc: stable@dpdk.org
>>
>> Signed-off-by: Chas Williams <ciwillia@brocade.com>

> Acked-by: Jan Blunck <jblunck@infradead.org>

Acked-by: Shrikrishna Khare <skhare@vmware.com>

Applied to dpdk-next-net/master, thanks.