[PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Jason Wang]

For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.

For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
("macvtap: Add support of packet capture on macvtap
device."). Multiqueue macvtap suffers from single qdisc lock
contention. This is because macvtap claims a non zero tx_queue_len and
it reuses this value as it socket receive queue size.Thanks to
IFF_NO_QUEUE, we can remove the lock contention without breaking
existing socket receive queue length logic.

Cc: Patrick McHardy <kaber@trash.net>
Cc: Vladislav Yasevich <vyasevic@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/net/macvlan.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index 47da435..09d8718 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -1056,7 +1056,7 @@ void macvlan_common_setup(struct net_device *dev)
 
 	dev->priv_flags	       &= ~IFF_TX_SKB_SHARING;
 	netif_keep_dst(dev);
-	dev->priv_flags	       |= IFF_UNICAST_FLT;
+	dev->priv_flags	       |= IFF_UNICAST_FLT | IFF_NO_QUEUE;
 	dev->netdev_ops		= &macvlan_netdev_ops;
 	dev->destructor		= free_netdev;
 	dev->header_ops		= &macvlan_hard_header_ops;
@@ -1067,7 +1067,6 @@ EXPORT_SYMBOL_GPL(macvlan_common_setup);
 static void macvlan_setup(struct net_device *dev)
 {
 	macvlan_common_setup(dev);
-	dev->tx_queue_len	= 0;
 }
 
 static int macvlan_port_create(struct net_device *dev)
-- 
2.1.4

Re: [PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Michael S. Tsirkin]

On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
> 
> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
> ("macvtap: Add support of packet capture on macvtap
> device."). Multiqueue macvtap suffers from single qdisc lock
> contention. This is because macvtap claims a non zero tx_queue_len and
> it reuses this value as it socket receive queue size.Thanks to
> IFF_NO_QUEUE, we can remove the lock contention without breaking
> existing socket receive queue length logic.
> 
> Cc: Patrick McHardy <kaber@trash.net>
> Cc: Vladislav Yasevich <vyasevic@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>

Seems to make sense. Give me a day or two to get over the jet lag
(and get out from under the pile of mail accumulated while I was traveling),
I'll review properly and ack.

> ---
>  drivers/net/macvlan.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
> index 47da435..09d8718 100644
> --- a/drivers/net/macvlan.c
> +++ b/drivers/net/macvlan.c
> @@ -1056,7 +1056,7 @@ void macvlan_common_setup(struct net_device *dev)
>  
>  	dev->priv_flags	       &= ~IFF_TX_SKB_SHARING;
>  	netif_keep_dst(dev);
> -	dev->priv_flags	       |= IFF_UNICAST_FLT;
> +	dev->priv_flags	       |= IFF_UNICAST_FLT | IFF_NO_QUEUE;
>  	dev->netdev_ops		= &macvlan_netdev_ops;
>  	dev->destructor		= free_netdev;
>  	dev->header_ops		= &macvlan_hard_header_ops;
> @@ -1067,7 +1067,6 @@ EXPORT_SYMBOL_GPL(macvlan_common_setup);
>  static void macvlan_setup(struct net_device *dev)
>  {
>  	macvlan_common_setup(dev);
> -	dev->tx_queue_len	= 0;
>  }
>  
>  static int macvlan_port_create(struct net_device *dev)
> -- 
> 2.1.4

Re: [PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Jason Wang]

On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>> > For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>> > 
>> > For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>> > ("macvtap: Add support of packet capture on macvtap
>> > device."). Multiqueue macvtap suffers from single qdisc lock
>> > contention. This is because macvtap claims a non zero tx_queue_len and
>> > it reuses this value as it socket receive queue size.Thanks to
>> > IFF_NO_QUEUE, we can remove the lock contention without breaking
>> > existing socket receive queue length logic.
>> > 
>> > Cc: Patrick McHardy <kaber@trash.net>
>> > Cc: Vladislav Yasevich <vyasevic@redhat.com>
>> > Cc: Michael S. Tsirkin <mst@redhat.com>
>> > Signed-off-by: Jason Wang <jasowang@redhat.com>
> Seems to make sense. Give me a day or two to get over the jet lag
> (and get out from under the pile of mail accumulated while I was traveling),
> I'll review properly and ack.
>

A note on this patch: only default qdisc were removed but we don't lose
the ability to attach a qdisc to macvtap (though it may cause lock
contention on multiqueue case).

Re: [PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Vlad Yasevich]

On 08/25/2015 07:30 AM, Jason Wang wrote:
> 
> 
> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>
>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>> ("macvtap: Add support of packet capture on macvtap
>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>> it reuses this value as it socket receive queue size.Thanks to
>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>> existing socket receive queue length logic.
>>>>
>>>> Cc: Patrick McHardy <kaber@trash.net>
>>>> Cc: Vladislav Yasevich <vyasevic@redhat.com>
>>>> Cc: Michael S. Tsirkin <mst@redhat.com>
>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> Seems to make sense. Give me a day or two to get over the jet lag
>> (and get out from under the pile of mail accumulated while I was traveling),
>> I'll review properly and ack.
>>
> 
> A note on this patch: only default qdisc were removed but we don't lose
> the ability to attach a qdisc to macvtap (though it may cause lock
> contention on multiqueue case).
> 

Wouldn't that lock contention be solved if we really had multiple queues
for multi-queue macvtaps?

-vlad

Re: [PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Jason Wang]

On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
> On 08/25/2015 07:30 AM, Jason Wang wrote:
>>
>> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>>
>>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>> ("macvtap: Add support of packet capture on macvtap
>>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>>> it reuses this value as it socket receive queue size.Thanks to
>>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>>> existing socket receive queue length logic.
>>>>>
>>>>> Cc: Patrick McHardy <kaber@trash.net>
>>>>> Cc: Vladislav Yasevich <vyasevic@redhat.com>
>>>>> Cc: Michael S. Tsirkin <mst@redhat.com>
>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>> Seems to make sense. Give me a day or two to get over the jet lag
>>> (and get out from under the pile of mail accumulated while I was traveling),
>>> I'll review properly and ack.
>>>
>> A note on this patch: only default qdisc were removed but we don't lose
>> the ability to attach a qdisc to macvtap (though it may cause lock
>> contention on multiqueue case).
>>
> Wouldn't that lock contention be solved if we really had multiple queues
> for multi-queue macvtaps?
>
> -vlad

Yes, but this introduce another layer of txq locks contention? And it
also needs macvlan multiqueue support. We used to do something like this
but switch to NETIF_F_LLTX finally. You may refer:

2c11455321f37da6fe6cc36353149f9ac9183334 macvlan: add multiqueue capability
8ffab51b3dfc54876f145f15b351c41f3f703195 macvlan: lockless tx path

Re: [PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Michael S. Tsirkin]

On Wed, Aug 26, 2015 at 01:45:30PM +0800, Jason Wang wrote:
> 
> 
> On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
> > On 08/25/2015 07:30 AM, Jason Wang wrote:
> >>
> >> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
> >>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
> >>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
> >>>>>
> >>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
> >>>>> ("macvtap: Add support of packet capture on macvtap
> >>>>> device."). Multiqueue macvtap suffers from single qdisc lock
> >>>>> contention. This is because macvtap claims a non zero tx_queue_len and
> >>>>> it reuses this value as it socket receive queue size.Thanks to
> >>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
> >>>>> existing socket receive queue length logic.
> >>>>>
> >>>>> Cc: Patrick McHardy <kaber@trash.net>
> >>>>> Cc: Vladislav Yasevich <vyasevic@redhat.com>
> >>>>> Cc: Michael S. Tsirkin <mst@redhat.com>
> >>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
> >>> Seems to make sense. Give me a day or two to get over the jet lag
> >>> (and get out from under the pile of mail accumulated while I was traveling),
> >>> I'll review properly and ack.
> >>>
> >> A note on this patch: only default qdisc were removed but we don't lose
> >> the ability to attach a qdisc to macvtap (though it may cause lock
> >> contention on multiqueue case).
> >>
> > Wouldn't that lock contention be solved if we really had multiple queues
> > for multi-queue macvtaps?
> >
> > -vlad
> 
> Yes, but this introduce another layer of txq locks contention?

I don't follow - why does it? Could you clarify please?

> And it
> also needs macvlan multiqueue support. We used to do something like this
> but switch to NETIF_F_LLTX finally. You may refer:
> 
> 2c11455321f37da6fe6cc36353149f9ac9183334 macvlan: add multiqueue capability
> 8ffab51b3dfc54876f145f15b351c41f3f703195 macvlan: lockless tx path

My concern is that the moment someone configures a non-standard qdisc
scalability suddenly disappears. That would also be tricky to debug in the
field as not a lot of developers use non-standard qdiscs.
What do you think?

-- 
MST

Re: [PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Jason Wang]

On 08/27/2015 06:43 PM, Michael S. Tsirkin wrote:
> On Wed, Aug 26, 2015 at 01:45:30PM +0800, Jason Wang wrote:
>>
>> On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
>>> On 08/25/2015 07:30 AM, Jason Wang wrote:
>>>> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>>>>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>>>>
>>>>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>>> ("macvtap: Add support of packet capture on macvtap
>>>>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>>>>> it reuses this value as it socket receive queue size.Thanks to
>>>>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>>>>> existing socket receive queue length logic.
>>>>>>>
>>>>>>> Cc: Patrick McHardy <kaber@trash.net>
>>>>>>> Cc: Vladislav Yasevich <vyasevic@redhat.com>
>>>>>>> Cc: Michael S. Tsirkin <mst@redhat.com>
>>>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>> Seems to make sense. Give me a day or two to get over the jet lag
>>>>> (and get out from under the pile of mail accumulated while I was traveling),
>>>>> I'll review properly and ack.
>>>>>
>>>> A note on this patch: only default qdisc were removed but we don't lose
>>>> the ability to attach a qdisc to macvtap (though it may cause lock
>>>> contention on multiqueue case).
>>>>
>>> Wouldn't that lock contention be solved if we really had multiple queues
>>> for multi-queue macvtaps?
>>>
>>> -vlad
>> Yes, but this introduce another layer of txq locks contention?
> I don't follow - why does it? Could you clarify please?

I believe Vlad wants to remove NETIF_F_LLTX. If yes, core will do an
extra tx lock at macvlan layer.

>
>> And it
>> also needs macvlan multiqueue support. We used to do something like this
>> but switch to NETIF_F_LLTX finally. You may refer:
>>
>> 2c11455321f37da6fe6cc36353149f9ac9183334 macvlan: add multiqueue capability
>> 8ffab51b3dfc54876f145f15b351c41f3f703195 macvlan: lockless tx path
> My concern is that the moment someone configures a non-standard qdisc
> scalability suddenly disappears. That would also be tricky to debug in the
> field as not a lot of developers use non-standard qdiscs.
> What do you think?

Probably not an issue. Non-standard qdisc may need be attached manually
after device creation, and we don't lose this ability with this patch.
(Unless somebody changes default_qdisc). Actually, user before
6acf54f1cf0a6747bac9fea26f34cfc5a9029523 does not expect any qdisc work
for macvtap like other stacked devices. This patch also restore this.

Re: [PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Vlad Yasevich]

On 08/27/2015 10:42 PM, Jason Wang wrote:
> 
> 
> On 08/27/2015 06:43 PM, Michael S. Tsirkin wrote:
>> On Wed, Aug 26, 2015 at 01:45:30PM +0800, Jason Wang wrote:
>>>
>>> On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
>>>> On 08/25/2015 07:30 AM, Jason Wang wrote:
>>>>> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>>>>>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>>>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>>>>>
>>>>>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>>>> ("macvtap: Add support of packet capture on macvtap
>>>>>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>>>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>>>>>> it reuses this value as it socket receive queue size.Thanks to
>>>>>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>>>>>> existing socket receive queue length logic.
>>>>>>>>
>>>>>>>> Cc: Patrick McHardy <kaber@trash.net>
>>>>>>>> Cc: Vladislav Yasevich <vyasevic@redhat.com>
>>>>>>>> Cc: Michael S. Tsirkin <mst@redhat.com>
>>>>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>>> Seems to make sense. Give me a day or two to get over the jet lag
>>>>>> (and get out from under the pile of mail accumulated while I was traveling),
>>>>>> I'll review properly and ack.
>>>>>>
>>>>> A note on this patch: only default qdisc were removed but we don't lose
>>>>> the ability to attach a qdisc to macvtap (though it may cause lock
>>>>> contention on multiqueue case).
>>>>>
>>>> Wouldn't that lock contention be solved if we really had multiple queues
>>>> for multi-queue macvtaps?
>>>>
>>>> -vlad
>>> Yes, but this introduce another layer of txq locks contention?
>> I don't follow - why does it? Could you clarify please?
> 
> I believe Vlad wants to remove NETIF_F_LLTX. If yes, core will do an
> extra tx lock at macvlan layer.

No, I don't want to remove it.  In a sense, it would function similar to
how it works when fwd_priv is populated.  I am still testing the code
as it's showing some strange artifacts...  could be due to keeping LLTX.

-vlad

> 
>>
>>> And it
>>> also needs macvlan multiqueue support. We used to do something like this
>>> but switch to NETIF_F_LLTX finally. You may refer:
>>>
>>> 2c11455321f37da6fe6cc36353149f9ac9183334 macvlan: add multiqueue capability
>>> 8ffab51b3dfc54876f145f15b351c41f3f703195 macvlan: lockless tx path
>> My concern is that the moment someone configures a non-standard qdisc
>> scalability suddenly disappears. That would also be tricky to debug in the
>> field as not a lot of developers use non-standard qdiscs.
>> What do you think?
> 
> Probably not an issue. Non-standard qdisc may need be attached manually
> after device creation, and we don't lose this ability with this patch.
> (Unless somebody changes default_qdisc). Actually, user before
> 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 does not expect any qdisc work
> for macvtap like other stacked devices. This patch also restore this.
> 
> 

Re: [PATCH net-next] macvtap/macvlan: use IFF_NO_QUEUE

[Jason Wang]

On 08/28/2015 08:25 PM, Vlad Yasevich wrote:
> On 08/27/2015 10:42 PM, Jason Wang wrote:
>> > 
>> > 
>> > On 08/27/2015 06:43 PM, Michael S. Tsirkin wrote:
>>> >> On Wed, Aug 26, 2015 at 01:45:30PM +0800, Jason Wang wrote:
>>>> >>>
>>>> >>> On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
>>>>> >>>> On 08/25/2015 07:30 AM, Jason Wang wrote:
>>>>>> >>>>> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>>>>>>> >>>>>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>>>>>>> >>>>>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>>>>> >>>>>>>> ("macvtap: Add support of packet capture on macvtap
>>>>>>>>> >>>>>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>>>>>>> >>>>>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>>>>>>> >>>>>>>> it reuses this value as it socket receive queue size.Thanks to
>>>>>>>>> >>>>>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>>>>>>> >>>>>>>> existing socket receive queue length logic.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Cc: Patrick McHardy <kaber@trash.net>
>>>>>>>>> >>>>>>>> Cc: Vladislav Yasevich <vyasevic@redhat.com>
>>>>>>>>> >>>>>>>> Cc: Michael S. Tsirkin <mst@redhat.com>
>>>>>>>>> >>>>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>>>> >>>>>> Seems to make sense. Give me a day or two to get over the jet lag
>>>>>>> >>>>>> (and get out from under the pile of mail accumulated while I was traveling),
>>>>>>> >>>>>> I'll review properly and ack.
>>>>>>> >>>>>>
>>>>>> >>>>> A note on this patch: only default qdisc were removed but we don't lose
>>>>>> >>>>> the ability to attach a qdisc to macvtap (though it may cause lock
>>>>>> >>>>> contention on multiqueue case).
>>>>>> >>>>>
>>>>> >>>> Wouldn't that lock contention be solved if we really had multiple queues
>>>>> >>>> for multi-queue macvtaps?
>>>>> >>>>
>>>>> >>>> -vlad
>>>> >>> Yes, but this introduce another layer of txq locks contention?
>>> >> I don't follow - why does it? Could you clarify please?
>> > 
>> > I believe Vlad wants to remove NETIF_F_LLTX. If yes, core will do an
>> > extra tx lock at macvlan layer.
> No, I don't want to remove it.  In a sense, it would function similar to
> how it works when fwd_priv is populated.  I am still testing the code
> as it's showing some strange artifacts...  could be due to keeping LLTX.
>
> -vlad
>

I see. I'm ok to wait for your code. But if a patch of just two lines
works, probably no need to try complex method.

Thanks