KNI discussion in userspace event

KNI discussion in userspace event

[Ferruh Yigit]

Hi,

There was an "Interworking with the Linux Kernel" discussion in the DPDK
userspace event, this mail is to summarize the output and to get more
comments from community.


Briefly, KNI mostly will stay as it is as an interworking with the Linux
kernel solution.
Out of tree kernel module concern is still there, but there is no clear
alternative to switch. And community still care about performance of KNI
and control path of KNI. Only KNI VHOST may go away. KNI PMD depends on
community interest. There was no modification request on KNI library and
sample app.


Discussed alternatives were:
* Tun/Tap
This won't be as fast as KNI and performance is an issue.

* virtio-user + vhost-net
This can be valid alternative, removes the out of tree kernel module
need. But missing control path. Proof of concept work will be done.

* Bifurcated driver
Not able to filter all traffic, not a full functional alternative.

* Upstreaming kernel module:
Stephen suggested upstreaming a generic shim layer and use it.


Future of the KNI:
* Remove ethtool support ?
Still there is some interest, will keep it. But not able to extend it to
other drivers with current design.

* Remove KNI VHOST?
There were no interest for this feature. I will send a deprecation
notice to remove this, and we can discuss more there.

* What to do with out of tree kernel module
It is still problem for OSVs and unfortunately it is staying.

* Switch completely to an alternative approach?
There won't be an action for a switch. virtio-user + vhost-net
alternative will be investigated.

*KNI PMD
Patch is in the mail list, missing comments. If it gets some
interest/comments/acks it may go in to next release.

* Any improvement on library or sample app?
Nothing listed.


Thanks,
ferruh

Re: KNI discussion in userspace event

[Igor Ryzhov]

Thank you, Ferruh.

As we are staying on the existing implementation, I think we can do some
improvements:
1. Implement more commands for net_device_ops.
2. Implement ethtool support the same way as net_device_ops are implemented
– send commands to application.
3. Add ability to set default MAC address for KNI interface. Now it is
random for all interfaces except those that work on igb or ixgbe.
4. Properly implement link state control feature. Now KNI interface is in
UNKNOWN state even after changing carrier flag to 1.

First two improvements are already done in KCP patches and can be easily
ported into the current code.
For the last two improvements I can send patches.

Best regards,
Igor

Re: KNI discussion in userspace event

[Thomas Monjalon]

2016-10-28 15:31, Ferruh Yigit:
> * virtio-user + vhost-net
> This can be valid alternative, removes the out of tree kernel module
> need. But missing control path. Proof of concept work will be done.

That's probably a smart alternative for packet injection.
What do you mean exactly by "missing control path"?

> * Remove ethtool support ?

That's the other part of KNI.
It works only for e1000/ixgbe. That's a niche.

> Still there is some interest, will keep it. But not able to extend it to
> other drivers with current design.

It should be removed one day.
We must seriously think about a generic alternative.
Either we add DPDK support in ethtool or we create a dpdk-ethtool.
(or at least a library as the one in examples/).
Or we do nothing and wait to have more hardware like Mellanox supporting
a kernel bifurcated driver approach.

> *KNI PMD
> Patch is in the mail list, missing comments. If it gets some
> interest/comments/acks it may go in to next release.

I'm not against KNI PMD but it looks strange to add more support
to an old dying approach.

Re: KNI discussion in userspace event

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> Sent: Friday, October 28, 2016 4:13 PM
> To: Yigit, Ferruh <ferruh.yigit@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] KNI discussion in userspace event
> 
> 2016-10-28 15:31, Ferruh Yigit:
> > * virtio-user + vhost-net
> > This can be valid alternative, removes the out of tree kernel module
> > need. But missing control path. Proof of concept work will be done.
> 
> That's probably a smart alternative for packet injection.
> What do you mean exactly by "missing control path"?

We'll have to see how it performs - which is the key gap for data path that KNI fills. Until we get an alternative with (nearly) equivalent performance, there will be demand for KNI to stick around.
The "control path" is the ethtool part, to get stats and do operations on the NIC using command-line tools.

> 
> > * Remove ethtool support ?
> 
> That's the other part of KNI.
> It works only for e1000/ixgbe. That's a niche.

Yes, it's something we need to remove, but again, we need an alternative first.

> 
> > Still there is some interest, will keep it. But not able to extend it
> > to other drivers with current design.
> 
> It should be removed one day.
> We must seriously think about a generic alternative.
> Either we add DPDK support in ethtool or we create a dpdk-ethtool.
> (or at least a library as the one in examples/).

I don't view that as a great path forward. Sure, we can do our own ethtool, but then people will look for ifconfig to work, and "ip" to work, etc. I view having a kernel proxy module as the best path here as it is tool agnostic on the userspace side, rather than trying to make every tool for working with kernel netdevs also have support for dpdk ports.

> Or we do nothing and wait to have more hardware like Mellanox supporting a
> kernel bifurcated driver approach.

Given the lack of other NICs supporting that, I think it could be quite a wait! Also, it doesn't work for virtio ports, for pcap ports, or any other ports which don't have physical hardware backing them. No reason you shouldn't be able to pull stats from all your dpdk ethdevs, not just the ones with physical hardware. The same ethdev APIs work for them, so should the same tools.

> 
> > *KNI PMD
> > Patch is in the mail list, missing comments. If it gets some
> > interest/comments/acks it may go in to next release.
> 
> I'm not against KNI PMD but it looks strange to add more support to an old
> dying approach.

I think the main idea here is to clean up the API - at least for the data path. There is no reason why we need special KNI RX/TX functions, when ethdev RX/TX functions could do the job. However, at a higher level, the more basic requirement is that whatever solution for the data-path to kernel from dpdk is, it needs to appear as an ethdev, and not as a special library with different APIs, as KNI is now.

/Bruce

Re: KNI discussion in userspace event

[Thomas Monjalon]

2016-10-28 15:51, Richardson, Bruce:
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> > 2016-10-28 15:31, Ferruh Yigit:
> > > * virtio-user + vhost-net
> > > This can be valid alternative, removes the out of tree kernel module
> > > need. But missing control path. Proof of concept work will be done.
> > 
> > That's probably a smart alternative for packet injection.
> > What do you mean exactly by "missing control path"?
> 
> We'll have to see how it performs - which is the key gap for data path that KNI fills. Until we get an alternative with (nearly) equivalent performance, there will be demand for KNI to stick around.
> The "control path" is the ethtool part, to get stats and do operations on the NIC using command-line tools.
> 
> > 
> > > * Remove ethtool support ?
> > 
> > That's the other part of KNI.
> > It works only for e1000/ixgbe. That's a niche.
> 
> Yes, it's something we need to remove, but again, we need an alternative first.
> 
> > 
> > > Still there is some interest, will keep it. But not able to extend it
> > > to other drivers with current design.
> > 
> > It should be removed one day.
> > We must seriously think about a generic alternative.
> > Either we add DPDK support in ethtool or we create a dpdk-ethtool.
> > (or at least a library as the one in examples/).
> 
> I don't view that as a great path forward. Sure, we can do our own ethtool, but then people will look for ifconfig to work, and "ip" to work, etc. I view having a kernel proxy module as the best path here as it is tool agnostic on the userspace side, rather than trying to make every tool for working with kernel netdevs also have support for dpdk ports.

Yes that's the ultimately best solution.
But:
- we need some cooperation of the kernel team
- ethtool manages a device (what DPDK provides) whereas iproute and others
manage a TCP/IP stack so is out of control of DPDK.

> > Or we do nothing and wait to have more hardware like Mellanox supporting a
> > kernel bifurcated driver approach.
> 
> Given the lack of other NICs supporting that, I think it could be quite a wait! Also, it doesn't work for virtio ports, for pcap ports, or any other ports which don't have physical hardware backing them. No reason you shouldn't be able to pull stats from all your dpdk ethdevs, not just the ones with physical hardware. The same ethdev APIs work for them, so should the same tools.

Yes, very good point.

> > > *KNI PMD
> > > Patch is in the mail list, missing comments. If it gets some
> > > interest/comments/acks it may go in to next release.
> > 
> > I'm not against KNI PMD but it looks strange to add more support to an old
> > dying approach.
> 
> I think the main idea here is to clean up the API - at least for the data path. There is no reason why we need special KNI RX/TX functions, when ethdev RX/TX functions could do the job. However, at a higher level, the more basic requirement is that whatever solution for the data-path to kernel from dpdk is, it needs to appear as an ethdev, and not as a special library with different APIs, as KNI is now.

Yes I agree to unifiy (and reduce) API.
Why this PMD is not more commented?
KNI users should be interested to review it.
Please dear community, we need more reviews!

Re: KNI discussion in userspace event

[Igor Ryzhov]

On Fri, Oct 28, 2016 at 7:13 PM, Thomas Monjalon <thomas.monjalon@6wind.com>
wrote:

> 2016-10-28 15:51, Richardson, Bruce:
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> > > 2016-10-28 15:31, Ferruh Yigit:
> > > > * virtio-user + vhost-net
> > > > This can be valid alternative, removes the out of tree kernel module
> > > > need. But missing control path. Proof of concept work will be done.
> > >
> > > That's probably a smart alternative for packet injection.
> > > What do you mean exactly by "missing control path"?
> >
> > We'll have to see how it performs - which is the key gap for data path
> that KNI fills. Until we get an alternative with (nearly) equivalent
> performance, there will be demand for KNI to stick around.
> > The "control path" is the ethtool part, to get stats and do operations
> on the NIC using command-line tools.
> >
> > >
> > > > * Remove ethtool support ?
> > >
> > > That's the other part of KNI.
> > > It works only for e1000/ixgbe. That's a niche.
> >
> > Yes, it's something we need to remove, but again, we need an alternative
> first.
> >
> > >
> > > > Still there is some interest, will keep it. But not able to extend it
> > > > to other drivers with current design.
> > >
> > > It should be removed one day.
> > > We must seriously think about a generic alternative.
> > > Either we add DPDK support in ethtool or we create a dpdk-ethtool.
> > > (or at least a library as the one in examples/).
> >
> > I don't view that as a great path forward. Sure, we can do our own
> ethtool, but then people will look for ifconfig to work, and "ip" to work,
> etc. I view having a kernel proxy module as the best path here as it is
> tool agnostic on the userspace side, rather than trying to make every tool
> for working with kernel netdevs also have support for dpdk ports.
>
> Yes that's the ultimately best solution.
> But:
> - we need some cooperation of the kernel team
> - ethtool manages a device (what DPDK provides) whereas iproute and others
> manage a TCP/IP stack so is out of control of DPDK.
>

That's not true.
iproute can control a lot of things like MAC address, promiscuous, MTU,
etc. that cannot be controlled with ethtool.
Just compare net_device_ops and ethtool_ops to see the difference.

And the question is not only about tools, it is also about how Linux kernel
works with network devices.
And it uses net_device_ops, not ethtool_ops.


>
> > > Or we do nothing and wait to have more hardware like Mellanox
> supporting a
> > > kernel bifurcated driver approach.
> >
> > Given the lack of other NICs supporting that, I think it could be quite
> a wait! Also, it doesn't work for virtio ports, for pcap ports, or any
> other ports which don't have physical hardware backing them. No reason you
> shouldn't be able to pull stats from all your dpdk ethdevs, not just the
> ones with physical hardware. The same ethdev APIs work for them, so should
> the same tools.
>
> Yes, very good point.
>
> > > > *KNI PMD
> > > > Patch is in the mail list, missing comments. If it gets some
> > > > interest/comments/acks it may go in to next release.
> > >
> > > I'm not against KNI PMD but it looks strange to add more support to an
> old
> > > dying approach.
> >
> > I think the main idea here is to clean up the API - at least for the
> data path. There is no reason why we need special KNI RX/TX functions, when
> ethdev RX/TX functions could do the job. However, at a higher level, the
> more basic requirement is that whatever solution for the data-path to
> kernel from dpdk is, it needs to appear as an ethdev, and not as a special
> library with different APIs, as KNI is now.
>
> Yes I agree to unifiy (and reduce) API.
> Why this PMD is not more commented?
> KNI users should be interested to review it.
> Please dear community, we need more reviews!
>

Re: KNI discussion in userspace event

[Thomas Monjalon]

2016-10-28 20:29, Igor Ryzhov:
> On Fri, Oct 28, 2016 Thomas Monjalon wrote:
> > 2016-10-28 15:51, Richardson, Bruce:
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> > > > 2016-10-28 15:31, Ferruh Yigit:
> > > > > * Remove ethtool support ?
> > > >
> > > > That's the other part of KNI.
> > > > It works only for e1000/ixgbe. That's a niche.
> > >
> > > Yes, it's something we need to remove, but again, we need an
> > > alternative first.
> > >
> > > >
> > > > > Still there is some interest, will keep it. But not able to extend it
> > > > > to other drivers with current design.
> > > >
> > > > It should be removed one day.
> > > > We must seriously think about a generic alternative.
> > > > Either we add DPDK support in ethtool or we create a dpdk-ethtool.
> > > > (or at least a library as the one in examples/).
> > >
> > > I don't view that as a great path forward. Sure, we can do our own
> > > ethtool, but then people will look for ifconfig to work, and "ip" to work,
> > > etc. I view having a kernel proxy module as the best path here as it is
> > > tool agnostic on the userspace side, rather than trying to make every
> > > tool for working with kernel netdevs also have support for dpdk ports.
> >
> > Yes that's the ultimately best solution.
> > But:
> > - we need some cooperation of the kernel team
> > - ethtool manages a device (what DPDK provides) whereas iproute and others
> > manage a TCP/IP stack so is out of control of DPDK.
> 
> That's not true.
> iproute can control a lot of things like MAC address, promiscuous, MTU,
> etc. that cannot be controlled with ethtool.
> Just compare net_device_ops and ethtool_ops to see the difference.

Yes you're right. iproute was not a good example :)

> And the question is not only about tools, it is also about how Linux kernel
> works with network devices.
> And it uses net_device_ops, not ethtool_ops.

What do you mean exactly? I feel you have something in mind.

Re: KNI discussion in userspace event

[Igor Ryzhov]

On Fri, Oct 28, 2016 at 9:40 PM, Thomas Monjalon <thomas.monjalon@6wind.com>
wrote:

> 2016-10-28 20:29, Igor Ryzhov:
> > On Fri, Oct 28, 2016 Thomas Monjalon wrote:
> > > 2016-10-28 15:51, Richardson, Bruce:
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> > > > > 2016-10-28 15:31, Ferruh Yigit:
> > > > > > * Remove ethtool support ?
> > > > >
> > > > > That's the other part of KNI.
> > > > > It works only for e1000/ixgbe. That's a niche.
> > > >
> > > > Yes, it's something we need to remove, but again, we need an
> > > > alternative first.
> > > >
> > > > >
> > > > > > Still there is some interest, will keep it. But not able to
> extend it
> > > > > > to other drivers with current design.
> > > > >
> > > > > It should be removed one day.
> > > > > We must seriously think about a generic alternative.
> > > > > Either we add DPDK support in ethtool or we create a dpdk-ethtool.
> > > > > (or at least a library as the one in examples/).
> > > >
> > > > I don't view that as a great path forward. Sure, we can do our own
> > > > ethtool, but then people will look for ifconfig to work, and "ip" to
> work,
> > > > etc. I view having a kernel proxy module as the best path here as it
> is
> > > > tool agnostic on the userspace side, rather than trying to make every
> > > > tool for working with kernel netdevs also have support for dpdk
> ports.
> > >
> > > Yes that's the ultimately best solution.
> > > But:
> > > - we need some cooperation of the kernel team
> > > - ethtool manages a device (what DPDK provides) whereas iproute and
> others
> > > manage a TCP/IP stack so is out of control of DPDK.
> >
> > That's not true.
> > iproute can control a lot of things like MAC address, promiscuous, MTU,
> > etc. that cannot be controlled with ethtool.
> > Just compare net_device_ops and ethtool_ops to see the difference.
>
> Yes you're right. iproute was not a good example :)
>
> > And the question is not only about tools, it is also about how Linux
> kernel
> > works with network devices.
> > And it uses net_device_ops, not ethtool_ops.
>
> What do you mean exactly? I feel you have something in mind.
>

My main point is that if we want to control DPDK ports from Linux, it
should be done with standard utilities.
Every standard utility like iproute just uses existing Linux kernel
interfaces and kernel in its turn uses net_device_ops to control the device.

For example, you want to set MTU of the network device.
Regardless of the utility you use to do that (even if you write your own),
there are two options – ioctl or netlink.
And regardless of the method you choose,  Linux kernel will then call
"ndo_change_mtu".

Re: KNI discussion in userspace event

[Vincent Jardin]

Le 28 octobre 2016 9:23:06 PM Igor Ryzhov <iryzhov@nfware.com> a écrit :

> On Fri, Oct 28, 2016 at 9:40 PM, Thomas Monjalon <thomas.monjalon@6wind.com>
> wrote:
>
>> 2016-10-28 20:29, Igor Ryzhov:
>> > On Fri, Oct 28, 2016 Thomas Monjalon wrote:
>> > > 2016-10-28 15:51, Richardson, Bruce:
>> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
>> > > > > 2016-10-28 15:31, Ferruh Yigit:
>> > > > > > * Remove ethtool support ?
>> > > > >
>> > > > > That's the other part of KNI.
>> > > > > It works only for e1000/ixgbe. That's a niche.
>> > > >
>> > > > Yes, it's something we need to remove, but again, we need an
>> > > > alternative first.
>> > > >
>> > > > >
>> > > > > > Still there is some interest, will keep it. But not able to
>> extend it
>> > > > > > to other drivers with current design.
>> > > > >
>> > > > > It should be removed one day.
>> > > > > We must seriously think about a generic alternative.
>> > > > > Either we add DPDK support in ethtool or we create a dpdk-ethtool.
>> > > > > (or at least a library as the one in examples/).
>> > > >
>> > > > I don't view that as a great path forward. Sure, we can do our own
>> > > > ethtool, but then people will look for ifconfig to work, and "ip" to
>> work,
>> > > > etc. I view having a kernel proxy module as the best path here as it
>> is
>> > > > tool agnostic on the userspace side, rather than trying to make every
>> > > > tool for working with kernel netdevs also have support for dpdk
>> ports.
>> > >
>> > > Yes that's the ultimately best solution.
>> > > But:
>> > > - we need some cooperation of the kernel team
>> > > - ethtool manages a device (what DPDK provides) whereas iproute and
>> others
>> > > manage a TCP/IP stack so is out of control of DPDK.
>> >
>> > That's not true.
>> > iproute can control a lot of things like MAC address, promiscuous, MTU,
>> > etc. that cannot be controlled with ethtool.
>> > Just compare net_device_ops and ethtool_ops to see the difference.
>>
>> Yes you're right. iproute was not a good example :)
>>
>> > And the question is not only about tools, it is also about how Linux
>> kernel
>> > works with network devices.
>> > And it uses net_device_ops, not ethtool_ops.
>>
>> What do you mean exactly? I feel you have something in mind.
>>
>
> My main point is that if we want to control DPDK ports from Linux, it
> should be done with standard utilities.
> Every standard utility like iproute just uses existing Linux kernel
> interfaces and kernel in its turn uses net_device_ops to control the device.
>
> For example, you want to set MTU of the network device.
> Regardless of the utility you use to do that (even if you write your own),
> there are two options – ioctl or netlink.
> And regardless of the method you choose,  Linux kernel will then call
> "ndo_change_mtu".

DPDK runs on non Linux too (FreeBSD, maybe Windows some days). So we should 
avoid creating such dependencies.

I do like the kind of bifurcated models  (à la RDMA like mlx PMDs) so port 
management is still done using plain OS drivers of the kernel. We should 
rather push toward proper bifurcated drivers:
  - so we decrease maintenance and avoid code duplication
  - we benefit of kernel's interface management

I am not expert of RDMA on FreeBSD and Windows but it seems available.

Re: KNI discussion in userspace event

[Stephen Hemminger]

On Fri, 28 Oct 2016 15:31:50 +0100
Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> Discussed alternatives were:
> * Tun/Tap
> This won't be as fast as KNI and performance is an issue.


That is a myth. Both require the some number of copies.
TUN/TAP copies is a syscall and KNI copies is a kthread.
Actually, the KNI method is worse because it has kernel thread
always running chewing a CPU. I.e it is pure poll mode.

Re: KNI discussion in userspace event

[Aws Ismail]

Hi Stephen, Ferruh,

As an end-user take on this (hence community comment) :), this ties into
the rte_eth_tap that Keith sent out and it has been acked and reviewed, So
I am trying to see the pros/cons of using this (kni pmd) vs. the tun/tap
PMD [1].

Previously, we were using Ferruh's KDP/KCP patches and those served our
purpose, but since the KDP/KCP idea has been rejected as yet another set of
out-of-tree kernel modules to maintain, it has not gotten much attention
since then.

We are hoping that tun/tap would be the way to go just because it looks
simple and easy to manage from a user app perspective.

Having said that, I am not sure what to make of this KNI PMD given
Stephen's comments. Could any one comment about the overall direction of
which solution to focus on? (tun/tap PMD or KNI pmd)?

[1]: http://dpdk.org/dev/patchwork/patch/16566/   (Keith's tun/tap PMD)

Thanks.

Aws\


On Fri, Oct 28, 2016 at 12:25 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Fri, 28 Oct 2016 15:31:50 +0100
> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> > Discussed alternatives were:
> > * Tun/Tap
> > This won't be as fast as KNI and performance is an issue.
>
>
> That is a myth. Both require the some number of copies.
> TUN/TAP copies is a syscall and KNI copies is a kthread.
> Actually, the KNI method is worse because it has kernel thread
> always running chewing a CPU. I.e it is pure poll mode.
>