From mboxrd@z Thu Jan 1 00:00:00 1970 From: John Fastabend Subject: Re: [PATCH v5 00/29] Support VFD and DPDK PF + kernel VF on i40e Date: Fri, 13 Jan 2017 15:04:03 -0800 Message-ID: <58795CE3.4080403@gmail.com> References: <4341B239C0EFF9468EE453F9E9F4604D3C5CA3DA@shsmsx102.ccr.corp.intel.com> <586D647A.5040607@research.att.com> <4341B239C0EFF9468EE453F9E9F4604D3C5D9598@shsmsx102.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Cc: "kaustubh@research.att.com" , "az5157@att.com" , "Fastabend, John R" To: "Chen, Jing D" , Vincent JARDIN , Scott Daniels , "dev@dpdk.org" Return-path: Received: from mail-pf0-f193.google.com (mail-pf0-f193.google.com [209.85.192.193]) by dpdk.org (Postfix) with ESMTP id 516052BF3 for ; Sat, 14 Jan 2017 00:04:17 +0100 (CET) Received: by mail-pf0-f193.google.com with SMTP id f144so10111684pfa.2 for ; Fri, 13 Jan 2017 15:04:17 -0800 (PST) In-Reply-To: <4341B239C0EFF9468EE453F9E9F4604D3C5D9598@shsmsx102.ccr.corp.intel.com> List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 17-01-11 07:47 PM, Chen, Jing D wrote: > Hi, Vincent, > >> -----Original Message----- >> From: Vincent JARDIN [mailto:vincent.jardin@6wind.com] >> Sent: Tuesday, January 10, 2017 9:30 PM >> To: Scott Daniels ; dev@dpdk.org >> Cc: kaustubh@research.att.com; az5157@att.com; Chen, Jing D >> >> Subject: Re: [dpdk-dev] [PATCH v5 00/29] Support VFD and DPDK PF + kernel VF on >> i40e >> >> Hi Scott, >> >> Le 04/01/2017 à 22:09, Scott Daniels a écrit : >>> With holidays we are a bit late with our thoughts, but would like to >>> toss them into the mix. >> >> Same, I hope I am not missing emails. I do appreciate your arguments, it >> provides lot of light. See below, >> >>> The original NAK is understandable, however having the ability to >>> configure the PF via DPDK is advantageous for several reasons: >>> >>> 1) While some functions may be duplicated and/or available from the kernel >>> driver, it is often not possible to introduce new kernel drivers into >>> production without a large amount of additional testing of the entire >>> platform which can cause a significant delay when introducing a DPDK based >>> product. If the PF control is a part of the DPDK environment, then only >>> the application needs to pass the operational testing before deployment; a >>> much more simple task. >> >> So we agree: you confirm that your foresee the benefits of using DPDK to >> *bypass the role of the Kernel being the PF* of reference for the >> hypervisor. >> >>> 2) If the driver changes are upstreamed into the kernel proper, the >>> difficulty of operational readiness testing increases as a new kernel is >>> introduced, and further undermines the ability to quickly and easily >>> release a DPDK based application into production. While the application >>> may eventually fall back on driver and/or kernel support, this could be >>> years away. >> >> I do agree with the benefits of the agilities and the upsides it brings. >> But they are other options to get the same agility without creating a >> fragmentation of PFs. >> >> For example, you do not have to update the whole kernel, you can just >> update the PF kernel module that can be upgraded with the latest needed >> features. >> >>> 3) As DPDK is being used to configure the NIC, it just seems to make >>> sense, for consistency, that the configuration capabilities should include >>> the ability to configure the PF as is proposed. >> >> From this perspective, the kernel modules are fine for the PF: most >> kernels of hypervisors support it without the need to upgrade their kernels. >> >> To summarize, I understand that you need a flexible way to upgrade PF >> features without touching/changing the kernel. So let's check the kernel >> module option? VFD brings some interesting capabilities, could it be a >> way to push and stimulate the i40e features instead of using DPDK? >> >> https://sourceforge.net/projects/e1000/files/i40e%20stable/ >> for instance could be better stimulated. > I figured I would weight in with my $0.02. > May I ask what if DPDK VF need a new extension function from PF? > Then, we'll have to build kernel community expertise and submit > patch to Linux PF and wait for merge. Correct, but it is the same story on DPDK repository as well. Either way patch must be generated, tested, submitted and wait until it is merged. At this point I think even the code is very similar sans style and naming conventions for these types of features. > Your proposal indicates DPDK community submitters will have to > ask Linux community to authorize if we'll have any requirements > in DPDK VF. I don't think this is really much of a problem in most cases the same company maintains both DPDK and Linux kernel drivers so hopefully everyone is aligned. > Comparing fragmentation, the extra dependency worry me most. > Can you imagine how long it will be for any VF features gets > ready? > > The Linux kernel releases every 6-8 weeks roughly depending on the release. In networking stack land features must be submitted for that kernel release before the first RC release. So although there may be benefits to pushing these features into DPDK vs Linux kernel I don't think actually getting the code in the kernel is one of them. To the point above about out of tree drivers. Most vendors do keep out of tree drivers with the latest features and fixes that can be supported on older kernels. Unfortunately, many of the interfaces like the type mentioned here would require kernel updates outside the drivers to add message handlers and driver callbacks for the new config messages. So it may not be viable at least unless we restructure some of the kernel to use out of tree drivers in the short term. So I'm sympathetic to argument (1) here. If _not_ updating the kernel is a hard requirement then we are a bit stuck from using existing/future Linux features. Further this is sort of a rolling problem as well. Today we are missing feature X and we add it and tomorrow we are missing a new feature Y. So if the feature list is not stable with deployment of new kernels its a bit hard to see how the Linux kernel PF model works without above mentioned restructuring. FWIW traditionally the Linux vendors handled this backporting for its customers and provide kernel updates using apt-get/dnf/whatever along with all the sanity testing to ensure they didn't break something along the way. We are slowly getting over this with work in BPF and other subsystems but the VF configuration is still very much hard coded for the moment. I might try to fix this though as a result of this thread. Open question for me on this though is the PF in these SRIOV cases ever being used by DPDK datapath or is it just being leveraged for configuration with primarily control traffic? So my take away from this is arguments about its not available in Linux kernel are not really good ones because its easy enough to add and get in stable kernels. But if kernel updates are a challenge then we either need a more robust kernel backporting and out of tree driver effort _or_ push them directly into PF driver of DPDK. I think longterm we can fix this in the kernel with some restructuring and better module updates but in the short term DPDK PMD updates seem to be the only option. I do think though if this gets pushed into DPDK PMD we shouldn't let the PF Linux driver simply not support the new mailbox messages. This will create one more dependency then I care to worry about. VF<->PF mailbox should work with any combination of VF and PF as long as version requirements are met. I think vendor maintainers can manage this. I'll comment on the specific features in another email because it is helpful to have a list in public mailing list with features folks are interested in. Any chance we could get a page up with a list of all these and where/when we expect support for them? Thanks, John john.r.fastabend@intel.com