From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 875BBC54E4A for ; Fri, 8 May 2020 20:47:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6DD2E2184D for ; Fri, 8 May 2020 20:47:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727095AbgEHUrM (ORCPT ); Fri, 8 May 2020 16:47:12 -0400 Received: from mga01.intel.com ([192.55.52.88]:15188 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726817AbgEHUrM (ORCPT ); Fri, 8 May 2020 16:47:12 -0400 IronPort-SDR: DuKItoW6HSsPJmI6EKlOM9RKO+Kz3gxu7ArplrGslXFAMo/Iqwse1lpMyHjc9Ogy09IVVIB44U BimMnZV9PeWQ== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2020 13:47:11 -0700 IronPort-SDR: Ws67Aagq7YgG2snBGVk1C1ZIy3paurbwnrKJK/rpqD2iq5oG5Lx0moxzX7x1HOFOuxtYO7W/k8 GGimup1grgVg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,369,1583222400"; d="scan'208";a="296258718" Received: from otc-nc-03.jf.intel.com (HELO otc-nc-03) ([10.54.39.25]) by fmsmga002.fm.intel.com with ESMTP; 08 May 2020 13:47:10 -0700 Date: Fri, 8 May 2020 13:47:10 -0700 From: "Raj, Ashok" To: "Tian, Kevin" Cc: Jason Gunthorpe , Alex Williamson , "Jiang, Dave" , "vkoul@kernel.org" , "megha.dey@linux.intel.com" , "maz@kernel.org" , "bhelgaas@google.com" , "rafael@kernel.org" , "gregkh@linuxfoundation.org" , "tglx@linutronix.de" , "hpa@zytor.com" , "Pan, Jacob jun" , "Liu, Yi L" , "Lu, Baolu" , "Kumar, Sanjay K" , "Luck, Tony" , "Lin, Jing" , "Williams, Dan J" , "kwankhede@nvidia.com" , "eric.auger@redhat.com" , "parav@mellanox.com" , "dmaengine@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "x86@kernel.org" , "linux-pci@vger.kernel.org" , "kvm@vger.kernel.org" , Ashok Raj Subject: Re: [PATCH RFC 00/15] Add VFIO mediated device support and IMS support for the idxd driver. Message-ID: <20200508204710.GA78778@otc-nc-03> References: <20200424124444.GJ13640@mellanox.com> <20200424181203.GU13640@mellanox.com> <20200426191357.GB13640@mellanox.com> <20200426214355.29e19d33@x1.home> <20200427115818.GE13640@mellanox.com> <20200427071939.06aa300e@x1.home> <20200427132218.GG13640@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Hi Jason In general your idea of moving pure emulation code to user space is a good strategy. On Wed, Apr 29, 2020 at 02:42:20AM -0700, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Monday, April 27, 2020 9:22 PM > > > > On Mon, Apr 27, 2020 at 07:19:39AM -0600, Alex Williamson wrote: > > > > > > It is not trivial masking. It is a 2000 line patch doing comprehensive > > > > emulation. > > > > > > Not sure what you're referring to, I see about 30 lines of code in > > > vdcm_vidxd_cfg_write() that specifically handle writes to the 4 BARs in > > > config space and maybe a couple hundred lines of code in total handling > > > config space emulation. Thanks, > > > > Look around vidxd_do_command() > > > > If I understand this flow properly.. > > > > Hi, Jason, > > I guess the 2000 lines mostly refer to the changes in mdev.c and vdev.c. > We did a break-down among them: > > 1) ~150 LOC for vdev initialization > 2) ~150 LOC for cfg space emulation > 3) ~230 LOC for mmio r/w emulation > 4) ~500 LOC for controlling the work queue (vidxd_do_command), > triggered by write emulation of IDXD_CMD_OFFSET register > 5) the remaining lines are all about vfio-mdev registration/callbacks, > for reporting mmio/irq resource, eventfd, mmap, etc. > > 1/2/3) are pure device emulation, which counts for ~500 LOC. > > 4) needs be in the kernel regardless of which uAPI is used, because it > talks to the physical work queue (enable, disable, drain, abort, reset, etc.) > > Then if just talking about ~500 LOC emulation code left in the kernel, > is it still a big concern to you? 😊 Even when uaccel was under development, one of the options was to use VFIO as the transport, goal was the same i.e to keep the user space have one interface. But the needs of generic user space application is significantly different from exporting a more functional device model to guest, which isn't full emulated device. which is why VFIO didn't make sense for native use. And when we move things from VFIO which is already established as a general device model and accepted by multiple VMM's it gives instant footing without a whole redesign. When we move things from VFIO to uaccel to bolt on the functionality like VFIO, I suspect we would be moving code/functionality from VFIO to Uaccel. I don't know what the net gain would be. IMS is being reworked based on your feedback. And for mdev since the code is minimal for emulation, and rest are control paths that need kernel code to deal with. For mdev, would you agree we can keep the current architecture, and investigate moving some emulation code to user space (say even for standard vfio_pci) and then expand scope later. Cheers Ashok