From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=0.2 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,URIBL_SBL,URIBL_SBL_A autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26D70C76188 for ; Tue, 16 Jul 2019 09:57:05 +0000 (UTC) Received: from dpdk.org (dpdk.org [92.243.14.124]) by mail.kernel.org (Postfix) with ESMTP id 8147E20880 for ; Tue, 16 Jul 2019 09:57:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=marvell.com header.i=@marvell.com header.b="UI1WHM0x"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=marvell.onmicrosoft.com header.i=@marvell.onmicrosoft.com header.b="SeLETlsq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8147E20880 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=marvell.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AC0DF3423; Tue, 16 Jul 2019 11:57:03 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 1ABD22C6A for ; Tue, 16 Jul 2019 11:57:01 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x6G9v0fB031561; Tue, 16 Jul 2019 02:57:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=5py3WpWS6gPy3bg+3e2LkCOGFq2nwL3ey9ATJqZJZ/8=; b=UI1WHM0xHJOMCTHUdo6fpJzWxkrGSjxRu3glptuLh7yxYJR1ebzzV+N4yDAJPU5Oegag yFKGzR3CMUfqBLiy6rBxrqwrHJKHEhcySsmlV1UE7nrV2KRVm9NP7s1XGC7JtW/vjlke +6DNUNG0Q7gRFnfwjPo0f8pFsN4AZK6ascZRx7CVskl7IVX08eUGBuMwxeKyL64ObcyM qLNe+sNVnszla48qLpc8Xe6C89KqMUZX/AZL8rxSLuZycBrVCIrDtbKmpWlkqDVjCIps KcdMry2JDj1Co4DOsBQ7yWdRgSeJJ3kKB7yOI9ajCyKBxYMyuXzogJwQwrDMQBlh8yfR dg== Received: from sc-exch02.marvell.com ([199.233.58.182]) by mx0b-0016f401.pphosted.com with ESMTP id 2ts0a22h9t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Tue, 16 Jul 2019 02:57:00 -0700 Received: from SC-EXCH04.marvell.com (10.93.176.84) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 16 Jul 2019 02:56:18 -0700 Received: from NAM05-BY2-obe.outbound.protection.outlook.com (104.47.50.54) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Tue, 16 Jul 2019 02:56:17 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nGz9NZ97mQHDUR6jHE+f20nneZILj3Ylo6cuMoJYeihq8cwgkQeDdu4kwiN5kCQSm6FZ0EJCmy+mD7zqEBgr7cvFDArAyCgW8BNi615qqwc9RURxCEto4dkf54lLvodV4udFhxzsU4a8kCihkS+pfeK+v+pwkruLKKWAcmA3ywOWPydblScqmOQORw8yg5446wbDyPBAe3DzdvDzlJJE5Ii3XNB7DrHVEqkYK9Yyz1GT1WwBJ2Uyxh+EzlkFZw4TQEVIYI18ntL3M1PsYbQJOYwgciVh5BPwDex6eXqthjfrKnUCE9/SLxeAAOyYmTTfn/MKiQ59AuNWjoYARZ2QFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5py3WpWS6gPy3bg+3e2LkCOGFq2nwL3ey9ATJqZJZ/8=; b=bHixRj/5+DN8PWpGgV1c57UTx6sfyME7e03DkH32rM0V/LVUunFXse4rVKUaL6n5xr4B2pORaHnV0UHVK2DiXYxpC6KrpSR5nhZGsFyPGFWQ3zjKLkjqjIXcFamOSTxvOiRKKlrF0WQr73s8nysJvDgEgIO6f/kQoSl4i8M8MeGi8kMV8TqkssdF+E1s2YUl2GylAkF6pofc1zs1qGNuLTftCol2+VDCONH36BZ+xf0zCzteI71dxDbUqptqmhtXJqGVeeOLinh9oj3zO7vKh0Go5KQ7Fy8YNg8EOaJ+rHMYNCTMRuZ4E63vnoxIDSxXrng41YUJZ1WHC6Uf0rmHJw== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=marvell.com;dmarc=pass action=none header.from=marvell.com;dkim=pass header.d=marvell.com;arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector2-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5py3WpWS6gPy3bg+3e2LkCOGFq2nwL3ey9ATJqZJZ/8=; b=SeLETlsqTu00udxc+UTTza2qnRj8xyaScDU9DCQNhvOFRLDYTET39/7iGn3MfnJkIGztBSqqmqX5EWGfeBCK8mpgWWZCcImQhKbPoSbhPRmsnKupr5hDCKUzIVbDlpG7ipT4EosStWMDvbP8RtWDbbYE0B7wla//wLDWgMgvnr8= Received: from BYAPR18MB2424.namprd18.prod.outlook.com (20.179.91.149) by BYAPR18MB3015.namprd18.prod.outlook.com (20.179.94.161) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2073.14; Tue, 16 Jul 2019 09:56:12 +0000 Received: from BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::2d42:12b6:aa2e:2862]) by BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::2d42:12b6:aa2e:2862%4]) with mapi id 15.20.2073.012; Tue, 16 Jul 2019 09:56:12 +0000 From: Jerin Jacob Kollanukkaran To: "Hyong Youb Kim (hyonkim)" , David Marchand , Thomas Monjalon , "Ferruh Yigit" , Alejandro Lucero , Anatoly Burakov CC: "dev@dpdk.org" , "John Daley (johndale)" , Shahed Shaikh , "Nithin Kumar Dabilpuram" Thread-Topic: [RFC PATCH] vfio: avoid re-installing irq handler Thread-Index: AdU7LOBENm/7fWlUSVuvO2rop1WEPwAbdfHgAAD0wuAAAgb8YAAERxzA Date: Tue, 16 Jul 2019 09:56:12 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [106.200.248.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1621032c-db7a-4400-0f66-08d709d3d60a x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:BYAPR18MB3015; x-ms-traffictypediagnostic: BYAPR18MB3015: x-ms-exchange-purlcount: 1 x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 0100732B76 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(366004)(39860400002)(396003)(346002)(136003)(376002)(189003)(199004)(13464003)(3846002)(81166006)(6436002)(81156014)(446003)(107886003)(53936002)(256004)(6246003)(99286004)(8676002)(6506007)(53546011)(86362001)(966005)(8936002)(6116002)(25786009)(5660300002)(76116006)(316002)(64756008)(11346002)(476003)(66556008)(66946007)(55016002)(6306002)(66446008)(66476007)(478600001)(9686003)(71200400001)(186003)(54906003)(14444005)(4326008)(33656002)(26005)(66066001)(2906002)(229853002)(76176011)(7696005)(52536014)(102836004)(74316002)(486006)(7736002)(305945005)(14454004)(68736007)(110136005)(71190400001); DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR18MB3015; H:BYAPR18MB2424.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Z9R/6aN3LeTHKW7+h4YVMQytZqW2aGgClxLHE890crgwBxTLefEr7ATQNcg3k0Qd0Wt+SMasisPw0llWutnj9+/EGs5F8T3eXybbAAwRJFw13XajYdwhMlklUn6ke0WoRnbFiQCrKFzGLZ/ow/C0FdpFzF5bQj5jA0uEOBRH1tEHvp8pQNCCYUd2M1LhQgyM/Qoa2U1HoxWKZnOSfsReWev8NDyXhBad9QueuIzN6PNl5Eq7NbBUpg8S/j0Cubu3q/Qd0eQZJnIfl0NJFeN83ni0+2N2BF5C3k+lSSxidzqS9xQw2jM9++auAOt8pjI+i1tJncTEcQSRw2Bn3fw8pBNzw0ob+Bfe/yBY4QKa1hB6vmlNnCbE1CpHcnjnnIxjIitwgbQI3Ck6WlQTBLo693ny/R4uxcJNWXQfuhOr8Y4= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 1621032c-db7a-4400-0f66-08d709d3d60a X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jul 2019 09:56:12.8048 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: jerinj@marvell.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR18MB3015 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:5.22.84,1.0.8 definitions=2019-07-16_02:2019-07-16,2019-07-16 signatures=0 Subject: Re: [dpdk-dev] [RFC PATCH] vfio: avoid re-installing irq handler X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Hyong Youb Kim (hyonkim) > Sent: Tuesday, July 16, 2019 1:19 PM > To: Jerin Jacob Kollanukkaran ; David Marchand > ; Thomas Monjalon > ; Ferruh Yigit ; Alejandro > Lucero ; Anatoly Burakov > > Cc: dev@dpdk.org; John Daley (johndale) ; Shahed > Shaikh ; Nithin Kumar Dabilpuram > > Subject: RE: [RFC PATCH] vfio: avoid re-installing irq handler >=20 > > -----Original Message----- > > From: Jerin Jacob Kollanukkaran > [...] > > > > > A rough patch for the approach mentioned earlier. It is only for > > > discussion. > > > > > http://mails.dpdk.org/archives/dev/2019-July/138113.html > > > > > > > > > > To try this out, first revert the following then apply. > > > > > commit 89aac60e0be9 ("vfio: fix interrupts race condition") > > > > > > > > Yes. This patch has to be to reverted. It changes the existing > > > > interrupt behavior and does not address the MSIX case as well. > > > > > > > > I think, The clean fix would be to introduce rte_intr_mask() and > > > > rte_intr_unmask() by abstracting the INTX and MSIX differences And > > > > let qede driver call it as needed. > > > > > > > > Thoughts? > > > > > > Hi, > > > > Hi Hyong, > > > > > > > > You are proposing these? > > > - Add rte_intr_mask_intx, rte_intr_unmask_intx. > > > No APIs for masking MSI/MSI-X as vfio-pci does not support that. > > > - Modify PMD irq handlers to use rte_intr_unmask_intx as necessary. > > > > No, introduce the rte_intr_mask() and rte_intr_unmask(). > > For MSIX + Linux VFIO, That API can return -ENOSUP as Linux VFIO+MSIX > > is not supporting. > > Another platform/eal may support it. > > >=20 > These generic names would invite people to use API, only to see it fail, = since > it only works with INTx.. It works for all non VFIO MSIx case. VFIO MSIx case it NOP(Yes, No need to = return error in that case) >=20 > > Mask and unmask is operation is known to all IRQ controllers. > > So, IMO, As far as abstraction is concerned it will be good fit. > > > > > That might be too intrusive. And too much work for the sake of INTx.. > > > Anyone really using/needing INTx these days? :-) > > > > Yup. Mask needs to called only for only qede INTx. Looks like qede Has > > MSIX and INTX separate handler. So this mask can go to qede INTx > > > > > > > > The following drivers call rte_intr_enable from their irq handlers. > > > So with explicit rte_intr_unmask_intx, all these would need to do > > > "if using intx, unmask"? > > > > > > atlantic, avp, axgbe, bnx2x, e1000, fm10k, ice, ixgbe, nfp, qede, > > > sfc, > > > vmxnet3 > > > > No change on these PMDs. > > >=20 > Why is that? >=20 > These drivers potentially have the same "lost" interrupt issue mentioned = in > the original redhat bz (qede + MSI). I *think* this observation led David= to > address them all through vfio changes, rather than fixing qede alone. >=20 > You want to introduce unmask API and use it only for qede in this cycle, = and > ask respective maintainers to fix their drivers in 19.11? Changing the rte_intr_enable() to rte_intr_unmask() is trivial on the plac= es Where existing drivers enable as unmask. If we understand it correctly: In case of non VFIO MSIX(INTx) and UIO ------------------------------------------------- AK1) Kernel receives interrupt AK2) Kernel _mask_ the interrupt AK3) Kernel notify the use space On usersapce: AU1) Driver specific interrupt handler invoked AU2) Handle the driver specific interrupt AU3) Call rte_intr_enable(), it will intern call VFIO_IRQ_SET_ACTION_UNMAS= K using VFIO_DEVICE_SET_IRQS to unmask the interrupt. In case of VFIO MSIX(INTx) ------------------------------------ BK1) Kernel receives interrupt. BK2) Kernel notify the use space On usersapce: BU1) Driver specific interrupt handler invoked BU2) Handle the driver specific interrupt BU3) Call rte_intr_enable(), it will intern call VFIO_IRQ_SET_ACTION_TRIGG= ER using VFIO_DEVICE_SET_IRQS to unmask the interrupt. VFIO_IRQ_SET_ACTION_TRIGGER: is the nasty one, it will free the existing in= terrupt handler and request new handler etc. Which is the source of the actual race conditional problem.=20 Ideally BU3 can be just NOP. Since we need to keep the same interrupt handl= er for both UIO and MSIX(I *think*) DPDK tends to use rte_intr_enable() which can work for AU3/BU3 as well. So we need, A light weight primitive, which unmask the AK2 incase of VFIO INTx by not o= verriding=20 The meaning of normal rte_intr_enable() which suppose not use for MSIX inte= rrupt in action due to racy behavior of VFIO_IRQ_SET_ACTION_TRIGGER Replacing AU3 and BU3 as rte_intr_unmask() would fix problem. Where rte_int= r_unmask() for BU3 is just NOP.