From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED83DC4708F for ; Wed, 2 Jun 2021 18:09:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D864D610A0 for ; Wed, 2 Jun 2021 18:09:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229697AbhFBSLO (ORCPT ); Wed, 2 Jun 2021 14:11:14 -0400 Received: from mail-mw2nam12on2059.outbound.protection.outlook.com ([40.107.244.59]:61792 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229662AbhFBSLK (ORCPT ); Wed, 2 Jun 2021 14:11:10 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kYJCdRvXAuZDa6maxerhTN1u3WBy3RqRLx8ZUqhrerA4wzBwAsDpDiQ1sNo0ce2+YAydlU1qAisYrJnAbKw1Yny7IMm7kKj+BT7oftspXCTPo5602GEwVV2nI8719ty5vJojI1zARGZlfUkO360C+LYxANu3zEtFoJ3ueL57qLHZ9ry7bS/RDfIMoBnMmNXbwDOsIGkPznduNe6Cq12yD53KyXFdAuCPfIpx9uQ20tGYuoX9jn8o6IWraaUKHrsAeK+97OxljOq4Rxh0K7dCWLoNr0nShnEIx+PUGm8R9Fgib1DzhTANiDohsPPsEvaL2TDMgNAmozOpRGVO5B2vlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q3dalfO5Zk0gfjeCpPpFzHRmwm68ntATfY7FMG/c54o=; b=Tl4dL8ToW/GyHCPe5dyroYBS2IqtTuos4ZEZGN7Srry7aKdeRXYKjD/UIcLsJd45CSlEYmDJ6E2x9t8CeQlnoaYeM6vKQg5Ml0xyQ3dsdEglj3GMXAKsZEAzJ6kIqSCleIhpk44VNBKRYyxA9fgN4j1iPqPkPjyJ8XCLmd+hFL5j28r9aTmZVg8Ui0yBU5dhQS5y9WyShVgZAtjPUiK+DXcvk7/adaaPtvxt6YdZgGVxpvxEgX1A0fGYaEMAidujrJckWL0lrhLyXXcYAgcJjjQ6gUahIaogSZhDZpYNE0FDzmJjjQg3pIPPEsxboUJFYSKpYteIaEVyXmwjjv/taw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q3dalfO5Zk0gfjeCpPpFzHRmwm68ntATfY7FMG/c54o=; b=LleGHzulBlXG5HbJFzikxqAxzIvLAJz2PDWl5/ChHzU7bFORH0dmMmeXyeZfWkI+ej8J9ilZEGUBOiPF4XTnfa/FN+1dTHtLIbOnvTI0mvjaVHI4l7jPWBBCwb98oYdRQN6NKaCRiZO3WGroj+kOFBlFS4Hay07/Gn40uIX3sdCJ7JjQ8XatP06GsUaVAs8orYMqtHPXrFKt4RO3/Ikljw3JeUoL9DncW0aI32GWPSBUK/BORqJ6951pk9ef7XN7FHP78QesvbdSuejUpvX12JsFU2WeBVUFVYE9wxvYDBY7or+L3jW4f5fcy5obZjtve+bOT4jH+Sbmrz5uvg9dpQ== Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=nvidia.com; Received: from BL0PR12MB5506.namprd12.prod.outlook.com (2603:10b6:208:1cb::22) by BL1PR12MB5272.namprd12.prod.outlook.com (2603:10b6:208:319::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4195.20; Wed, 2 Jun 2021 18:09:26 +0000 Received: from BL0PR12MB5506.namprd12.prod.outlook.com ([fe80::3d51:a3b9:8611:684e]) by BL0PR12MB5506.namprd12.prod.outlook.com ([fe80::3d51:a3b9:8611:684e%6]) with mapi id 15.20.4195.020; Wed, 2 Jun 2021 18:09:26 +0000 Date: Wed, 2 Jun 2021 15:09:25 -0300 From: Jason Gunthorpe To: Alex Williamson Cc: "Tian, Kevin" , Jean-Philippe Brucker , "Jiang, Dave" , "Raj, Ashok" , "kvm@vger.kernel.org" , Jonathan Corbet , Robin Murphy , LKML , "iommu@lists.linux-foundation.org" , David Gibson , Kirti Wankhede , David Woodhouse , Jason Wang Subject: Re: [RFC] /dev/ioasid uAPI proposal Message-ID: <20210602180925.GH1002214@nvidia.com> References: <20210528200311.GP1002214@nvidia.com> <20210601162225.259923bc.alex.williamson@redhat.com> <20210602160140.GV1002214@nvidia.com> <20210602111117.026d4a26.alex.williamson@redhat.com> <20210602173510.GE1002214@nvidia.com> <20210602120111.5e5bcf93.alex.williamson@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210602120111.5e5bcf93.alex.williamson@redhat.com> X-Originating-IP: [47.55.113.94] X-ClientProxiedBy: BL1PR13CA0222.namprd13.prod.outlook.com (2603:10b6:208:2bf::17) To BL0PR12MB5506.namprd12.prod.outlook.com (2603:10b6:208:1cb::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (47.55.113.94) by BL1PR13CA0222.namprd13.prod.outlook.com (2603:10b6:208:2bf::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.9 via Frontend Transport; Wed, 2 Jun 2021 18:09:25 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1loVIz-000JWP-3L; Wed, 02 Jun 2021 15:09:25 -0300 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d8666174-6536-4539-9033-08d925f18e97 X-MS-TrafficTypeDiagnostic: BL1PR12MB5272: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:397; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HdBQzsj5KxHF81CJeKOHfi8H8hd/C9JDMUmxf0d1EpUolH5dukBXh0WcbZJwYi9KgBXAOTnlTloXjHz6rxAW0iHl1aw2Fa80MX/XRdPYF9mbU2bVOr7aVH5uqQEAl+sXKeCTwaGaqCQ+VFYi9Mry4FackXUwuOFowsRnhFpfeFhiGuk0kN2lA/JCStNNxt7hCJntYGrQyrVq/DNcI1QjQOMflhhTbgcbv9oSiugRrT7CN/x1wbLbS4JsHhJEZfJm2TgQuN1f32F9uf0Egw3w9TKD8oVEZR+kGDrq9nGaBKivcZrjsCGGdSvD6i/U+xrtey8tbn5HqMFfhdx3mySUDZXsmRnLEYsVqEbtnsWYi02Hiik/ipzzflpAaufRSfOzfml0g32sl8AJz7tYR2PFZqfdO75ZefZ2jSGP8p4p+vk+hNkm3Xa8SOi4YTgdOrc5TJM/AjinrOK5XNceAXmEtY/ss7Rc10yucIylRDl2JXxh2Lss2OU6ptSPyHHNUEiJd6Ef33NLDfL3rXT4noHyh6Z0fyXxr5nrYaXamjh92OJM3+kVaslew5qFXkj0DQYEqnylFJsW6AwIa2tGUuddkFTeMm7H3BKf/sZhmjk3wMo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR12MB5506.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(346002)(136003)(39860400002)(366004)(376002)(396003)(1076003)(9786002)(5660300002)(9746002)(4326008)(6916009)(38100700002)(33656002)(8676002)(8936002)(36756003)(86362001)(7416002)(478600001)(66476007)(66556008)(186003)(2906002)(26005)(2616005)(426003)(54906003)(316002)(83380400001)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?g6+IxBia++5CpL/tSFEKYzPQuaBBv9cw9JfWdVZzKwBomqHmthKy4TGQTsmz?= =?us-ascii?Q?hKNvxaoYHEUmFTccE9ZXcNNXEw+GSw3QFUfLCPzsni9rQGwmCX285CewG7IZ?= =?us-ascii?Q?Jmzq1PFZ1nZl9wwv9Kopr1LE8CPs2LmV6RLdfbSQhwmstJuhL6vDIu4DLHjL?= =?us-ascii?Q?5PsAe49+8CDpd/Rt3hNWdrSlKqEW/lELbNijZ/3y754sROQ3EOg4UYp2Ll1r?= =?us-ascii?Q?R8og2+fykLjLptbmFG71jttRXJ4Yc73FfsfVC3OnbRYTxyDkgClYLgE6BfwC?= =?us-ascii?Q?al35DvIXJuLyCH4ZCJu+MRz4zJVtD4iCU2h1F8d42tnGMQCCel6TghlLMs89?= =?us-ascii?Q?h9An5JQr6dKDe+CJycSi+tC8TwrtPjsNqMpmBdhvbY4FvEDUwkA+3Mr5vcv1?= =?us-ascii?Q?M+CUurM69v1SNSfWEt1VpaqCuQpmtE15K79xwTWZEILQZrbl/Gp9mER639au?= =?us-ascii?Q?MhdhFUca1TpVYlJqZCSLSwcLsuBuw8wjdkbnCiZLsZPQnDNU+ClQQkSDfkD6?= =?us-ascii?Q?znC7Cweas6FjXury9DaOZHDL+m91F/QFNzJoOuXh+CSLdJePyHqUqRdVNtUX?= =?us-ascii?Q?9PwCaMeVVLz48xnQF2qeTix2HlKWRcVEsncLcdrvTEitOD0kZWtJrI1R5XAS?= =?us-ascii?Q?br9EX7QXegJDDMuWDC88wAimPhZgldJT6WLurEYE31KoXrzknTEsjtzF/Npq?= =?us-ascii?Q?t+uujKyckfUk4AbJhmSSIJdYfpctnMWMYEi/SlXlPgFVolgdkmAFzuhTRaIB?= =?us-ascii?Q?lbSLcpFCTeP1t6ubefjjibhK9nUHIJcK2GVLGCqOgTwxnhgRzXBcdJxOaGPx?= =?us-ascii?Q?5nBHFmRXmmxsyujyhb0RbCyCvvbKQ0RAKp/ieFLLaNDUyb7+6HfYxHslZo5V?= =?us-ascii?Q?mk4yka6ocfYSgGL7UQBna8LkjFe6hx8CH2C7DpF6xwLdS8illbl0x58dueHI?= =?us-ascii?Q?IYPGkgVeXzx+DnIdx8M3svpEp3BYHOC6g4VTj2CZFVkrFgt5TfYjxJDH6onn?= =?us-ascii?Q?2s/bKEBRcYx+K9RXyvQogqSmrObAqw9jrIaD3339C/2t6JLB4e7o8g0dd6A4?= =?us-ascii?Q?c8wptS0FqxObUD8JKCdsNP3JrhTmGPfax9wVp7ByB4TMpGscPqWI8+KoWwg6?= =?us-ascii?Q?shGCaqEfom+7WQU4Ym4olybKTe2Fdde6A40KWuK3zTJSFOLZttzVm/Vy16Xw?= =?us-ascii?Q?wAGtugqm2gmmX6u1R5iS2viDwklHOui7rhLuhlYna8AtIrg+03uBmnnoLA1Q?= =?us-ascii?Q?RZA7mkYt05d/dV2OFcWa7Tgk9Le2dxITvfoIgFeWXdExBpYzz155e44s+KfJ?= =?us-ascii?Q?WOGVaFu3dHAtDTaXLrTcKGJ2?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d8666174-6536-4539-9033-08d925f18e97 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2021 18:09:26.0314 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KFZpXywWizMlXmPCZgwMcfwiqtBjyd0lO4XTOxDVp2Xrt7ZdmIxGP3XtWpq2o6PR X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5272 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jun 02, 2021 at 12:01:11PM -0600, Alex Williamson wrote: > On Wed, 2 Jun 2021 14:35:10 -0300 > Jason Gunthorpe wrote: > > > On Wed, Jun 02, 2021 at 11:11:17AM -0600, Alex Williamson wrote: > > > > > > > > present and be able to test if DMA for that device is cache > > > > > > coherent. > > > > > > > > Why is this such a strong linkage to VFIO and not just a 'hey kvm > > > > emulate wbinvd' flag from qemu? > > > > > > IIRC, wbinvd has host implications, a malicious user could tell KVM to > > > emulate wbinvd then run the op in a loop and induce a disproportionate > > > load on the system. We therefore wanted a way that it would only be > > > enabled when required. > > > > I think the non-coherentness is vfio_device specific? eg a specific > > device will decide if it is coherent or not? > > No, this is specifically whether DMA is cache coherent to the > processor, ie. in the case of wbinvd whether the processor needs to > invalidate its cache in order to see data from DMA. I'm confused. This is x86, all DMA is cache coherent unless the device is doing something special. > > If yes I'd recast this to call kvm_arch_register_noncoherent_dma() > > from the VFIO_GROUP_NOTIFY_SET_KVM in the struct vfio_device > > implementation and not link it through the IOMMU. > > The IOMMU tells us if DMA is cache coherent, VFIO_DMA_CC_IOMMU maps to > IOMMU_CAP_CACHE_COHERENCY for all domains within a container. And this special IOMMU mode is basically requested by the device driver, right? Because if you use this mode you have to also use special programming techniques. This smells like all the "snoop bypass" stuff from PCIE (for GPUs even) in a different guise - it is device triggered, not platform triggered behavior. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFAC9C47093 for ; Wed, 2 Jun 2021 18:09:31 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A345761090 for ; Wed, 2 Jun 2021 18:09:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A345761090 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 765794045A; Wed, 2 Jun 2021 18:09:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zAfPt0Rv97oW; Wed, 2 Jun 2021 18:09:30 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id 5F94540460; Wed, 2 Jun 2021 18:09:30 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3CE2BC000E; Wed, 2 Jun 2021 18:09:30 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id BD66BC0001 for ; Wed, 2 Jun 2021 18:09:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id AC15182A72 for ; Wed, 2 Jun 2021 18:09:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=nvidia.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CV0bxR5sa7GW for ; Wed, 2 Jun 2021 18:09:27 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2051.outbound.protection.outlook.com [40.107.244.51]) by smtp1.osuosl.org (Postfix) with ESMTPS id 9584182A56 for ; Wed, 2 Jun 2021 18:09:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kYJCdRvXAuZDa6maxerhTN1u3WBy3RqRLx8ZUqhrerA4wzBwAsDpDiQ1sNo0ce2+YAydlU1qAisYrJnAbKw1Yny7IMm7kKj+BT7oftspXCTPo5602GEwVV2nI8719ty5vJojI1zARGZlfUkO360C+LYxANu3zEtFoJ3ueL57qLHZ9ry7bS/RDfIMoBnMmNXbwDOsIGkPznduNe6Cq12yD53KyXFdAuCPfIpx9uQ20tGYuoX9jn8o6IWraaUKHrsAeK+97OxljOq4Rxh0K7dCWLoNr0nShnEIx+PUGm8R9Fgib1DzhTANiDohsPPsEvaL2TDMgNAmozOpRGVO5B2vlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q3dalfO5Zk0gfjeCpPpFzHRmwm68ntATfY7FMG/c54o=; b=Tl4dL8ToW/GyHCPe5dyroYBS2IqtTuos4ZEZGN7Srry7aKdeRXYKjD/UIcLsJd45CSlEYmDJ6E2x9t8CeQlnoaYeM6vKQg5Ml0xyQ3dsdEglj3GMXAKsZEAzJ6kIqSCleIhpk44VNBKRYyxA9fgN4j1iPqPkPjyJ8XCLmd+hFL5j28r9aTmZVg8Ui0yBU5dhQS5y9WyShVgZAtjPUiK+DXcvk7/adaaPtvxt6YdZgGVxpvxEgX1A0fGYaEMAidujrJckWL0lrhLyXXcYAgcJjjQ6gUahIaogSZhDZpYNE0FDzmJjjQg3pIPPEsxboUJFYSKpYteIaEVyXmwjjv/taw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q3dalfO5Zk0gfjeCpPpFzHRmwm68ntATfY7FMG/c54o=; b=LleGHzulBlXG5HbJFzikxqAxzIvLAJz2PDWl5/ChHzU7bFORH0dmMmeXyeZfWkI+ej8J9ilZEGUBOiPF4XTnfa/FN+1dTHtLIbOnvTI0mvjaVHI4l7jPWBBCwb98oYdRQN6NKaCRiZO3WGroj+kOFBlFS4Hay07/Gn40uIX3sdCJ7JjQ8XatP06GsUaVAs8orYMqtHPXrFKt4RO3/Ikljw3JeUoL9DncW0aI32GWPSBUK/BORqJ6951pk9ef7XN7FHP78QesvbdSuejUpvX12JsFU2WeBVUFVYE9wxvYDBY7or+L3jW4f5fcy5obZjtve+bOT4jH+Sbmrz5uvg9dpQ== Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=nvidia.com; Received: from BL0PR12MB5506.namprd12.prod.outlook.com (2603:10b6:208:1cb::22) by BL1PR12MB5272.namprd12.prod.outlook.com (2603:10b6:208:319::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4195.20; Wed, 2 Jun 2021 18:09:26 +0000 Received: from BL0PR12MB5506.namprd12.prod.outlook.com ([fe80::3d51:a3b9:8611:684e]) by BL0PR12MB5506.namprd12.prod.outlook.com ([fe80::3d51:a3b9:8611:684e%6]) with mapi id 15.20.4195.020; Wed, 2 Jun 2021 18:09:26 +0000 Date: Wed, 2 Jun 2021 15:09:25 -0300 From: Jason Gunthorpe To: Alex Williamson Subject: Re: [RFC] /dev/ioasid uAPI proposal Message-ID: <20210602180925.GH1002214@nvidia.com> References: <20210528200311.GP1002214@nvidia.com> <20210601162225.259923bc.alex.williamson@redhat.com> <20210602160140.GV1002214@nvidia.com> <20210602111117.026d4a26.alex.williamson@redhat.com> <20210602173510.GE1002214@nvidia.com> <20210602120111.5e5bcf93.alex.williamson@redhat.com> Content-Disposition: inline In-Reply-To: <20210602120111.5e5bcf93.alex.williamson@redhat.com> X-Originating-IP: [47.55.113.94] X-ClientProxiedBy: BL1PR13CA0222.namprd13.prod.outlook.com (2603:10b6:208:2bf::17) To BL0PR12MB5506.namprd12.prod.outlook.com (2603:10b6:208:1cb::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (47.55.113.94) by BL1PR13CA0222.namprd13.prod.outlook.com (2603:10b6:208:2bf::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.9 via Frontend Transport; Wed, 2 Jun 2021 18:09:25 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1loVIz-000JWP-3L; Wed, 02 Jun 2021 15:09:25 -0300 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d8666174-6536-4539-9033-08d925f18e97 X-MS-TrafficTypeDiagnostic: BL1PR12MB5272: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:397; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HdBQzsj5KxHF81CJeKOHfi8H8hd/C9JDMUmxf0d1EpUolH5dukBXh0WcbZJwYi9KgBXAOTnlTloXjHz6rxAW0iHl1aw2Fa80MX/XRdPYF9mbU2bVOr7aVH5uqQEAl+sXKeCTwaGaqCQ+VFYi9Mry4FackXUwuOFowsRnhFpfeFhiGuk0kN2lA/JCStNNxt7hCJntYGrQyrVq/DNcI1QjQOMflhhTbgcbv9oSiugRrT7CN/x1wbLbS4JsHhJEZfJm2TgQuN1f32F9uf0Egw3w9TKD8oVEZR+kGDrq9nGaBKivcZrjsCGGdSvD6i/U+xrtey8tbn5HqMFfhdx3mySUDZXsmRnLEYsVqEbtnsWYi02Hiik/ipzzflpAaufRSfOzfml0g32sl8AJz7tYR2PFZqfdO75ZefZ2jSGP8p4p+vk+hNkm3Xa8SOi4YTgdOrc5TJM/AjinrOK5XNceAXmEtY/ss7Rc10yucIylRDl2JXxh2Lss2OU6ptSPyHHNUEiJd6Ef33NLDfL3rXT4noHyh6Z0fyXxr5nrYaXamjh92OJM3+kVaslew5qFXkj0DQYEqnylFJsW6AwIa2tGUuddkFTeMm7H3BKf/sZhmjk3wMo= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB5506.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(136003)(39860400002)(366004)(376002)(396003)(1076003)(9786002)(5660300002)(9746002)(4326008)(6916009)(38100700002)(33656002)(8676002)(8936002)(36756003)(86362001)(7416002)(478600001)(66476007)(66556008)(186003)(2906002)(26005)(2616005)(426003)(54906003)(316002)(83380400001)(66946007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?g6+IxBia++5CpL/tSFEKYzPQuaBBv9cw9JfWdVZzKwBomqHmthKy4TGQTsmz?= =?us-ascii?Q?hKNvxaoYHEUmFTccE9ZXcNNXEw+GSw3QFUfLCPzsni9rQGwmCX285CewG7IZ?= =?us-ascii?Q?Jmzq1PFZ1nZl9wwv9Kopr1LE8CPs2LmV6RLdfbSQhwmstJuhL6vDIu4DLHjL?= =?us-ascii?Q?5PsAe49+8CDpd/Rt3hNWdrSlKqEW/lELbNijZ/3y754sROQ3EOg4UYp2Ll1r?= =?us-ascii?Q?R8og2+fykLjLptbmFG71jttRXJ4Yc73FfsfVC3OnbRYTxyDkgClYLgE6BfwC?= =?us-ascii?Q?al35DvIXJuLyCH4ZCJu+MRz4zJVtD4iCU2h1F8d42tnGMQCCel6TghlLMs89?= =?us-ascii?Q?h9An5JQr6dKDe+CJycSi+tC8TwrtPjsNqMpmBdhvbY4FvEDUwkA+3Mr5vcv1?= =?us-ascii?Q?M+CUurM69v1SNSfWEt1VpaqCuQpmtE15K79xwTWZEILQZrbl/Gp9mER639au?= =?us-ascii?Q?MhdhFUca1TpVYlJqZCSLSwcLsuBuw8wjdkbnCiZLsZPQnDNU+ClQQkSDfkD6?= =?us-ascii?Q?znC7Cweas6FjXury9DaOZHDL+m91F/QFNzJoOuXh+CSLdJePyHqUqRdVNtUX?= =?us-ascii?Q?9PwCaMeVVLz48xnQF2qeTix2HlKWRcVEsncLcdrvTEitOD0kZWtJrI1R5XAS?= =?us-ascii?Q?br9EX7QXegJDDMuWDC88wAimPhZgldJT6WLurEYE31KoXrzknTEsjtzF/Npq?= =?us-ascii?Q?t+uujKyckfUk4AbJhmSSIJdYfpctnMWMYEi/SlXlPgFVolgdkmAFzuhTRaIB?= =?us-ascii?Q?lbSLcpFCTeP1t6ubefjjibhK9nUHIJcK2GVLGCqOgTwxnhgRzXBcdJxOaGPx?= =?us-ascii?Q?5nBHFmRXmmxsyujyhb0RbCyCvvbKQ0RAKp/ieFLLaNDUyb7+6HfYxHslZo5V?= =?us-ascii?Q?mk4yka6ocfYSgGL7UQBna8LkjFe6hx8CH2C7DpF6xwLdS8illbl0x58dueHI?= =?us-ascii?Q?IYPGkgVeXzx+DnIdx8M3svpEp3BYHOC6g4VTj2CZFVkrFgt5TfYjxJDH6onn?= =?us-ascii?Q?2s/bKEBRcYx+K9RXyvQogqSmrObAqw9jrIaD3339C/2t6JLB4e7o8g0dd6A4?= =?us-ascii?Q?c8wptS0FqxObUD8JKCdsNP3JrhTmGPfax9wVp7ByB4TMpGscPqWI8+KoWwg6?= =?us-ascii?Q?shGCaqEfom+7WQU4Ym4olybKTe2Fdde6A40KWuK3zTJSFOLZttzVm/Vy16Xw?= =?us-ascii?Q?wAGtugqm2gmmX6u1R5iS2viDwklHOui7rhLuhlYna8AtIrg+03uBmnnoLA1Q?= =?us-ascii?Q?RZA7mkYt05d/dV2OFcWa7Tgk9Le2dxITvfoIgFeWXdExBpYzz155e44s+KfJ?= =?us-ascii?Q?WOGVaFu3dHAtDTaXLrTcKGJ2?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d8666174-6536-4539-9033-08d925f18e97 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2021 18:09:26.0314 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KFZpXywWizMlXmPCZgwMcfwiqtBjyd0lO4XTOxDVp2Xrt7ZdmIxGP3XtWpq2o6PR X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5272 Cc: Jean-Philippe Brucker , "Tian, Kevin" , "Jiang, Dave" , "Raj, Ashok" , "kvm@vger.kernel.org" , Jonathan Corbet , David Woodhouse , Jason Wang , LKML , Kirti Wankhede , "iommu@lists.linux-foundation.org" , Robin Murphy , David Gibson X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Wed, Jun 02, 2021 at 12:01:11PM -0600, Alex Williamson wrote: > On Wed, 2 Jun 2021 14:35:10 -0300 > Jason Gunthorpe wrote: > > > On Wed, Jun 02, 2021 at 11:11:17AM -0600, Alex Williamson wrote: > > > > > > > > present and be able to test if DMA for that device is cache > > > > > > coherent. > > > > > > > > Why is this such a strong linkage to VFIO and not just a 'hey kvm > > > > emulate wbinvd' flag from qemu? > > > > > > IIRC, wbinvd has host implications, a malicious user could tell KVM to > > > emulate wbinvd then run the op in a loop and induce a disproportionate > > > load on the system. We therefore wanted a way that it would only be > > > enabled when required. > > > > I think the non-coherentness is vfio_device specific? eg a specific > > device will decide if it is coherent or not? > > No, this is specifically whether DMA is cache coherent to the > processor, ie. in the case of wbinvd whether the processor needs to > invalidate its cache in order to see data from DMA. I'm confused. This is x86, all DMA is cache coherent unless the device is doing something special. > > If yes I'd recast this to call kvm_arch_register_noncoherent_dma() > > from the VFIO_GROUP_NOTIFY_SET_KVM in the struct vfio_device > > implementation and not link it through the IOMMU. > > The IOMMU tells us if DMA is cache coherent, VFIO_DMA_CC_IOMMU maps to > IOMMU_CAP_CACHE_COHERENCY for all domains within a container. And this special IOMMU mode is basically requested by the device driver, right? Because if you use this mode you have to also use special programming techniques. This smells like all the "snoop bypass" stuff from PCIE (for GPUs even) in a different guise - it is device triggered, not platform triggered behavior. Jason _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu