From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60712C433ED for ; Tue, 4 May 2021 18:00:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2D95561076 for ; Tue, 4 May 2021 18:00:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232143AbhEDSBu (ORCPT ); Tue, 4 May 2021 14:01:50 -0400 Received: from mail-mw2nam12on2083.outbound.protection.outlook.com ([40.107.244.83]:19585 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S230285AbhEDSBs (ORCPT ); Tue, 4 May 2021 14:01:48 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hmNpu9UT0d4FQDwxy93P9PB/ZSUoSyR7DImgaLwa+Y4/NfdLyoyoMCkK2xNMYfQxHdBnBBS4FV9RGIg/zsI5JfNTVVy8BoQ4jGzcaAeXJzygdCQrE5yltZhrZPc3GKM+JJa/mZiR4pmG3FAZ5BCpeLpiRtHI7yjyBEvlLUV6IKNGOiKSzHqjTeRjxmgSYKxF1H9sS4GY6PlXjsTzufluZE5L6OAO3h9GyVcMuS+sVD9M8eqMWDLYfyvOVtUeZgHzjmWETNx7eAjC1tjWxgSU9uU19Pa5X9VqiMVAUCYE7qsQxSlFQDkuYUOZBygbpEiCEvYcdMCBKvIa65lmN25AcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tES7OamPG2yhB1EsQjtDi+wmW2DPq3IAhbWd332A87w=; b=g5VKMtT46muXRiukkP235bFYGji9c8D0kFVSjAl6d+i5a/nY4AwstnEyVAjxNINrvNneKaBFuTWk/CUuFUJGyI3savG1IWGmIG8/WuBoKzE7z250rQMEg1L2Aq9qO3OvOuXwCDn9jPspm8pJyG8pagu1oy+Gc05zDz5kKVUL8Gj8REC9PdjRR61FrnIwmJZ19lOisrUyBfZbo3t9d7cN+Bs8xhyM8ZoVElkRyVwAEF3qJrFDUOpKEa/8lrMEEypg7m6YYYGnXgAKnghKU23As+oayj9SCCbAnikJIiIuGaStHIEHoYHRYolYo5dzkQGGkT6OyOxMnLNVzgSCE23nDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tES7OamPG2yhB1EsQjtDi+wmW2DPq3IAhbWd332A87w=; b=pCoR8RpzDIRFKAgAqVijxTfGIfVEEiXzE3iq5fP0oum8pe7jbZHPP7h8I41gXArpj8OozCYE8K4NyI19ZmfIORliZ0F60tM0zffvoqib2XXFhf77YspOEcfExLL6iZD+pKZm9k7FKw2PKRceVBbWKEvRfVqVRsGFOux21OFNG8RBUK3Uz9Z9U1mnN3I7TNpbGaGl/DHX90BIOwslNu5zyfTEyaoELCTpLMt5rkAo4NYWvjVfUSaIX3hGouMAGYgY4ORTyQcEsPOdekYzSi1RhryurLVEFqkZ6xsL6MH9O2jozpfilax6t9Pmrw2Wclc097fHWMbfay/rTzPTS1//5g== Authentication-Results: linux.intel.com; dkim=none (message not signed) header.d=none;linux.intel.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BY5PR12MB4067.namprd12.prod.outlook.com (2603:10b6:a03:212::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4087.26; Tue, 4 May 2021 18:00:52 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::81ef:d824:43ae:aef5]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::81ef:d824:43ae:aef5%4]) with mapi id 15.20.4087.044; Tue, 4 May 2021 18:00:52 +0000 Date: Tue, 4 May 2021 15:00:50 -0300 From: Jason Gunthorpe To: Jacob Pan Cc: "Tian, Kevin" , Alex Williamson , "Liu, Yi L" , Auger Eric , Jean-Philippe Brucker , LKML , Joerg Roedel , Lu Baolu , David Woodhouse , "iommu@lists.linux-foundation.org" , "cgroups@vger.kernel.org" , Tejun Heo , Li Zefan , Johannes Weiner , Jean-Philippe Brucker , Jonathan Corbet , "Raj, Ashok" , "Wu, Hao" , "Jiang, Dave" Subject: Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs Message-ID: <20210504180050.GB1370958@nvidia.com> References: <20210421133312.15307c44@redhat.com> <20210421230301.GP1370958@nvidia.com> <20210422121020.GT1370958@nvidia.com> <20210423114944.GF1370958@nvidia.com> <20210426123817.GQ1370958@nvidia.com> <20210504084148.4f61d0b5@jacob-builder> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210504084148.4f61d0b5@jacob-builder> X-Originating-IP: [47.55.113.94] X-ClientProxiedBy: BLAPR03CA0090.namprd03.prod.outlook.com (2603:10b6:208:329::35) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (47.55.113.94) by BLAPR03CA0090.namprd03.prod.outlook.com (2603:10b6:208:329::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.24 via Frontend Transport; Tue, 4 May 2021 18:00:52 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1ldzLm-000XJA-HA; Tue, 04 May 2021 15:00:50 -0300 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b0b3456b-1dd2-4322-d232-08d90f268e5a X-MS-TrafficTypeDiagnostic: BY5PR12MB4067: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5pGOoH6MvliesIB5f7UBBHXtZ1tyrUFS+FmO3RfDSypVMTwzBr6z7AvlEjMCZTxlvY6L3rtrhZxZjSEtSOVw7ja0UvfjW5yPCjpFFC5imZWEVuKaVXARQt2j7OQaCdKzIuQMKDKVn+wgqVQX+OFefneOjB/yz1LXgFdC1IcUdtoj+KUVGxa69cUWcXkp9hsWU1pkMaI54VCqMCaXgXeYzCfRR9iA65a4z7GBfLOZryiqEHriVlzQThc6+rPfgLq2GcpYZdSUzZWTdEHhu7H2qBuFABk5Uug0ZvmMiPzO/07uUBfCqYsQFjOLYpu19dunXP8sG4NXTKuunlfW1eVfgQui6xEyLyHbspSX3OAUaO8iCYwwXVk3vm+vWUMyp68E2k4B7cld+093YD6RgKjxZz37+oC2fmcvEoNiR0S7aJz4vMTFDJn7RTA49Xgaoq4dRhto0za/oE1YP1Pdfa9RT87LLFhwAyDJrB4QebjlxzrTFUW5SWjG5aX/RekxLPyzW9cmjSX+svSrDj/WcBNbfmGUGIH9JZJqmG0KJgo1OFuH+aREpqI5KelHVVlqixtzIc1fNtbAhZrcK5zS3ef4+04qSvNSupWKvmd4PTEqUPc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(136003)(376002)(366004)(346002)(396003)(478600001)(26005)(4326008)(33656002)(1076003)(5660300002)(2906002)(9786002)(66946007)(66476007)(86362001)(36756003)(2616005)(186003)(8676002)(8936002)(83380400001)(6916009)(7416002)(9746002)(426003)(316002)(54906003)(38100700002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?iaY2CYjaLXDy5p9iIQURMD6Vsgr1M3WZhs3S7NjLgYufl/gLO8VTwGMvrvst?= =?us-ascii?Q?In9jNjOX8rBydBljNubQZWvVVRavZDdYxKOATIxlaixS1r/AiYbuZH8FT9zO?= =?us-ascii?Q?JHJsl7Ttl/ydMIcu4fkyhvVwqfZ9fguVz8Y3j60fN0OFWOA6A92MUWap0nMg?= =?us-ascii?Q?+Kc4WOm0tNXKQIeCvkfz3FehifjpYIHSHRDIIKm9tm4dhEF36EYCZ4S+aSpM?= =?us-ascii?Q?2Ud9n2YT7xiupCW8DF4P9puRs0VhED9XhIT+1tssHZE8K9RhuhysIyA8WuBi?= =?us-ascii?Q?KT0dNwBFrMS+p2kP6yEjjoPLeXiPZW6lGYi3Btw5M1sPt6p16503RmBR9vx+?= =?us-ascii?Q?vj5+Baz3HCEWLhqZ719HScJb3GpJzg04JVhXWlnBij+5QCXCFut7waQEPq2q?= =?us-ascii?Q?0B8N7hLZfVQEdeZmyPvp1mdEBddFKVzwmwELAZDtedeQIcCbDVEEMtv1fEBg?= =?us-ascii?Q?mssLRypsIEwdKRJeEK+OKxiy+UgmB/WZS34FcwoIgVT7TqADTpwmbzJDxrR4?= =?us-ascii?Q?Ib8Jj4bYRUMCWEEXvtZEMsj9s18mpSFYN+DJ/XOGWBkNbqkNAXZaCzj7hqo3?= =?us-ascii?Q?E+GXBEhHF04cEapc5cNimWDkDRapxwDkIeyyBWF+gpoNSiqJJ1K4BwJ54/WW?= =?us-ascii?Q?yXOcSBCHgMylTha/V18Jv/U2Mt6EauPaHKFiizFMDD+v05V0eOwUksqKywSa?= =?us-ascii?Q?Fk3uLdYMGKKRlY9dCJMMtPMtXmT2tJGcSOliB9sAhZSL2Sp3WrXZB//QSjjF?= =?us-ascii?Q?UzbYIZqaDV3SYmjgS2MieyyozlEALSk5xKERiZtTDq1LxoKYcT8SNXXulR8H?= =?us-ascii?Q?43T7ERqiQXH4McI8FovmXJ5eiumiJAYfjWF254DKA4Nfyn5w+k4K/1mCWJgw?= =?us-ascii?Q?4Md1OjPRm9+4I/hDlskM0n/OUYJtkzB1/JD/5YailV6p/JyjRGXVZQMBxIAv?= =?us-ascii?Q?5Cj0lZ48oThrKUE7P0pMQ4LHGDtMnGRgpb2Njbz+AqeJY/r8goTDTacmb8IC?= =?us-ascii?Q?XFEJ9bkQb02MHnePKe1iO4qIaKdHm4hyX3jx4EXqR9Kb5ekIDN9Rj7WrLvC7?= =?us-ascii?Q?xpoKhvi35332RJQ1p6pQBfkyUOhjRG4rRoIS0PPWEF6BqRk1OwszlkbiLdsR?= =?us-ascii?Q?MKHNKCrr5DNwaCkEqIu5NI0Q+Zhd7vKUpZmy+9N7NzeWdaFFCN4kFrgm1kEs?= =?us-ascii?Q?1AxUZ94EQp+htAX6KxStjye6a5pSAaSnty1+3TZGfcnnME/ikcKemvJBmWfm?= =?us-ascii?Q?h/LXyXFdl8hOidqoTh9EyL13lO+Vcb74tkk1YdzYPTw53o5fY9ysJEKHOET0?= =?us-ascii?Q?U5ZucBzdfq2kaSlCZ1odxDNX?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: b0b3456b-1dd2-4322-d232-08d90f268e5a X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 May 2021 18:00:52.3744 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KWpAzv8dPFPy9NwXobWlQBST8JDIIAMZRS5lfcJvKL/3xElwOaRsA9BevUmasAt2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4067 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, May 04, 2021 at 08:41:48AM -0700, Jacob Pan wrote: > > > > > > (also looking at ioasid.c, why do we need such a thin and odd wrapper > > > around xarray?) > > > > > > > I'll leave it to Jean and Jacob. > Could you elaborate? I mean stuff like this: int ioasid_set_data(ioasid_t ioasid, void *data) { struct ioasid_data *ioasid_data; int ret = 0; spin_lock(&ioasid_allocator_lock); ioasid_data = xa_load(&active_allocator->xa, ioasid); if (ioasid_data) rcu_assign_pointer(ioasid_data->private, data); else ret = -ENOENT; spin_unlock(&ioasid_allocator_lock); /* * Wait for readers to stop accessing the old private data, so the * caller can free it. */ if (!ret) synchronize_rcu(); return ret; } EXPORT_SYMBOL_GPL(ioasid_set_data); It is a weird way to use xarray to have a structure which itself is just a wrapper around another RCU protected structure. Make the caller supply the ioasid_data memory, embedded in its own element, get rid of the void * and rely on XA_ZERO_ENTRY to hold allocated but not active entries. Make the synchronize_rcu() the caller responsiblity, and callers should really be able to use call_rcu() Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DD17C433ED for ; Tue, 4 May 2021 18:00:58 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 298CF613CB for ; Tue, 4 May 2021 18:00:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 298CF613CB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id DA9166068F; Tue, 4 May 2021 18:00:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 53vQfiC-w4Xb; Tue, 4 May 2021 18:00:57 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id C9C66605E6; Tue, 4 May 2021 18:00:56 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B0FEFC000D; Tue, 4 May 2021 18:00:56 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 064F4C0001 for ; Tue, 4 May 2021 18:00:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id DB7778348C for ; Tue, 4 May 2021 18:00:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=nvidia.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GY-3szKopYiQ for ; Tue, 4 May 2021 18:00:54 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2072.outbound.protection.outlook.com [40.107.244.72]) by smtp1.osuosl.org (Postfix) with ESMTPS id 0F41B8343B for ; Tue, 4 May 2021 18:00:53 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hmNpu9UT0d4FQDwxy93P9PB/ZSUoSyR7DImgaLwa+Y4/NfdLyoyoMCkK2xNMYfQxHdBnBBS4FV9RGIg/zsI5JfNTVVy8BoQ4jGzcaAeXJzygdCQrE5yltZhrZPc3GKM+JJa/mZiR4pmG3FAZ5BCpeLpiRtHI7yjyBEvlLUV6IKNGOiKSzHqjTeRjxmgSYKxF1H9sS4GY6PlXjsTzufluZE5L6OAO3h9GyVcMuS+sVD9M8eqMWDLYfyvOVtUeZgHzjmWETNx7eAjC1tjWxgSU9uU19Pa5X9VqiMVAUCYE7qsQxSlFQDkuYUOZBygbpEiCEvYcdMCBKvIa65lmN25AcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tES7OamPG2yhB1EsQjtDi+wmW2DPq3IAhbWd332A87w=; b=g5VKMtT46muXRiukkP235bFYGji9c8D0kFVSjAl6d+i5a/nY4AwstnEyVAjxNINrvNneKaBFuTWk/CUuFUJGyI3savG1IWGmIG8/WuBoKzE7z250rQMEg1L2Aq9qO3OvOuXwCDn9jPspm8pJyG8pagu1oy+Gc05zDz5kKVUL8Gj8REC9PdjRR61FrnIwmJZ19lOisrUyBfZbo3t9d7cN+Bs8xhyM8ZoVElkRyVwAEF3qJrFDUOpKEa/8lrMEEypg7m6YYYGnXgAKnghKU23As+oayj9SCCbAnikJIiIuGaStHIEHoYHRYolYo5dzkQGGkT6OyOxMnLNVzgSCE23nDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tES7OamPG2yhB1EsQjtDi+wmW2DPq3IAhbWd332A87w=; b=pCoR8RpzDIRFKAgAqVijxTfGIfVEEiXzE3iq5fP0oum8pe7jbZHPP7h8I41gXArpj8OozCYE8K4NyI19ZmfIORliZ0F60tM0zffvoqib2XXFhf77YspOEcfExLL6iZD+pKZm9k7FKw2PKRceVBbWKEvRfVqVRsGFOux21OFNG8RBUK3Uz9Z9U1mnN3I7TNpbGaGl/DHX90BIOwslNu5zyfTEyaoELCTpLMt5rkAo4NYWvjVfUSaIX3hGouMAGYgY4ORTyQcEsPOdekYzSi1RhryurLVEFqkZ6xsL6MH9O2jozpfilax6t9Pmrw2Wclc097fHWMbfay/rTzPTS1//5g== Authentication-Results: linux.intel.com; dkim=none (message not signed) header.d=none; linux.intel.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BY5PR12MB4067.namprd12.prod.outlook.com (2603:10b6:a03:212::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4087.26; Tue, 4 May 2021 18:00:52 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::81ef:d824:43ae:aef5]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::81ef:d824:43ae:aef5%4]) with mapi id 15.20.4087.044; Tue, 4 May 2021 18:00:52 +0000 Date: Tue, 4 May 2021 15:00:50 -0300 From: Jason Gunthorpe To: Jacob Pan Subject: Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs Message-ID: <20210504180050.GB1370958@nvidia.com> References: <20210421133312.15307c44@redhat.com> <20210421230301.GP1370958@nvidia.com> <20210422121020.GT1370958@nvidia.com> <20210423114944.GF1370958@nvidia.com> <20210426123817.GQ1370958@nvidia.com> <20210504084148.4f61d0b5@jacob-builder> Content-Disposition: inline In-Reply-To: <20210504084148.4f61d0b5@jacob-builder> X-Originating-IP: [47.55.113.94] X-ClientProxiedBy: BLAPR03CA0090.namprd03.prod.outlook.com (2603:10b6:208:329::35) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (47.55.113.94) by BLAPR03CA0090.namprd03.prod.outlook.com (2603:10b6:208:329::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.24 via Frontend Transport; Tue, 4 May 2021 18:00:52 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1ldzLm-000XJA-HA; Tue, 04 May 2021 15:00:50 -0300 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b0b3456b-1dd2-4322-d232-08d90f268e5a X-MS-TrafficTypeDiagnostic: BY5PR12MB4067: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5pGOoH6MvliesIB5f7UBBHXtZ1tyrUFS+FmO3RfDSypVMTwzBr6z7AvlEjMCZTxlvY6L3rtrhZxZjSEtSOVw7ja0UvfjW5yPCjpFFC5imZWEVuKaVXARQt2j7OQaCdKzIuQMKDKVn+wgqVQX+OFefneOjB/yz1LXgFdC1IcUdtoj+KUVGxa69cUWcXkp9hsWU1pkMaI54VCqMCaXgXeYzCfRR9iA65a4z7GBfLOZryiqEHriVlzQThc6+rPfgLq2GcpYZdSUzZWTdEHhu7H2qBuFABk5Uug0ZvmMiPzO/07uUBfCqYsQFjOLYpu19dunXP8sG4NXTKuunlfW1eVfgQui6xEyLyHbspSX3OAUaO8iCYwwXVk3vm+vWUMyp68E2k4B7cld+093YD6RgKjxZz37+oC2fmcvEoNiR0S7aJz4vMTFDJn7RTA49Xgaoq4dRhto0za/oE1YP1Pdfa9RT87LLFhwAyDJrB4QebjlxzrTFUW5SWjG5aX/RekxLPyzW9cmjSX+svSrDj/WcBNbfmGUGIH9JZJqmG0KJgo1OFuH+aREpqI5KelHVVlqixtzIc1fNtbAhZrcK5zS3ef4+04qSvNSupWKvmd4PTEqUPc= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BY5PR12MB3827.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(136003)(376002)(366004)(346002)(396003)(478600001)(26005)(4326008)(33656002)(1076003)(5660300002)(2906002)(9786002)(66946007)(66476007)(86362001)(36756003)(2616005)(186003)(8676002)(8936002)(83380400001)(6916009)(7416002)(9746002)(426003)(316002)(54906003)(38100700002)(66556008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?iaY2CYjaLXDy5p9iIQURMD6Vsgr1M3WZhs3S7NjLgYufl/gLO8VTwGMvrvst?= =?us-ascii?Q?In9jNjOX8rBydBljNubQZWvVVRavZDdYxKOATIxlaixS1r/AiYbuZH8FT9zO?= =?us-ascii?Q?JHJsl7Ttl/ydMIcu4fkyhvVwqfZ9fguVz8Y3j60fN0OFWOA6A92MUWap0nMg?= =?us-ascii?Q?+Kc4WOm0tNXKQIeCvkfz3FehifjpYIHSHRDIIKm9tm4dhEF36EYCZ4S+aSpM?= =?us-ascii?Q?2Ud9n2YT7xiupCW8DF4P9puRs0VhED9XhIT+1tssHZE8K9RhuhysIyA8WuBi?= =?us-ascii?Q?KT0dNwBFrMS+p2kP6yEjjoPLeXiPZW6lGYi3Btw5M1sPt6p16503RmBR9vx+?= =?us-ascii?Q?vj5+Baz3HCEWLhqZ719HScJb3GpJzg04JVhXWlnBij+5QCXCFut7waQEPq2q?= =?us-ascii?Q?0B8N7hLZfVQEdeZmyPvp1mdEBddFKVzwmwELAZDtedeQIcCbDVEEMtv1fEBg?= =?us-ascii?Q?mssLRypsIEwdKRJeEK+OKxiy+UgmB/WZS34FcwoIgVT7TqADTpwmbzJDxrR4?= =?us-ascii?Q?Ib8Jj4bYRUMCWEEXvtZEMsj9s18mpSFYN+DJ/XOGWBkNbqkNAXZaCzj7hqo3?= =?us-ascii?Q?E+GXBEhHF04cEapc5cNimWDkDRapxwDkIeyyBWF+gpoNSiqJJ1K4BwJ54/WW?= =?us-ascii?Q?yXOcSBCHgMylTha/V18Jv/U2Mt6EauPaHKFiizFMDD+v05V0eOwUksqKywSa?= =?us-ascii?Q?Fk3uLdYMGKKRlY9dCJMMtPMtXmT2tJGcSOliB9sAhZSL2Sp3WrXZB//QSjjF?= =?us-ascii?Q?UzbYIZqaDV3SYmjgS2MieyyozlEALSk5xKERiZtTDq1LxoKYcT8SNXXulR8H?= =?us-ascii?Q?43T7ERqiQXH4McI8FovmXJ5eiumiJAYfjWF254DKA4Nfyn5w+k4K/1mCWJgw?= =?us-ascii?Q?4Md1OjPRm9+4I/hDlskM0n/OUYJtkzB1/JD/5YailV6p/JyjRGXVZQMBxIAv?= =?us-ascii?Q?5Cj0lZ48oThrKUE7P0pMQ4LHGDtMnGRgpb2Njbz+AqeJY/r8goTDTacmb8IC?= =?us-ascii?Q?XFEJ9bkQb02MHnePKe1iO4qIaKdHm4hyX3jx4EXqR9Kb5ekIDN9Rj7WrLvC7?= =?us-ascii?Q?xpoKhvi35332RJQ1p6pQBfkyUOhjRG4rRoIS0PPWEF6BqRk1OwszlkbiLdsR?= =?us-ascii?Q?MKHNKCrr5DNwaCkEqIu5NI0Q+Zhd7vKUpZmy+9N7NzeWdaFFCN4kFrgm1kEs?= =?us-ascii?Q?1AxUZ94EQp+htAX6KxStjye6a5pSAaSnty1+3TZGfcnnME/ikcKemvJBmWfm?= =?us-ascii?Q?h/LXyXFdl8hOidqoTh9EyL13lO+Vcb74tkk1YdzYPTw53o5fY9ysJEKHOET0?= =?us-ascii?Q?U5ZucBzdfq2kaSlCZ1odxDNX?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: b0b3456b-1dd2-4322-d232-08d90f268e5a X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 May 2021 18:00:52.3744 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KWpAzv8dPFPy9NwXobWlQBST8JDIIAMZRS5lfcJvKL/3xElwOaRsA9BevUmasAt2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4067 Cc: Jean-Philippe Brucker , "Tian, Kevin" , "Jiang, Dave" , "Raj, Ashok" , Jonathan Corbet , Jean-Philippe Brucker , LKML , "iommu@lists.linux-foundation.org" , Alex Williamson , Li Zefan , Johannes Weiner , Tejun Heo , "cgroups@vger.kernel.org" , "Wu, Hao" , David Woodhouse X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Tue, May 04, 2021 at 08:41:48AM -0700, Jacob Pan wrote: > > > > > > (also looking at ioasid.c, why do we need such a thin and odd wrapper > > > around xarray?) > > > > > > > I'll leave it to Jean and Jacob. > Could you elaborate? I mean stuff like this: int ioasid_set_data(ioasid_t ioasid, void *data) { struct ioasid_data *ioasid_data; int ret = 0; spin_lock(&ioasid_allocator_lock); ioasid_data = xa_load(&active_allocator->xa, ioasid); if (ioasid_data) rcu_assign_pointer(ioasid_data->private, data); else ret = -ENOENT; spin_unlock(&ioasid_allocator_lock); /* * Wait for readers to stop accessing the old private data, so the * caller can free it. */ if (!ret) synchronize_rcu(); return ret; } EXPORT_SYMBOL_GPL(ioasid_set_data); It is a weird way to use xarray to have a structure which itself is just a wrapper around another RCU protected structure. Make the caller supply the ioasid_data memory, embedded in its own element, get rid of the void * and rely on XA_ZERO_ENTRY to hold allocated but not active entries. Make the synchronize_rcu() the caller responsiblity, and callers should really be able to use call_rcu() Jason _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Gunthorpe Subject: Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs Date: Tue, 4 May 2021 15:00:50 -0300 Message-ID: <20210504180050.GB1370958@nvidia.com> References: <20210421133312.15307c44@redhat.com> <20210421230301.GP1370958@nvidia.com> <20210422121020.GT1370958@nvidia.com> <20210423114944.GF1370958@nvidia.com> <20210426123817.GQ1370958@nvidia.com> <20210504084148.4f61d0b5@jacob-builder> Mime-Version: 1.0 Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tES7OamPG2yhB1EsQjtDi+wmW2DPq3IAhbWd332A87w=; b=pCoR8RpzDIRFKAgAqVijxTfGIfVEEiXzE3iq5fP0oum8pe7jbZHPP7h8I41gXArpj8OozCYE8K4NyI19ZmfIORliZ0F60tM0zffvoqib2XXFhf77YspOEcfExLL6iZD+pKZm9k7FKw2PKRceVBbWKEvRfVqVRsGFOux21OFNG8RBUK3Uz9Z9U1mnN3I7TNpbGaGl/DHX90BIOwslNu5zyfTEyaoELCTpLMt5rkAo4NYWvjVfUSaIX3hGouMAGYgY4ORTyQcEsPOdekYzSi1RhryurLVEFqkZ6xsL6MH9O2jozpfilax6t9Pmrw2Wclc097fHWMbfay/rTzPTS1//5g== Content-Disposition: inline In-Reply-To: <20210504084148.4f61d0b5@jacob-builder> List-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Jacob Pan Cc: "Tian, Kevin" , Alex Williamson , "Liu, Yi L" , Auger Eric , Jean-Philippe Brucker , LKML , Joerg Roedel , Lu Baolu , David Woodhouse , "iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org" , "cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Tejun Heo , Li Zefan , Johannes Weiner , Jean-Philippe Brucker , Jonathan Corbet , "Raj, Ashok" , "Wu, Hao" , "Jiang, Dave" On Tue, May 04, 2021 at 08:41:48AM -0700, Jacob Pan wrote: > > > > > > (also looking at ioasid.c, why do we need such a thin and odd wrapper > > > around xarray?) > > > > > > > I'll leave it to Jean and Jacob. > Could you elaborate? I mean stuff like this: int ioasid_set_data(ioasid_t ioasid, void *data) { struct ioasid_data *ioasid_data; int ret = 0; spin_lock(&ioasid_allocator_lock); ioasid_data = xa_load(&active_allocator->xa, ioasid); if (ioasid_data) rcu_assign_pointer(ioasid_data->private, data); else ret = -ENOENT; spin_unlock(&ioasid_allocator_lock); /* * Wait for readers to stop accessing the old private data, so the * caller can free it. */ if (!ret) synchronize_rcu(); return ret; } EXPORT_SYMBOL_GPL(ioasid_set_data); It is a weird way to use xarray to have a structure which itself is just a wrapper around another RCU protected structure. Make the caller supply the ioasid_data memory, embedded in its own element, get rid of the void * and rely on XA_ZERO_ENTRY to hold allocated but not active entries. Make the synchronize_rcu() the caller responsiblity, and callers should really be able to use call_rcu() Jason