From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751950AbdL0JVI (ORCPT ); Wed, 27 Dec 2017 04:21:08 -0500 Received: from mail-sn1nam02on0046.outbound.protection.outlook.com ([104.47.36.46]:43692 "EHLO NAM02-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751145AbdL0JVC (ORCPT ); Wed, 27 Dec 2017 04:21:02 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Suravee.Suthikulpanit@amd.com; From: Suravee Suthikulpanit To: linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org Cc: joro@8bytes.org, jroedel@suse.de, alex.williamson@redhat.com, Suravee Suthikulpanit Subject: [RFC PATCH v2 2/2] iommu/amd: Add support for fast IOTLB flushing Date: Wed, 27 Dec 2017 04:20:35 -0500 Message-Id: <1514366435-12723-3-git-send-email-suravee.suthikulpanit@amd.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1514366435-12723-1-git-send-email-suravee.suthikulpanit@amd.com> References: <1514366435-12723-1-git-send-email-suravee.suthikulpanit@amd.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [114.109.128.54] X-ClientProxiedBy: HK2PR02CA0155.apcprd02.prod.outlook.com (10.171.30.15) To MWHPR12MB1742.namprd12.prod.outlook.com (10.175.55.13) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 11d7473f-755f-4d03-23e7-08d54d0b254b X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(5600026)(4604075)(48565401081)(2017052603307)(7153060);SRVR:MWHPR12MB1742; X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1742;3:heP+UGlRkctk/MlkUR07+F3iVenKbJUSSkGRQg30I8N6SHrKs7quLEpoHpQcDvcFscCq29OyuPpVXgJ28OplqZ7A4rBRghkFNdAmuJ1ks6KMPgRF4DPZ8ZUkOD68son4nWWRpxGrccx5j6y7z46F6Ali4razL+I3uFJVtoobr5whWxA1DcHP8SRefFdMxrLbFkOB8CF5pODmKytgFTsZNt+44JAn6L7p7PCXrX1s5L/6sWegd5d5L8F9bj1MiMkw;25:fUTcLWMYbIRnf+/wt9I/fg27FXWbkyjSmzWg+iL7LR+7CjcjrJOPm0MUNFr+62hTeJ2II3+oAIQAyjS7gxAxBg3jmGpLXlRgMafvK0lxzRk+KAMs2/ukao7yPYeYlEkSGuTXmk3P5hpn6RRkXEvxLIV5RkS2/YvUD95TeamdqsxQ+seB7sh+zYMFwSQ2Hhnu9GGJBuibx8JgpYhZ7XDnHaHomgwfrNRTjGpaok4TFss2oufF/jLTzUqbZftTiY8oNn44jAWdzStO1n+ivGV5krIyC3N/AeAcIsE3vs2Qc5x8Gw+Nne43Wz/nEn5z67qs4ghuT3o1jWmI0V2CGSGXSQ==;31:uXGw9Fi50hU7nwsoGyxpQRf0sGq2oZkYUgGQvxFMc2Suq2bJhFo9zpNGhY3egMnYoL8O5JZycsBnTlYXwg680Z4fzt8JoijmGVnSEVToGvFdI6qJSlEOnPTDx044c3ul6q04arwF55V/g3vI9h9v2PgVoTYUYTvE37Sn/3giaJhRvOJlvrsIpfy/MZLQy43zT35K64Ii00HtDa9uGE/3bROQo+L9dO55OB/zqLPOROE= X-MS-TrafficTypeDiagnostic: MWHPR12MB1742: X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1742;20:AakeINgcWk48A7i7s44J1T2YX9E+O1PVKFcPU+MRh7tf7Jd4/tWyrDGDdL7BHpVTYBa//pUnFCVUkzNbd32IHiz5I6AXt7bma0l6C/09/Cdmm5fUuLOz6oEXhYp2I1PPbIFstPTGxxFbkkGoepxCtVOrx3JLpXGGfgTjYda25OHAenVddDmvjfyrn7N2lFTGgWglL17kuVwW/4YXEGOAAlUYsDZ8SD7ZgFFNThnu6r/A2lE/GEvRyDMT+S1k8peZs0S12C/fxB8LumINfrtEso3Ywo5P/SVVbPji1YKUZqbuYVTHnd+omhQb+CZvh78gGvQ1LCPnQErJL/aKkLX9XRVKnJ9ow/nhvM4c6BS0Jb0qccyPyBHGbN9J3LJJzILp8jejBz9xiQQ78oYAwVpgY01b0Zyxus1ttrM/TR0/rofC68ydWWPJRJJrcFC977FCqtV4ofCQVUk01riZpxi4OpgU1d5QpJ2HhfK5kd9Yf6zwpg8oCvA/dbWCIVck4jPi;4:/TjKEZqeLs9MI/9vavsx5tmEtXiZdlObGgiUcKYGaHgcewXPolxCBK8EPHPxLu5OHGEmV+zG6iqZ/2ytedtM2BZTUJmJjmPqARKFO+IxS1rqo3KsdZW49eAcqEY9qeMthw7CkF/qoXNcNJrkUkxLOQ0B+uxuPLxmTN4w6lJZv63Cwxagk5333oMp3RV0o6rB2rl5uVZ0qTVeYfL3pC7rIyuQ0NqFfzVhf8r0fqHGFULeg1hp6AAyHPnUtdybqXIyGrnP0/fErtF8Ltyd2DhEHWVCcUohOlXaoBbsVt1eNWm7RiYZ+DQfaf9cdiqYYzAz X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040470)(2401047)(5005006)(8121501046)(3231023)(944501075)(93006095)(93001095)(10201501046)(3002001)(6055026)(6041268)(20161123564045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123558120)(6072148)(201708071742011);SRVR:MWHPR12MB1742;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:MWHPR12MB1742; X-Forefront-PRVS: 0534947130 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(396003)(376002)(39380400002)(366004)(346002)(39860400002)(199004)(189003)(68736007)(25786009)(106356001)(105586002)(6486002)(8936002)(97736004)(66066001)(86362001)(47776003)(3846002)(6116002)(4720700003)(5660300001)(7736002)(305945005)(6666003)(2950100002)(8676002)(81166006)(81156014)(50226002)(72206003)(478600001)(316002)(16526018)(16586007)(2906002)(4326008)(59450400001)(53936002)(51416003)(50466002)(48376002)(6512007)(36756003)(5890100001)(52116002)(386003)(6506007)(76176011)(217873001);DIR:OUT;SFP:1101;SCL:1;SRVR:MWHPR12MB1742;H:ssuthiku-rhel74.localdomain;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;MWHPR12MB1742;23:zEsyplSHqKD8hwerHmdzTa/L/M7qhMVpqBRNfkFnp?= =?us-ascii?Q?GOXnRwvL3qHb1iws0jrVgGZnZvpKwdIpGiVYkoF3fuPuaJibtMuLEdwvShhg?= =?us-ascii?Q?Hz2EJNui1Bo5/jEsQQWQ0dsTpFlZsgCQ9/MpidzyaqVSE71JNzhPN7d16mGn?= =?us-ascii?Q?ZSPZmAaz0B65slba6K330ENB3ZmU1rPwha9owZB609pfeTHxtiebtsMBxk+L?= =?us-ascii?Q?zaBC/sTKJmFxRKjgjwUFTfV8/tPQoTFY3udZmQzt6/uRVb/tLDNWoPioXMo8?= =?us-ascii?Q?8NX503h+BoI6mo9CW3ZWPjGayUlR6MnyKXdRX04OK5FgYLYKNxFxbHY60b24?= =?us-ascii?Q?QQ/ye/4ATWXjvOFcQhokiUHqdUjhg9cU88Zl+SAL7KvswRp3p5EuF61Q5y9N?= =?us-ascii?Q?NKqrQu4H7qr/vsuuyujLPFYCQbVga/IxmFWtPzvo/zHcS5BZ0HGWpv79o5ci?= =?us-ascii?Q?M1Ur89ZAnaBJv9wEJ17D0eeMQHEMr1NBM0IXiekjZpMgz5MOyJnPG17gqj+B?= =?us-ascii?Q?58j7WwXyQaSEKf/FciIhYx+RJ/jiwdcBy1mVU8tqzDIcjj8x1mWaYLU1yxx/?= =?us-ascii?Q?f6rt9zXwnZ3zwzIR6d/RVrIidwV6IRF4EHeWNRZVDVlTjyCNldT5xC28lvjf?= =?us-ascii?Q?1CwYqbxkBUMeV+5B5Z3zxa+MXKq/wWHQdrbgh35CS/iYLCYQrK3GMtIfgXLZ?= =?us-ascii?Q?YO8VxbFrnc8/zdT9dU+mXYhkU0p+Tr05gFVSNho/yZjEIU5htuaendZOlI22?= =?us-ascii?Q?1RwfuE8WJj22A9CSqy27nHl4/yaROm1PFwGAJ9OqI76abaruTDXaILo0eVy1?= =?us-ascii?Q?y0iW3n9XYftPW/KsrcNUClRm0CMUCAKtDNusCSmu9X4WuV9ShvmO1n16DpdR?= =?us-ascii?Q?oTk9aHasi8ugYxnBsvSMv7gsKSrxZ18aLWlvtNoVgeS8zWa6wiilHAUPImvw?= =?us-ascii?Q?UbGAkLd85rdA1FQjmiCSAHJKoc5Zi0A3o8LrnEWXMo2CFJwdPEgmOmYmczbY?= =?us-ascii?Q?Ny8Aaa0AtS527IVWjRT2jmou2EEJ/ji/yf/kzC6+3oH3vz7y/7Jbfxf1cEut?= =?us-ascii?Q?chYDLjKzD2lKl1MBhxE6NlywBfK6BEhixyOayjhensmk25yNqRnrFgrw0OmT?= =?us-ascii?Q?9aolx5Ln5ALJyiPOHCWGc71riDWpCjkKsHUfXb5JQrubJclOD2SeZLUdxOG2?= =?us-ascii?Q?585on3GmEhqBow=3D?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1742;6:e4V0kB+2BNzHQ9NIUa+YU7juQqNK4yNck4Jadfmqfza/LL4Q3nuLvS/WieCBjQm6Xej9mj9cfma4fwI4tRZ8KuEN1nQ2bKYuM/8FREpCLEZPCPzCcFz6VcbpIL9IzGhTZaAuLVCGIfmN6e2cy1UrzS2Z9hYMXb11pkO3SKT+ZA8d19fy0aK+yONuC2fV06zh5Js3WmjLG1uOEE65XjXE7OootwKyjarNk9osVYxhSyg/0w+MDWH2a3L5jv46WAMVyV06e0k1jjYNyYLPde7kG2TwmkUaAfTa95M8QjEKyEPNCmo8QESZw1G9oxSmY+WHVydagZkRMsV5sKPZm+wOqfQTZDSHzzQU3/Vmy87AzO8=;5:9i/4r9doXwaQwfyqMSsZvas1E7gGcekpyawtFqxQu8TUXU62NTwBTm0HJY2NXbDcEJDWRn4goxT+qIAFM6nMOGLKr+6lN7sjiDsc3O/NTFtTNW9PJ86qBoBpoaSKJCOHbkDugeSZsWCO0QIxo/I6BBeH5HIfh7hbpsr5gTec8o4=;24:yInDlxU5WPTAFOedhikknxDr5c3Ib0ShGrNhXJrU0wg+QJEpZxsdo6MUQAShXL0SQYlk+Lu3RWvG8dEO8QteXycpWFGaQeGzsGHLoq1bQfg=;7:r6ch5Edzd91C52i5L1hiREEZvZX7BXDQe74Kqk6IAp2pB2ACFVssyFzXt5AB6nr3kMwoRl3nZtzIERUmv9LQG+apDCE+pAKlH/8STumkVM8UOWn+QUkSZNEv4FyBRB5/VDhi7DU8I23HD7UifGYryhLFMz4+ZcvPl5bkj2nCAAn3RBeHtkM7UvtNJVz1YzJkbwDDVJ5IvHmvkUXbLJFFVJP8gTl7BqNayBqUpUySE4fBS+MaYKbUw8/doVortn4e SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1742;20:MNxA+GZtVB0w59F1lwtjpoNHwK+nMuTFEczSW8mG2pZiXiwa9XGWkZDsNE2N8dxQnAZak5NvZNMdHxdb+ndCx6wypuabfyghfCUC775l1x3gRHwDLhB8g/+PS/tJmTAWOpJzX1cNx1DoxW+OW+QtxYhYiarZeUimT7CSvMgt/WtB7mcyiZumiFogfc3KFNKB255pLDJwZIVTOeqMtCTOCX7xKzpzFIB/Q0KAjTXKIGxSqSjz9Ohe9mDCKXOWo9nB X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Dec 2017 09:20:58.8141 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 11d7473f-755f-4d03-23e7-08d54d0b254b X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1742 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Implement the newly added IOTLB flushing interface for AMD IOMMU. Signed-off-by: Suravee Suthikulpanit --- drivers/iommu/amd_iommu.c | 73 ++++++++++++++++++++++++++++++++++++++++- drivers/iommu/amd_iommu_init.c | 7 ++++ drivers/iommu/amd_iommu_types.h | 7 ++++ 3 files changed, 86 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 7d5eb00..42fe365 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -129,6 +129,12 @@ struct dma_ops_domain { static struct iova_domain reserved_iova_ranges; static struct lock_class_key reserved_rbtree_key; +struct amd_iommu_flush_entries { + struct list_head list; + unsigned long iova; + size_t size; +}; + /**************************************************************************** * * Helper functions @@ -3043,7 +3049,6 @@ static size_t amd_iommu_unmap(struct iommu_domain *dom, unsigned long iova, unmap_size = iommu_unmap_page(domain, iova, page_size); mutex_unlock(&domain->api_lock); - domain_flush_tlb_pde(domain); domain_flush_complete(domain); return unmap_size; @@ -3163,6 +3168,69 @@ static bool amd_iommu_is_attach_deferred(struct iommu_domain *domain, return dev_data->defer_attach; } +static void amd_iommu_flush_iotlb_all(struct iommu_domain *domain) +{ + struct protection_domain *dom = to_pdomain(domain); + + domain_flush_tlb_pde(dom); +} + +static void amd_iommu_iotlb_range_add(struct iommu_domain *domain, + unsigned long iova, size_t size) +{ + struct amd_iommu_flush_entries *entry, *p; + unsigned long flags; + bool found = false; + + spin_lock_irqsave(&amd_iommu_flush_list_lock, flags); + list_for_each_entry(p, &amd_iommu_flush_list, list) { + if (iova != p->iova) + continue; + + if (size > p->size) { + p->size = size; + pr_debug("%s: update range: iova=%#lx, size = %#lx\n", + __func__, p->iova, p->size); + } + found = true; + break; + } + + if (!found) { + entry = kzalloc(sizeof(struct amd_iommu_flush_entries), + GFP_KERNEL); + if (entry) { + pr_debug("%s: new range: iova=%lx, size=%#lx\n", + __func__, iova, size); + + entry->iova = iova; + entry->size = size; + list_add(&entry->list, &amd_iommu_flush_list); + } + } + spin_unlock_irqrestore(&amd_iommu_flush_list_lock, flags); +} + +static void amd_iommu_iotlb_sync(struct iommu_domain *domain) +{ + struct protection_domain *pdom = to_pdomain(domain); + struct amd_iommu_flush_entries *entry, *next; + unsigned long flags; + + /* Note: + * Currently, IOMMU driver just flushes the whole IO/TLB for + * a given domain. So, just remove entries from the list here. + */ + spin_lock_irqsave(&amd_iommu_flush_list_lock, flags); + list_for_each_entry_safe(entry, next, &amd_iommu_flush_list, list) { + list_del(&entry->list); + kfree(entry); + } + spin_unlock_irqrestore(&amd_iommu_flush_list_lock, flags); + + domain_flush_tlb_pde(pdom); +} + const struct iommu_ops amd_iommu_ops = { .capable = amd_iommu_capable, .domain_alloc = amd_iommu_domain_alloc, @@ -3181,6 +3249,9 @@ static bool amd_iommu_is_attach_deferred(struct iommu_domain *domain, .apply_resv_region = amd_iommu_apply_resv_region, .is_attach_deferred = amd_iommu_is_attach_deferred, .pgsize_bitmap = AMD_IOMMU_PGSIZES, + .flush_iotlb_all = amd_iommu_flush_iotlb_all, + .iotlb_range_add = amd_iommu_iotlb_range_add, + .iotlb_sync = amd_iommu_iotlb_sync, }; /***************************************************************************** diff --git a/drivers/iommu/amd_iommu_init.c b/drivers/iommu/amd_iommu_init.c index 6fe2d03..e8f8cee 100644 --- a/drivers/iommu/amd_iommu_init.c +++ b/drivers/iommu/amd_iommu_init.c @@ -185,6 +185,12 @@ struct ivmd_header { bool amd_iommu_force_isolation __read_mostly; /* + * IOTLB flush list + */ +LIST_HEAD(amd_iommu_flush_list); +spinlock_t amd_iommu_flush_list_lock; + +/* * List of protection domains - used during resume */ LIST_HEAD(amd_iommu_pd_list); @@ -2490,6 +2496,7 @@ static int __init early_amd_iommu_init(void) __set_bit(0, amd_iommu_pd_alloc_bitmap); spin_lock_init(&amd_iommu_pd_lock); + spin_lock_init(&amd_iommu_flush_list_lock); /* * now the data structures are allocated and basically initialized diff --git a/drivers/iommu/amd_iommu_types.h b/drivers/iommu/amd_iommu_types.h index f6b24c7..c3f4a7e 100644 --- a/drivers/iommu/amd_iommu_types.h +++ b/drivers/iommu/amd_iommu_types.h @@ -668,6 +668,13 @@ struct iommu_dev_data { extern struct list_head amd_iommu_pd_list; /* + * Declarations for the global flush list to support + * iotlb_range_add() and iotlb_sync. + */ +extern spinlock_t amd_iommu_flush_list_lock; +extern struct list_head amd_iommu_flush_list; + +/* * Structure defining one entry in the device table */ struct dev_table_entry { -- 1.8.3.1