From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AD86C43381 for ; Tue, 9 Mar 2021 12:15:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A954E65266 for ; Tue, 9 Mar 2021 12:15:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A954E65266 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3C1508D00E7; Tue, 9 Mar 2021 07:15:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 396818D007F; Tue, 9 Mar 2021 07:15:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C47D8D00E7; Tue, 9 Mar 2021 07:15:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0050.hostedemail.com [216.40.44.50]) by kanga.kvack.org (Postfix) with ESMTP id F0B108D007F for ; Tue, 9 Mar 2021 07:15:45 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B0EC3180AD811 for ; Tue, 9 Mar 2021 12:15:45 +0000 (UTC) X-FDA: 77900231850.01.0AA9C47 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2041.outbound.protection.outlook.com [40.107.94.41]) by imf16.hostedemail.com (Postfix) with ESMTP id A4D89801915F for ; Tue, 9 Mar 2021 12:15:44 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RlxNGXA0CY2SH+Mx4qTU7ziO0gJNlUVq1o2zrd9W8R2aDZWL7XiqbBG7LK1csz6xAunbH/l2bJopZcCK1pyRX9jz5Ye5sWFiGtaOcLi5p1LXiaM8vvLes/J0fdM3qpXg51Hx+qCZAVFov5KQiPCrRek/A4BweaITnJk4Dz0NiO1VhC/1iz8PHcmYFR8SjmpRrUD+GWLA/tmJ3THH/1kphQmNE+ahHBymPjj7XAcRQumg+iL2gBR81GwMW1AKJlS2CCaFWR1wHTnBLCDQaG1LTyS760TyKIKcMJ+F7jQpIMAQStITmVsPuic3wtIAeo1zS143fztjWOHCHuub04VKWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X2Ye7ZOmoNbr4DRG4frF5xGSw/0sOPNl69k/TwAEMAE=; b=ffsRp5a4E0KYBxfTQ4vzfERhXbC5n8fnvrMOmDUeJynNP1DYmL1Ci8cIgaGuOiqzjKONi58WvMtJe7rJ/1ayEQ55dLf0JcpF/lj3TNV4VSfcWwiRK82dc6vLGa3NumEbVGDcg2/rEO2/NoymLJVa23fcILFFyXLQQid+80AqS6kFkDsYSQzUN/paKVwc7Wn+f2dD19ja5A8HW7G3LAy2QBu+3yVTF5jhoNBUULYGm7ZfLZ9b604od3G6sxzfsBHo+fhJuJZIzMIw4EQ/hACgEm5iLG6o81iIhbE32aSjHxxl+ASMSIqiRVdSQBiA7VVqp5QEMTPdKqDIPIU8ArW3Mg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X2Ye7ZOmoNbr4DRG4frF5xGSw/0sOPNl69k/TwAEMAE=; b=KlmETmEU3F6Xlouuw9IdtTYAaSCCvE+TeQ4sChnN3bm/e2eV4lnDIyxBoDFkd0uP3IRHM8hZvWr5hGY/03zHZEwsUxvCdtytGFSkPOpDMPfMijnrTVoN7rElpUrqYXCfD2CB0VuIQla265W14svV9qfs7o2UCOhrDjMlTTmC3aXimdYZFEAxbuFinK4eZqT/AkSsTzF2iTtOuqpLLMYLFd27/6T2Y4E3edpQew6lr0UONtHLl4C+vrErcQNEcT76wjN2L+7B5LhAIfZ36tfPZL1cpJxaruD72p16jFWSonPO6cj0+TAewQ7rhxOSrBrJyysFl2vOfqp5kbnRp9UVMQ== Received: from BN9PR03CA0910.namprd03.prod.outlook.com (2603:10b6:408:107::15) by SA0PR12MB4366.namprd12.prod.outlook.com (2603:10b6:806:72::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.23; Tue, 9 Mar 2021 12:15:43 +0000 Received: from BN8NAM11FT028.eop-nam11.prod.protection.outlook.com (2603:10b6:408:107:cafe::ef) by BN9PR03CA0910.outlook.office365.com (2603:10b6:408:107::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Tue, 9 Mar 2021 12:15:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT028.mail.protection.outlook.com (10.13.176.225) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.3912.17 via Frontend Transport; Tue, 9 Mar 2021 12:15:43 +0000 Received: from localhost (172.20.145.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 9 Mar 2021 12:15:38 +0000 From: Alistair Popple To: , , , CC: , , , , , , , "Alistair Popple" , Jason Gunthorpe Subject: [PATCH v5 6/8] mm: Selftests for exclusive device memory Date: Tue, 9 Mar 2021 23:15:03 +1100 Message-ID: <20210309121505.23608-7-apopple@nvidia.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210309121505.23608-1-apopple@nvidia.com> References: <20210309121505.23608-1-apopple@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: dfca3be2-0b8c-42c0-ca8d-08d8e2f50fc4 X-MS-TrafficTypeDiagnostic: SA0PR12MB4366: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DW4LiaW/XdCkE4pdAwVTNpbBSiBH4n0e4Jra/oBFie+9OSnGyS4l/I6ov8QPbtBxvIR+RA0mK0wnTNwQVMG5yaaM8t8r7Y+ikvdub82jpZzYBuIMW37mXvWMBkcUVaFmLEII+Iq+Scylwn8sjq4qBwkV0H5BvMijSbFzlgHgHV0EAUR+4VOTgJ9aVzUXy9b3DQMMb3Ng9rmM/d5fIKSi67sguX8vFPmoe3Gaw5atEOrwKcGufZfe0Yz3qN7V7VDRdJ0RDwgM6vAFkCOkaaAXD4Udu5FVuN05ASyx+PUKdnAJduDx2kuSsMJW9ckrKep/frIXJeOjO2lDQI47PybRGC1axJE4fAgYn/KXzG4KRNohdiw0gB5DpUa5/JJpeR3/Ala0g/vOSepG23nW1hS+ra6P1wzLaE9kqcx+IR3PPxIQyzylEaBQ0NN4i8E1B8D/NcimSTKx8bC8ulJYY6MVIVHmPqIUMtW/7P97+yVhhHzJeywmDGtD8bLZl84YGCgA1V44YtZJgMiP6haE3UJsOpDDhhBWUgBst89MOaD+Kx1k2twvdaYVj2jKrErS3MaE5hmczbmh4szS7Y+BZ2f1zDkTO/G4yym132A6QxZqESG36sz6buRhnPv0HRzs/D6ogxIa7kKccxUk/DcDeZA40OxYJrOV92FVcHO2hxs0UM7vYAfWDnTDCp54tpMmj0kw X-Forefront-Antispam-Report: CIP:216.228.112.34;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid03.nvidia.com;CAT:NONE;SFS:(4636009)(396003)(39860400002)(136003)(376002)(346002)(36840700001)(46966006)(34020700004)(54906003)(7636003)(110136005)(82310400003)(8936002)(36756003)(356005)(426003)(8676002)(47076005)(82740400003)(316002)(36906005)(83380400001)(36860700001)(2906002)(336012)(70586007)(30864003)(6666004)(5660300002)(4326008)(1076003)(478600001)(186003)(26005)(107886003)(16526019)(86362001)(2616005)(70206006);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Mar 2021 12:15:43.0806 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dfca3be2-0b8c-42c0-ca8d-08d8e2f50fc4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT028.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4366 X-Stat-Signature: 3p7h7isc8qu5g5wpuw3ns7ejkfpnis4h X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A4D89801915F Received-SPF: none (nvidia.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=NAM10-MW2-obe.outbound.protection.outlook.com; client-ip=40.107.94.41 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615292144-926195 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Adds some selftests for exclusive device memory. Signed-off-by: Alistair Popple Acked-by: Jason Gunthorpe Tested-by: Ralph Campbell Reviewed-by: Ralph Campbell --- lib/test_hmm.c | 126 +++++++++++++- lib/test_hmm_uapi.h | 2 + tools/testing/selftests/vm/hmm-tests.c | 219 +++++++++++++++++++++++++ 3 files changed, 346 insertions(+), 1 deletion(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 80a78877bd93..305a9d9e2b4c 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -25,6 +25,7 @@ #include #include #include +#include =20 #include "test_hmm_uapi.h" =20 @@ -46,6 +47,7 @@ struct dmirror_bounce { unsigned long cpages; }; =20 +#define DPT_XA_TAG_ATOMIC 1UL #define DPT_XA_TAG_WRITE 3UL =20 /* @@ -218,7 +220,7 @@ static bool dmirror_interval_invalidate(struct mmu_in= terval_notifier *mni, * the invalidation is handled as part of the migration process. */ if (range->event =3D=3D MMU_NOTIFY_MIGRATE && - range->migrate_pgmap_owner =3D=3D dmirror->mdevice) + range->owner =3D=3D dmirror->mdevice) return true; =20 if (mmu_notifier_range_blockable(range)) @@ -619,6 +621,54 @@ static void dmirror_migrate_alloc_and_copy(struct mi= grate_vma *args, } } =20 +static int dmirror_check_atomic(struct dmirror *dmirror, unsigned long s= tart, + unsigned long end) +{ + unsigned long pfn; + + for (pfn =3D start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); pfn++) { + void *entry; + struct page *page; + + entry =3D xa_load(&dmirror->pt, pfn); + page =3D xa_untag_pointer(entry); + if (xa_pointer_tag(entry) =3D=3D DPT_XA_TAG_ATOMIC) + return -EPERM; + } + + return 0; +} + +static int dmirror_atomic_map(unsigned long start, unsigned long end, + struct page **pages, struct dmirror *dmirror) +{ + unsigned long pfn, mapped =3D 0; + int i; + + /* Map the migrated pages into the device's page tables. */ + mutex_lock(&dmirror->mutex); + + for (i =3D 0, pfn =3D start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); p= fn++, i++) { + void *entry; + + if (!pages[i]) + continue; + + entry =3D pages[i]; + entry =3D xa_tag_pointer(entry, DPT_XA_TAG_ATOMIC); + entry =3D xa_store(&dmirror->pt, pfn, entry, GFP_ATOMIC); + if (xa_is_err(entry)) { + mutex_unlock(&dmirror->mutex); + return xa_err(entry); + } + + mapped++; + } + + mutex_unlock(&dmirror->mutex); + return mapped; +} + static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, struct dmirror *dmirror) { @@ -661,6 +711,71 @@ static int dmirror_migrate_finalize_and_map(struct m= igrate_vma *args, return 0; } =20 +static int dmirror_exclusive(struct dmirror *dmirror, + struct hmm_dmirror_cmd *cmd) +{ + unsigned long start, end, addr; + unsigned long size =3D cmd->npages << PAGE_SHIFT; + struct mm_struct *mm =3D dmirror->notifier.mm; + struct page *pages[64]; + struct dmirror_bounce bounce; + unsigned long next; + int ret; + + start =3D cmd->addr; + end =3D start + size; + if (end < start) + return -EINVAL; + + /* Since the mm is for the mirrored process, get a reference first. */ + if (!mmget_not_zero(mm)) + return -EINVAL; + + mmap_read_lock(mm); + for (addr =3D start; addr < end; addr =3D next) { + int i, mapped; + + if (end < addr + (ARRAY_SIZE(pages) << PAGE_SHIFT)) + next =3D end; + else + next =3D addr + (ARRAY_SIZE(pages) << PAGE_SHIFT); + + ret =3D make_device_exclusive_range(mm, addr, next, pages, NULL); + mapped =3D dmirror_atomic_map(addr, next, pages, dmirror); + for (i =3D 0; i < ret; i++) { + if (pages[i]) { + unlock_page(pages[i]); + put_page(pages[i]); + } + } + + if (addr + (mapped << PAGE_SHIFT) < next) { + mmap_read_unlock(mm); + mmput(mm); + return -EBUSY; + } + } + mmap_read_unlock(mm); + mmput(mm); + + /* Return the migrated data for verification. */ + ret =3D dmirror_bounce_init(&bounce, start, size); + if (ret) + return ret; + mutex_lock(&dmirror->mutex); + ret =3D dmirror_do_read(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret =3D=3D 0) { + if (copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr, + bounce.size)) + ret =3D -EFAULT; + } + + cmd->cpages =3D bounce.cpages; + dmirror_bounce_fini(&bounce); + return ret; +} + static int dmirror_migrate(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd) { @@ -949,6 +1064,15 @@ static long dmirror_fops_unlocked_ioctl(struct file= *filp, ret =3D dmirror_migrate(dmirror, &cmd); break; =20 + case HMM_DMIRROR_EXCLUSIVE: + ret =3D dmirror_exclusive(dmirror, &cmd); + break; + + case HMM_DMIRROR_CHECK_EXCLUSIVE: + ret =3D dmirror_check_atomic(dmirror, cmd.addr, + cmd.addr + (cmd.npages << PAGE_SHIFT)); + break; + case HMM_DMIRROR_SNAPSHOT: ret =3D dmirror_snapshot(dmirror, &cmd); break; diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h index 670b4ef2a5b6..f14dea5dcd06 100644 --- a/lib/test_hmm_uapi.h +++ b/lib/test_hmm_uapi.h @@ -33,6 +33,8 @@ struct hmm_dmirror_cmd { #define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) #define HMM_DMIRROR_MIGRATE _IOWR('H', 0x02, struct hmm_dmirror_cmd) #define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x03, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x04, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_= cmd) =20 /* * Values returned in hmm_dmirror_cmd.ptr for HMM_DMIRROR_SNAPSHOT. diff --git a/tools/testing/selftests/vm/hmm-tests.c b/tools/testing/selft= ests/vm/hmm-tests.c index 5d1ac691b9f4..5d3c5db9ed3a 100644 --- a/tools/testing/selftests/vm/hmm-tests.c +++ b/tools/testing/selftests/vm/hmm-tests.c @@ -1485,4 +1485,223 @@ TEST_F(hmm2, double_map) hmm_buffer_free(buffer); } =20 +/* + * Basic check of exclusive faulting. + */ +TEST_F(hmm, exclusive) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + int *ptr; + int ret; + + npages =3D ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift; + ASSERT_NE(npages, 0); + size =3D npages << self->page_shift; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + + buffer->ptr =3D mmap(NULL, size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Map memory exclusively for device access. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_EXCLUSIVE, buffer, npages= ); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + /* Fault pages back to system memory and check them. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i]++, i); + + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i+1); + + /* Check atomic access revoked */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_CHECK_EXCLUSIVE, buffer, = npages); + ASSERT_EQ(ret, 0); + + hmm_buffer_free(buffer); +} + +TEST_F(hmm, exclusive_shared) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + int *ptr; + int ret, i; + + npages =3D ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift; + ASSERT_NE(npages, 0); + size =3D npages << self->page_shift; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + + buffer->ptr =3D mmap(NULL, size, + PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Map memory exclusively for device access. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_EXCLUSIVE, buffer, npages= ); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + /* Fault pages back to system memory and check them. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i]++, i); + + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i+1); + + /* Check atomic access revoked */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_CHECK_EXCLUSIVE, buffer, = npages); + ASSERT_FALSE(ret); + + /* Map memory exclusively for device access again to check process tear= down */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_EXCLUSIVE, buffer, npages= ); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + hmm_buffer_free(buffer); +} + +/* + * Same as above but for shared anonymous memory. + */ +TEST_F(hmm, exclusive_mprotect) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + int *ptr; + int ret; + + npages =3D ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift; + ASSERT_NE(npages, 0); + size =3D npages << self->page_shift; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + + buffer->ptr =3D mmap(NULL, size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Map memory exclusively for device access. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_EXCLUSIVE, buffer, npages= ); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + ret =3D mprotect(buffer->ptr, size, PROT_READ); + ASSERT_EQ(ret, 0); + + /* Simulate a device writing system memory. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_WRITE, buffer, npages); + ASSERT_EQ(ret, -EPERM); + + hmm_buffer_free(buffer); +} + +/* + * Check copy-on-write works. + */ +TEST_F(hmm, exclusive_cow) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + int *ptr; + int ret; + + npages =3D ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift; + ASSERT_NE(npages, 0); + size =3D npages << self->page_shift; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + + buffer->ptr =3D mmap(NULL, size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Map memory exclusively for device access. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_EXCLUSIVE, buffer, npages= ); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + fork(); + + /* Fault pages back to system memory and check them. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i]++, i); + + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i+1); + + hmm_buffer_free(buffer); +} + TEST_HARNESS_MAIN --=20 2.20.1