From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <SRS0=GvXl=HM=kvack.org=owner-linux-mm@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-6.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no
	version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id C75F3C433E6
	for <linux-mm@archiver.kernel.org>; Wed, 10 Feb 2021 03:40:20 +0000 (UTC)
Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17])
	by mail.kernel.org (Postfix) with ESMTP id 5084264DE7
	for <linux-mm@archiver.kernel.org>; Wed, 10 Feb 2021 03:40:20 +0000 (UTC)
DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5084264DE7
Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com
Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org
Received: by kanga.kvack.org (Postfix)
	id 9C2A76B0006; Tue,  9 Feb 2021 22:40:19 -0500 (EST)
Received: by kanga.kvack.org (Postfix, from userid 40)
	id 972FE6B006C; Tue,  9 Feb 2021 22:40:19 -0500 (EST)
X-Delivered-To: int-list-linux-mm@kvack.org
Received: by kanga.kvack.org (Postfix, from userid 63042)
	id 888486B006E; Tue,  9 Feb 2021 22:40:19 -0500 (EST)
X-Delivered-To: linux-mm@kvack.org
Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112])
	by kanga.kvack.org (Postfix) with ESMTP id 724C66B0006
	for <linux-mm@kvack.org>; Tue,  9 Feb 2021 22:40:19 -0500 (EST)
Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251])
	by forelay05.hostedemail.com (Postfix) with ESMTP id 29BBC185447FE
	for <linux-mm@kvack.org>; Wed, 10 Feb 2021 03:40:19 +0000 (UTC)
X-FDA: 77800955358.29.5E80CE6
Received: from hqnvemgate26.nvidia.com (hqnvemgate26.nvidia.com [216.228.121.65])
	by imf08.hostedemail.com (Postfix) with ESMTP id 8514280191E9
	for <linux-mm@kvack.org>; Wed, 10 Feb 2021 03:40:16 +0000 (UTC)
Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA)
	id <B6023559f0000>; Tue, 09 Feb 2021 19:40:15 -0800
Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL111.nvidia.com
 (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 10 Feb
 2021 03:40:15 +0000
Received: from nvdebian.localnet (172.20.145.6) by DRHQMAIL107.nvidia.com
 (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 10 Feb
 2021 03:40:12 +0000
From: Alistair Popple <apopple@nvidia.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
CC: <linux-mm@kvack.org>, <nouveau@lists.freedesktop.org>,
	<bskeggs@redhat.com>, <akpm@linux-foundation.org>,
	<linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	<kvm-ppc@vger.kernel.org>, <dri-devel@lists.freedesktop.org>,
	<jhubbard@nvidia.com>, <rcampbell@nvidia.com>, <jglisse@redhat.com>
Subject: Re: [PATCH 1/9] mm/migrate.c: Always allow device private pages to migrate
Date: Wed, 10 Feb 2021 14:40:10 +1100
Message-ID: <1780857.6Ip0F2Sa4d@nvdebian>
In-Reply-To: <20210209133932.GD4718@ziepe.ca>
References: <20210209010722.13839-1-apopple@nvidia.com> <20210209010722.13839-2-apopple@nvidia.com> <20210209133932.GD4718@ziepe.ca>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"
X-Originating-IP: [172.20.145.6]
X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To
 DRHQMAIL107.nvidia.com (10.27.9.16)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1;
	t=1612928415; bh=tAmJ0+G97SMFAbiRRa3d8nEsiux+3wqoFMSUz5oWXEY=;
	h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References:
	 MIME-Version:Content-Transfer-Encoding:Content-Type:
	 X-Originating-IP:X-ClientProxiedBy;
	b=HROK2Gi63VmhyROJ40EZiAFFzcCoLcWvPFvxuOfp205Epk8rw5YTtBCGIYoAb9hxk
	 CnfqGNLFdzz2Xtsj3yumfUK/9+hBtfXrJEmX4iVqiMoaXgM+TMGxt18tqaG5sw0ScD
	 43ZxLw3DvD9R33aTIvurLvOsng5qWOn/AzmOA29piHPJdAChtXqQDQvf303F0aYOBT
	 svQatmXEX23sU3Ao/P9UZXWmvJHz/I0b/xjuoB1mUuhQHJ7wLVFQzTgKe0in2pLJIP
	 kkoNwUFXagV0YIb9Ks2CHAtmM5KrXk8JUZRq0BVnbpF6TD5kPWxc0Cx+2iFqHy0KUt
	 zNFD/OabsiQEw==
X-Stat-Signature: xp8akhcm49h9nxor8k37sw4c6tgm76jc
X-Rspamd-Server: rspam02
X-Rspamd-Queue-Id: 8514280191E9
Received-SPF: none (nvidia.com>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from="<apopple@nvidia.com>"; helo=hqnvemgate26.nvidia.com; client-ip=216.228.121.65
X-HE-DKIM-Result: pass/pass
X-HE-Tag: 1612928416-773636
X-Bogosity: Ham, tests=bogofilter, spamicity=0.001782, version=1.2.4
Sender: owner-linux-mm@kvack.org
Precedence: bulk
X-Loop: owner-majordomo@kvack.org
List-ID: <linux-mm.kvack.org>

On Wednesday, 10 February 2021 12:39:32 AM AEDT Jason Gunthorpe wrote:
> On Tue, Feb 09, 2021 at 12:07:14PM +1100, Alistair Popple wrote:
> > Device private pages are used to represent device memory that is not
> > directly accessible from the CPU. Extra references to a device private
> > page are only used to ensure the struct page itself remains valid whilst
> > waiting for migration entries. Therefore extra references should not
> > prevent device private page migration as this can lead to failures to
> > migrate pages back to the CPU which are fatal to the user process.
> 
> This should identify the extra references in expected_count, just
> disabling this protection seems unsafe, ZONE_DEVICE is not so special
> that the refcount means nothing

This is similar to what migarte_vma_check_page() does now. The issue is that a 
migration wait takes a reference on the device private page so you can end up 
with one thread stuck waiting for migration whilst the other can't migrate due 
to the extra refcount.

Given device private pages can't undergo GUP and that it's not possible to 
differentiate the migration wait refcount from any other refcount we assume 
any possible extra reference must be from migration wait.

> Is this a side effect of the extra refcounts that Ralph was trying to
> get rid of? I'd rather see that work finished :)

I'd like to see that finished too but I don't think it would help here as this 
is not a side effect of that.

 - Alistair

> Jason