From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D727EC2BA83 for ; Thu, 13 Feb 2020 16:25:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 99BF0217F4 for ; Thu, 13 Feb 2020 16:25:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="jvWeugFl" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 99BF0217F4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1BE436B0569; Thu, 13 Feb 2020 11:25:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 16EB56B056B; Thu, 13 Feb 2020 11:25:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 084546B056C; Thu, 13 Feb 2020 11:25:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id E536F6B0569 for ; Thu, 13 Feb 2020 11:25:13 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 84C4D181AEF15 for ; Thu, 13 Feb 2020 16:25:13 +0000 (UTC) X-FDA: 76485628506.15.fight08_39d66e432e801 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 1B34318034AA9 for ; Thu, 13 Feb 2020 16:13:18 +0000 (UTC) X-HE-Tag: fight08_39d66e432e801 X-Filterd-Recvd-Size: 5789 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Thu, 13 Feb 2020 16:13:17 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 01DG8bDG081734; Thu, 13 Feb 2020 16:12:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : in-reply-to; s=corp-2020-01-29; bh=yHRbjdVuk7O7QTB/SjSFhnXU75C7osXyU0rPDtWVgp0=; b=jvWeugFlLgnUtPZV3zT/RfNd7MXtn71rfu3xJf+uZV8MDnarcab7qQ3eY115BAA2yaWu uisSxZ/Zor0C7JATmfGl9Ivol5GXXxsW2Ln8EtFs+DMB2SqX8YYVwE5I5lGNp1jwDs5d GYAO8lUWQhITMuoOz5Mu4LGQeI/t2dXpOXQy0k4pk4g7GOQJAB5j/KryAUKvoZWafgoY EUT5Tgn+Vf4VCQ7PpliP+KVvdD1o181QVmKrnZTbaAzBoQbfDMbmXX0Tuyci3iansPa5 xhGFVJCuBVFMNGBX5MZ0ZB5Fib/z6e9mO0lumFc6gum1twB4fXjTGC6XiiYE5XPjZIcP 3Q== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 2y2k88k9pg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Feb 2020 16:12:57 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 01DFvwfh134048; Thu, 13 Feb 2020 16:12:56 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3030.oracle.com with ESMTP id 2y4k8043y6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Feb 2020 16:12:56 +0000 Received: from abhmp0018.oracle.com (abhmp0018.oracle.com [141.146.116.24]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 01DGCqVO027196; Thu, 13 Feb 2020 16:12:52 GMT Received: from ca-dmjordan1.us.oracle.com (/10.211.9.48) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 13 Feb 2020 08:12:51 -0800 Date: Thu, 13 Feb 2020 11:13:07 -0500 From: Daniel Jordan To: Jason Gunthorpe Cc: Daniel Jordan , lsf-pc@lists.linuxfoundation.org, linux-mm@kvack.org, Dan Williams , Dave Hansen , Tim Chen , Mike Kravetz , Herbert Xu , Steffen Klassert , Tejun Heo , Peter Zijlstra , Alex Williamson Subject: Re: [LSF/MM/BPF TOPIC] kernel multithreading with padata Message-ID: <20200213161307.3n5t62lqeyuljnhl@ca-dmjordan1.us.oracle.com> References: <20200212224731.kmss6o6agekkg3mw@ca-dmjordan1.us.oracle.com> <20200212233100.GF31668@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200212233100.GF31668@ziepe.ca> User-Agent: NeoMutt/20180716 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9530 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0 suspectscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002130121 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9530 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 lowpriorityscore=0 suspectscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 impostorscore=0 clxscore=1015 spamscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002130121 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Feb 12, 2020 at 07:31:00PM -0400, Jason Gunthorpe wrote: > On Wed, Feb 12, 2020 at 05:47:31PM -0500, Daniel Jordan wrote: > > padata has been undergoing some surgery over the last year[0] and now seems > > ready for another enhancement: splitting up and multithreading CPU-intensive > > kernel work. > > > > Quoting from an earlier series[1], the problem I'm trying to solve is > > > > A single CPU can spend an excessive amount of time in the kernel operating > > on large amounts of data. Often these situations arise during initialization- > > and destruction-related tasks, where the data involved scales with system > > size. These long-running jobs can slow startup and shutdown of applications > > and the system itself while extra CPUs sit idle. > > > > Here are the current consumers: > > > > - struct page init (boot, hotplug, pmem) > > - VFIO page pinning (kvm guest init) > > - fallocating a hugetlb file (database shared memory init) > > > > On a large-memory server, DRAM page init is ~23% of kernel boot (3.5s/15.2s), > > and it takes over a minute to start a VFIO-enabled kvm guest or fallocate a > > hugetlb file that occupy a significant fraction of memory. This work results > > in 7-20x speedups and is currently increasing the uptime of our production > > kernels. > > > > Future areas include munmap/exit, umount, and __ib_umem_release. Some of these > > need coarse locks broken up for multithreading (zone->lock, lru_lock). > > I'm aware of this ib_umem_release request, it would be interesting to > see, the main workload here is put_page and dma_unmap Ah yes, I see it gets all the way down to zone->lock, so I should've said _all_ of the future cases need coarse locks broken. By the way, there's an idea for dealing with zone->lock that I haven't yet had time to look at. http://lkml.kernel.org/r/20181018111632.GM5819@techsingularity.net