From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEA92C636CC for ; Tue, 7 Feb 2023 22:54:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230027AbjBGWyi (ORCPT ); Tue, 7 Feb 2023 17:54:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229995AbjBGWyf (ORCPT ); Tue, 7 Feb 2023 17:54:35 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 529C427D5A for ; Tue, 7 Feb 2023 14:54:34 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id z13so4638123wmp.2 for ; Tue, 07 Feb 2023 14:54:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/GpMHLadhKBQ6+m2UPO236KZ/4fJUKcv4tl+ZjXMWOg=; b=sMbBMtAeCWA4vXGzLDmhjeUh/rfI6B5G76ykaIK+xBZyv8R88RMEuIgkvv4SL4sE5j XcCJCzyfIuCuP5ias1DX7y7J7Bw7iiicfY2/1rDoJgrENb2jXeeOZ+Z/B0ny4qnZqGXU W0Hzb1/J0OUk8+fbEEv/sEHP6SLRhfl/rTt05Xb+BlP0Mc4wBFq14RZC9Xc9fS4wQ52h LPd/lv9A7eqoRMRzarmYETR0tE3btg1LBGvTbhCLxwvHEKY/JVAKPHQw8ubkw/TdL74H 37Th2rf+1FsJ1Gvq92Sg5Hh7v7MGKaeWZewQPBs1ovU7oaekwd96HY8Df9NBiSz70QTG ah5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/GpMHLadhKBQ6+m2UPO236KZ/4fJUKcv4tl+ZjXMWOg=; b=I3Y6i0wTY6P6MckIbubg4e6XAEsVKYi/rhq7hMTcxIAYDNVEwbeYyDTbz0AOmef9mm n5RmUnsJcRfRDkbuCEjpIxCenAt4otum4IBgKyaifHzfhIztSh1K0+nAnrwhjcVnYyxc 6O7ljmg1kabYWs4R92NYz08vHFLRVDHe07WMpnDgCYuYbzOYB7GflUXDSH0d3DJusJ+f dTq/U/MKqWMtafpWvy2DVs9bnKAksl2jq2pk4izJq6eLjiMt/Q/Jg8q4+Jgh9ryGFxao fHQp0ubHFozqo2xna0ruTR9kzDM56SF6/dWK2xNM3CdoG8Rx5w20zXrMUL1Cxhd8eDHt FWZQ== X-Gm-Message-State: AO0yUKUtogxECoT9TGRUIUTwlmJhjZCYpqZCNrY/ZX5Bpjj5eEDPl8E/ vlEpTDdzPiMNTnUS7JeeN6iw98qNxrSvgKCiFhXIJw4qLx3+QM1I X-Google-Smtp-Source: AK7set+enYQAT6EptDabqZUbDV3esLdckJONxaTZ47keMG7iGdk0lZlEyJHzl1jWSuwBCj1kot2g4OCCuRxyvO0pSs4= X-Received: by 2002:a05:600c:220f:b0:3e0:15c:860f with SMTP id z15-20020a05600c220f00b003e0015c860fmr64175wml.115.1675810472693; Tue, 07 Feb 2023 14:54:32 -0800 (PST) MIME-Version: 1.0 References: <20230202012348.885402-1-nhuck@google.com> In-Reply-To: From: Nathan Huckleberry Date: Tue, 7 Feb 2023 14:54:21 -0800 Message-ID: Subject: Re: [dm-devel] [PATCH] dm-verity: Remove WQ_UNBOUND. To: Mikulas Patocka Cc: Mike Snitzer , linux-kernel@vger.kernel.org, Eric Biggers , dm-devel@redhat.com, Sami Tolvanen , Alasdair Kergon Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Feb 7, 2023 at 12:16 AM Mikulas Patocka wrote: > > > > On Wed, 1 Feb 2023, Nathan Huckleberry wrote: > > > Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely > > due to the asymmetric architecture of ARM64 processors. > > > > I've been unable to reproduce the results that claim WQ_UNBOUND gives a > > performance boost on x86-64. > > > > This flag is causing performance issues for multiple subsystems within > > Android. Notably, the same slowdown exists for decompression with > > EROFS. > > > > | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND | > > |-----------------------|------------|---------------| > > | verity wait time (us) | 11746 | 119 (-98%) | > > | erofs wait time (us) | 357805 | 174205 (-51%) | > > > > | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND | > > |----------------------------|-----------=---|-------------| > > | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s | > > | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s | > > | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s | > > > > Cc: Sami Tolvanen > > Cc: Eric Biggers > > Signed-off-by: Nathan Huckleberry > > --- > > drivers/md/dm-verity-target.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c > > index ccf5b852fbf7..020fd2341025 100644 > > --- a/drivers/md/dm-verity-target.c > > +++ b/drivers/md/dm-verity-target.c > > @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv) > > goto bad; > > } > > > > - /* WQ_UNBOUND greatly improves performance when running on ramdisk */ > > - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND; > > + wq_flags = WQ_MEM_RECLAIM; > > + > > /* > > * Using WQ_HIGHPRI improves throughput and completion latency by > > * reducing wait times when reading from a dm-verity device. > > Hi > > If you remove WQ_UNBOUND, you should also change the last argument of > alloc_workqueue from num_online_cpus() to either 0 or 1. Try both 0 and 1 > and tell us which performs better. They look roughly the same on ARM64. There's a slight advantage for using 1, but they're too close to conclusively say one is better. Thanks, Huck > > Mikulas > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ABE30C636CD for ; Tue, 7 Feb 2023 22:54:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675810486; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=MlP6aMbZqEeujl4uEbjyUi3YLkkTO/vKMvwu5xisibc=; b=RLJYtwBN/Mdw6uN4/7XpQBiag84lpLx04LBHtVibrHlkV6xKHmxnzKYGybmQ+IrXjcqzKN dAZTnkd95DV8Q2pwspexeL0+dLPXJl4spjN7P0Xh1/3vVsEG7ofaTYC4RVTjxX7sh1cT7f dXj4t7a3MKk85mLdRZ/Re2ouJrIhX68= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-426-JCKMZeLLNUO-5g7gZ9kh3w-1; Tue, 07 Feb 2023 17:54:43 -0500 X-MC-Unique: JCKMZeLLNUO-5g7gZ9kh3w-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C997C1C270A5; Tue, 7 Feb 2023 22:54:41 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9CF7A492B21; Tue, 7 Feb 2023 22:54:37 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 48EE4194658D; Tue, 7 Feb 2023 22:54:37 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id C04131946589 for ; Tue, 7 Feb 2023 22:54:35 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 9E364C15BAD; Tue, 7 Feb 2023 22:54:35 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast06.extmail.prod.ext.rdu2.redhat.com [10.11.55.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 971B0C15BA0 for ; Tue, 7 Feb 2023 22:54:35 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7989918A6461 for ; Tue, 7 Feb 2023 22:54:35 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-657-AkAciy-oPFyrX4ArW3qedQ-1; Tue, 07 Feb 2023 17:54:34 -0500 X-MC-Unique: AkAciy-oPFyrX4ArW3qedQ-1 Received: by mail-wm1-f50.google.com with SMTP id bg26so12131140wmb.0 for ; Tue, 07 Feb 2023 14:54:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/GpMHLadhKBQ6+m2UPO236KZ/4fJUKcv4tl+ZjXMWOg=; b=AQE0Ta6yqgiu6ALx3LmXYwgF6fpqhYnLjuDtWBEXoRHaJEuRRr/pDTnEn3tV/HKYU3 BVvSfmNu5Jja3uBzyWUda7xYOqYvQ90JjbLOPOvQYRwn3GDDky6mdBhQ/qkS6YiSsHkV OoLEGivZwwHA6QChi0O2iaWUx2jHJbugAwFFKF8CbLndqXLV5W8/kP42Rg4E4salp5lS guzmUqCwxUZxoCwWFSI/ao2AZL7uI1y5kQ0ps43ZYdhrVWk/o01Hlb7skV8zp7MkiSpD cXxqa+TsaSi4T8AKpfGvyHW/Kcu0+bN3ivya1aK3i7VHf6AES2/dcYMbeFBGZj7+YcXd OgyA== X-Gm-Message-State: AO0yUKV30e3YOL/Odt+BI/zLwrygVS0lEy+HT3QX4DJ0FgLas+oYrx1z dPRpLvVgZS40mr1XD5atZev1GladAVzeXBwS6Ebw9g== X-Google-Smtp-Source: AK7set+enYQAT6EptDabqZUbDV3esLdckJONxaTZ47keMG7iGdk0lZlEyJHzl1jWSuwBCj1kot2g4OCCuRxyvO0pSs4= X-Received: by 2002:a05:600c:220f:b0:3e0:15c:860f with SMTP id z15-20020a05600c220f00b003e0015c860fmr64175wml.115.1675810472693; Tue, 07 Feb 2023 14:54:32 -0800 (PST) MIME-Version: 1.0 References: <20230202012348.885402-1-nhuck@google.com> In-Reply-To: From: Nathan Huckleberry Date: Tue, 7 Feb 2023 14:54:21 -0800 Message-ID: To: Mikulas Patocka X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Subject: Re: [dm-devel] [PATCH] dm-verity: Remove WQ_UNBOUND. X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mike Snitzer , linux-kernel@vger.kernel.org, Eric Biggers , dm-devel@redhat.com, Sami Tolvanen , Alasdair Kergon Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Tue, Feb 7, 2023 at 12:16 AM Mikulas Patocka wrote: > > > > On Wed, 1 Feb 2023, Nathan Huckleberry wrote: > > > Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely > > due to the asymmetric architecture of ARM64 processors. > > > > I've been unable to reproduce the results that claim WQ_UNBOUND gives a > > performance boost on x86-64. > > > > This flag is causing performance issues for multiple subsystems within > > Android. Notably, the same slowdown exists for decompression with > > EROFS. > > > > | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND | > > |-----------------------|------------|---------------| > > | verity wait time (us) | 11746 | 119 (-98%) | > > | erofs wait time (us) | 357805 | 174205 (-51%) | > > > > | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND | > > |----------------------------|-----------=---|-------------| > > | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s | > > | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s | > > | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s | > > > > Cc: Sami Tolvanen > > Cc: Eric Biggers > > Signed-off-by: Nathan Huckleberry > > --- > > drivers/md/dm-verity-target.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c > > index ccf5b852fbf7..020fd2341025 100644 > > --- a/drivers/md/dm-verity-target.c > > +++ b/drivers/md/dm-verity-target.c > > @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv) > > goto bad; > > } > > > > - /* WQ_UNBOUND greatly improves performance when running on ramdisk */ > > - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND; > > + wq_flags = WQ_MEM_RECLAIM; > > + > > /* > > * Using WQ_HIGHPRI improves throughput and completion latency by > > * reducing wait times when reading from a dm-verity device. > > Hi > > If you remove WQ_UNBOUND, you should also change the last argument of > alloc_workqueue from num_online_cpus() to either 0 or 1. Try both 0 and 1 > and tell us which performs better. They look roughly the same on ARM64. There's a slight advantage for using 1, but they're too close to conclusively say one is better. Thanks, Huck > > Mikulas > -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel