From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D894C433EF for ; Fri, 8 Oct 2021 07:35:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 21B50610D1 for ; Fri, 8 Oct 2021 07:35:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 21B50610D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 93C246B0071; Fri, 8 Oct 2021 03:35:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8EAF96B0072; Fri, 8 Oct 2021 03:35:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B27A900002; Fri, 8 Oct 2021 03:35:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0168.hostedemail.com [216.40.44.168]) by kanga.kvack.org (Postfix) with ESMTP id 69C516B0071 for ; Fri, 8 Oct 2021 03:35:35 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 11BBF18210E85 for ; Fri, 8 Oct 2021 07:35:35 +0000 (UTC) X-FDA: 78672460230.15.282BB20 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 9C73BD03BFE6 for ; Fri, 8 Oct 2021 07:35:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633678533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TWJM4mde5WNZKPeQbiPbITXNzh/uwce4/rOsOzTt7kA=; b=OEdEb8Im+83BhX63u2W8T9y7zT5us48LWb+YGuxbpzAl+9ngJWymGZVxSdmiArqqX9Fsuk cpz68uDTnIbYZLpyjmZ9+2rlJGk6ag7oh0P4GmeKpLIzQhX8RCitIC+J6kIL9NOq4NVRnx iBCtCfMMm2esgeWAadnJTRK2ESN3fvM= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-126-VQ8VnmrINFK9f0Mp7UlMkg-1; Fri, 08 Oct 2021 03:35:32 -0400 X-MC-Unique: VQ8VnmrINFK9f0Mp7UlMkg-1 Received: by mail-wr1-f69.google.com with SMTP id k16-20020a5d6290000000b00160753b430fso6626302wru.11 for ; Fri, 08 Oct 2021 00:35:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:to:cc:references:from:organization:subject :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=TWJM4mde5WNZKPeQbiPbITXNzh/uwce4/rOsOzTt7kA=; b=On0sxRdWWUJyvWRTjpcMhreMnFqBZqGaP86qV5qg/RHajfcHwasizlqfO15iWLUJ65 hMAjSnwkRp0mVvzp5mVnPuf+/ytEXzzv/4d2yUj2IcbhyWGCfE36QnmXjXY9LUQg/BYc Z8i5cHsb8owuheCwmBM7OG67IwGmndZu9nQycJW734KR/0YNjsE0JVXCSVIshqiygVou ci/r5GdQFfSAzt5NHT9F3N6ET9IGf0+YTTlB7Fbk3majqprGQeIdWoRHlgWBVbPqk5VJ zWjIVj43rTqOYrk7JetHjtq7bIFk11OURecVCQdmhOfKm+ckrL7xUhQkDnklA9IF2/fc K88A== X-Gm-Message-State: AOAM533aZgmND+Lz79pM9OxJ52IgUrscw1vKXx9N3Uci8V2rnGwJMIHI BFQQy8PHRThtut9G56j3S8sOvc2duaLl8PAov/nY+gpsgqjej5lkHvjAnraurfEbzHk0+fR67rn ar2J7xJKmiCs= X-Received: by 2002:a05:600c:1d1f:: with SMTP id l31mr1788511wms.44.1633678531032; Fri, 08 Oct 2021 00:35:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwFkzTDe9ji1jO/xBwXfgqZYCkTV97BH9m5OJEaRZWSr5Pr72efXoUijH5Tu+L5uftU3NbeYg== X-Received: by 2002:a05:600c:1d1f:: with SMTP id l31mr1788478wms.44.1633678530771; Fri, 08 Oct 2021 00:35:30 -0700 (PDT) Received: from [192.168.3.132] (p5b0c676e.dip0.t-ipconnect.de. [91.12.103.110]) by smtp.gmail.com with ESMTPSA id i3sm1647345wrn.34.2021.10.08.00.35.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 08 Oct 2021 00:35:30 -0700 (PDT) To: Nadav Amit Cc: Andrew Morton , LKML , Linux-MM , Peter Xu , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin , "x86@kernel.org" References: <20210925205423.168858-1-namit@vmware.com> <20210925205423.168858-3-namit@vmware.com> <5485fae5-3cd6-9dc3-0579-dc8aab8a3de1@redhat.com> <5356D62E-1900-4E92-AF23-AA5625EFFD92@vmware.com> <1952fc7c-fb21-7d0e-661b-afa59b4580e5@redhat.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH 2/2] mm/mprotect: do not flush on permission promotion Message-ID: Date: Fri, 8 Oct 2021 09:35:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Rspamd-Queue-Id: 9C73BD03BFE6 X-Stat-Signature: q49f4m3yetsptr95amoacufiib8w3yjz Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OEdEb8Im; spf=none (imf21.hostedemail.com: domain of david@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam06 X-HE-Tag: 1633678534-977419 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: >> >> Any numbers would be helpful. >> >>> If you want, I will write a microbenchmarks and give you numbers. >>> If you look for further optimizations (although you did not indicate >>> so), such as doing the TLB batching from do_mprotect_key(), >>> (i.e. batching across VMAs), we can discuss it and apply it on >>> top of these patches. >> >> I think this patch itself is sufficient if we can show a benefit; I do= wonder if existing benchmarks could already show a benefit, I feel like = they should if this makes a difference. Excessive mprotect() usage (prote= ct<>unprotect) isn't something unusual. >=20 > I do not know about a concrete benchmark (other than my workload, which= I cannot share right now) that does excessive mprotect() in a way that w= ould actually be measurable on the overall performance. I would argue tha= t many many optimizations in the kernel are such that would not have so m= easurable benefit by themselves on common macrobenchmarks. >=20 > Anyhow, per your request I created a small micro-benchmark that runs mp= rotect(PROT_READ) and mprotect(PROT_READ|PROT_WRITE) in a loop and measur= ed the time it took to do the latter (where a writeprotect is not needed)= . I ran the benchmark on a VM (guest) on top of KVM. >=20 > The cost (cycles) per mprotect(PROT_READ|PROT_WRITE) operation: >=20 > 1 thread 2 threads > -------- --------- > w/patch: 2496 2505 =09 > w/o patch: 5342 10458 >=20 For my taste, the above numbers are sufficient, thanks! > [ The results for 1 thread might seem strange as one can expect the ove= rhead in this case to be no more than ~250 cycles, which is the time a TL= B invalidation of a single PTE takes. Yet, this overhead are probably rel= ated to =E2=80=9Cpage fracturing=E2=80=9D, which happens when the VM memo= ry is backed by 4KB pages. In such scenarios, a single PTE invalidation i= n the VM can cause on Intel a full TLB flush. The full flush is needed to= ensure that if the invalidated address was mapped through huge page in t= he VM, any relevant 4KB mapping that is cached in the TLB (after fracturi= ng due to the 4KB GPA->HPA mapping) would be removed.] Very nice analysis :) >=20 > Let me know if you want me to share the micro-benchmark with you. I am = not going to mention the results in the commit log, because I think the o= verhead of unnecessary TLB invalidation is well established. Just let me clarify why I am asking at all, it could be that: a) The optimization is effective and applicable to many workloads b) The optimization is effective and applicable to some workloads=20 ("micro benchmark") c) The optimization is ineffective d) The optimization is wrong IMHO: We can rule out d) by review and tests. We can rule out c) by=20 simple benchmarks easily. Maybe extend the patch description by something like: "The benefit of this optimization can already be visible when doing=20 mprotect(PROT_READ) -> mprotect(PROT_READ|PROT_WRITE) on a single=20 thread, because we end up requiring basically no TLB flushes. The=20 optimization gets even more significant with more threads. See [1] for=20 simple micro benchmark results." Something like that would be good enough for my taste. --=20 Thanks, David / dhildenb