From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDD60C48BCF for ; Sat, 12 Jun 2021 19:11:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8EC5761001 for ; Sat, 12 Jun 2021 19:11:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230136AbhFLTNP (ORCPT ); Sat, 12 Jun 2021 15:13:15 -0400 Received: from mail-ed1-f50.google.com ([209.85.208.50]:46629 "EHLO mail-ed1-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229985AbhFLTNP (ORCPT ); Sat, 12 Jun 2021 15:13:15 -0400 Received: by mail-ed1-f50.google.com with SMTP id s15so140260edt.13 for ; Sat, 12 Jun 2021 12:11:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BF4KTa2Agq0jbJMydgzuSSPg/d4XEACgY0H+fIBjXaw=; b=srFpC0SeLeMkVrDjlzK1DW3HKDXhgIITNsH1dfNDZIW98UQGzwOjxIsfL4oL/Par+m c325BPpRkPl+4uPJznfiMMw7K2cnTPKKPhxMmSpG37iPrSA1T3WHK7bXik4gzuEu/wyc 31SiUM5cp8kGuZyoC6TJ0E5SNnylKtM6oIFXlVXhuZT2abo/g0jaqNy3BTW1QSgdkGTr 3nXeZaipGtuuyWN5IeOMXVRivOnUS5ld7q0xAPSQJVTVIR56tvaL63vVUVxpxyDtqqBP mXFfee3XX/86wOJFUUlgn/QkWD3yH2i09HfpVWOYukEUPsl32LVj2mBotqEd7mNaEOhF ZCkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BF4KTa2Agq0jbJMydgzuSSPg/d4XEACgY0H+fIBjXaw=; b=HJ3ZNHz2NKW4cMNaN1Fjj7ltDKQIDGRBMfZjL8bX928golY2hykPHI25lIJWja0kPa ytpEwKsll7423FPrOP6ATsmCBs8oVUjR38ENgX+dz1sy1DjvJFDARenIQdlBxPOjZKAc ETvZB4t/EIfkrMM13KvyqpzOQRAFMxa0bvuCvv5HKmJP0rR6HFvG5qNa79GYWblfM4Lf fWwIPZH+Q8hNfuixb3/Kri21Qn9Y8sOiQ2UiRVYUbsR92H70DRyLg1I+pung9pXquUNj nPLs3rj4aAO9tZlGYGbe9A9puqLlJXOlvyHeoWnMYd9eSSSDAPTLcaLUfEmqQYXTpBsO DYbQ== X-Gm-Message-State: AOAM532ObwYKuzqpUXaxyN8X5nrmOvjnNVS5Y2wU80K097GmulCJRres pGkhH7RozNt1442Fwoc/jdtxChy30IltRZqaufAr X-Google-Smtp-Source: ABdhPJwqQz6M3ATeXs9AKuLxnc18TagDqn4vmELAttLKvYikYYiBAp4p1LMRAX02oCZAi3M7C7gTAQXrwzciEN+qB30= X-Received: by 2002:aa7:c782:: with SMTP id n2mr9796203eds.77.1623525014362; Sat, 12 Jun 2021 12:10:14 -0700 (PDT) MIME-Version: 1.0 References: <20210111081821.3041587-1-morbo@google.com> <20210407211704.367039-1-morbo@google.com> In-Reply-To: From: Bill Wendling Date: Sat, 12 Jun 2021 12:10:03 -0700 Message-ID: Subject: Re: [PATCH v9] pgo: add clang's Profile Guided Optimization infrastructure To: Peter Zijlstra Cc: Kees Cook , Jonathan Corbet , Masahiro Yamada , Linux Doc Mailing List , LKML , Linux Kbuild mailing list , clang-built-linux , Andrew Morton , Nathan Chancellor , Nick Desaulniers , Sami Tolvanen , Fangrui Song , "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-doc@vger.kernel.org ")On Sat, Jun 12, 2021 at 11:15 AM Peter Zijlstra wrote: > > On Sat, Jun 12, 2021 at 10:25:57AM -0700, Bill Wendling wrote: > > On Sat, Jun 12, 2021 at 9:59 AM Peter Zijlstra wrote: > > > > > > On Wed, Apr 07, 2021 at 02:17:04PM -0700, Bill Wendling wrote: > > > > From: Sami Tolvanen > > > > > > > > Enable the use of clang's Profile-Guided Optimization[1]. To generate a > > > > profile, the kernel is instrumented with PGO counters, a representative > > > > workload is run, and the raw profile data is collected from > > > > /sys/kernel/debug/pgo/profraw. > > > > > > > > The raw profile data must be processed by clang's "llvm-profdata" tool > > > > before it can be used during recompilation: > > > > > > > > $ cp /sys/kernel/debug/pgo/profraw vmlinux.profraw > > > > $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw > > > > > > > > Multiple raw profiles may be merged during this step. > > > > > > > > The data can now be used by the compiler: > > > > > > > > $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ... > > > > > > > > This initial submission is restricted to x86, as that's the platform we > > > > know works. This restriction can be lifted once other platforms have > > > > been verified to work with PGO. > > > > > > *sigh*, and not a single x86 person on Cc, how nice :-/ > > > > > This tool is generic and, despite the fact that it's first enabled for > > x86, it contains no x86-specific code. The reason we're restricting it > > to x86 is because it's the platform we tested on. > > You're modifying a lot of x86 files, you don't think it's good to let us > know? Worse, afaict this -fprofile-generate changes code generation, > and we definitely want to know about that. > I got the list of people to add from the scripts/get_maintainer.pl. The files you list below are mostly changes in Makefile, so it added the kbuild maintainers and list. There's a small change to the linker script to add the clang PGO data section, which is defined in "include/asm-generic/vmlinux.lds.h". Using the "kernel/gcov" initial implementation as a guildlline (2521f2c228ad750701ba4702484e31d876dbc386), there's one intel people CC'ed, but he didn't sign off on it. These patches were available for review for months now, and posted to all of the lists and CC'ed to the people from scripts/get_maintainers.pl. Perhaps that program should be improved? > > > > arch/x86/Kconfig | 1 + > > > > arch/x86/boot/Makefile | 1 + > > > > arch/x86/boot/compressed/Makefile | 1 + > > > > arch/x86/crypto/Makefile | 4 + > > > > arch/x86/entry/vdso/Makefile | 1 + > > > > arch/x86/kernel/vmlinux.lds.S | 2 + > > > > arch/x86/platform/efi/Makefile | 1 + > > > > arch/x86/purgatory/Makefile | 1 + > > > > arch/x86/realmode/rm/Makefile | 1 + > > > > arch/x86/um/vdso/Makefile | 1 + > > > > > > +CFLAGS_PGO_CLANG := -fprofile-generate > > > > +export CFLAGS_PGO_CLANG > > > > And which of the many flags in noinstr disables this? > > > > > These flags aren't used with PGO. So there's no need to disable them. > > Supposedly -fprofile-generate adds instrumentation to the generated > code. noinstr *MUST* disable that. If not, this is a complete > non-starter for x86. "noinstr" has "notrace", which is defined as "__attribute__((__no_instrument_function__))", which is honored by both gcc and clang. > > > Also, and I don't see this answered *anywhere*, why are you not using > > > perf for this? Your link even mentions Sampling Profilers (and I happen > > > to know there's been significant effort to make perf output work as > > > input for the PGO passes of the various compilers). > > > > > Instruction-based (non-sampling) profiling gives us a better > > context-sensitive profile, making PGO more impactful. It's also useful > > for coverage whereas sampling profiles cannot. > > We've got KCOV and GCOV support already. Coverage is also not an > argument mentioned anywhere else. Coverage can go pound sand, we really > don't need a third means of getting that. > Those aren't useful for clang-based implementations. And I like to look forward to potential improvements. > Do you have actual numbers that back up the sampling vs instrumented > argument? Having the instrumentation will affect performance which can > scew the profile just the same. > Instrumentation counts the number of times a branch is taken. Sampling is at a gross level, where if the sampling time is fine enough, you can get an idea of where the hot spots are, but it won't give you the fine-grained information that clang finds useful. Essentially, while sampling can "capture the hot spots very well", relying solely on sampling is basically leaving optimization on the floor. Our optimizations experts here have determined, through data of course, that instrumentation is the best option for PGO. > Also, sampling tends to capture the hot spots very well. -bw