From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B4B8C4742C for ; Thu, 5 Nov 2020 04:34:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0EE2C2087D for ; Thu, 5 Nov 2020 04:34:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1604550856; bh=nzOvFi2K6r9ky9rNj+0evTDuBFWrs6aixLzCukn2RG4=; h=From:To:CC:Subject:Date:List-ID:From; b=wAJaUIClJDt83/jy95O2E4xXDXLkVfTpTAl3T78LXLChmcugVOk/PQ7YgG3b0eGkx K30CEazTGjfMT8pYNIJBRAi0a6MUwHQ3veY+3qkwipf3t5k5D/YyW4B188m6mARPn9 l1RiKR1qB8rTjSBUYI/yQ8w02OgQUU60WLNEfK4c= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730411AbgKEEeP convert rfc822-to-8bit (ORCPT ); Wed, 4 Nov 2020 23:34:15 -0500 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:23302 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1730705AbgKEEeO (ORCPT ); Wed, 4 Nov 2020 23:34:14 -0500 Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.16.0.42/8.16.0.42) with SMTP id 0A54X32F017947 for ; Wed, 4 Nov 2020 20:34:13 -0800 Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net with ESMTP id 34jthepagf-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 04 Nov 2020 20:34:13 -0800 Received: from intmgw002.08.frc2.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Wed, 4 Nov 2020 20:34:12 -0800 Received: by devbig012.ftw2.facebook.com (Postfix, from userid 137359) id 80C262EC8E04; Wed, 4 Nov 2020 20:34:03 -0800 (PST) From: Andrii Nakryiko To: , , , CC: , Subject: [PATCH v2 bpf-next 00/11] libbpf: split BTF support Date: Wed, 4 Nov 2020 20:33:50 -0800 Message-ID: <20201105043402.2530976-1-andrii@kernel.org> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8BIT X-FB-Internal: Safe Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737 definitions=2020-11-05_01:2020-11-05,2020-11-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 mlxscore=0 suspectscore=8 mlxlogscore=999 malwarescore=0 phishscore=0 impostorscore=0 bulkscore=0 priorityscore=1501 clxscore=1015 lowpriorityscore=0 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011050032 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org This patch set adds support for generating and deduplicating split BTF. This is an enhancement to the BTF, which allows to designate one BTF as the "base BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g., kernel module BTF), which are building upon and extending base BTF with extra types and strings. Once loaded, split BTF appears as a single unified BTF superset of base BTF, with continuous and transparent numbering scheme. This allows all the existing users of BTF to work correctly and stay agnostic to the base/split BTFs composition. The only difference is in how to instantiate split BTF: it requires base BTF to be alread instantiated and passed to btf__new_xxx_split() or btf__parse_xxx_split() "constructors" explicitly. This split approach is necessary if we are to have a reasonably-sized kernel module BTFs. By deduping each kernel module's BTF individually, resulting module BTFs contain copies of a lot of kernel types that are already present in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my kernel configuration with 700 modules built, non-split BTF approach results in 115MBs of BTFs across all modules. With split BTF deduplication approach, total size is down to 5.2MBs total, which is on part with vmlinux BTF (at around 4MBs). This seems reasonable and practical. As to why we'd need kernel module BTFs, that should be pretty obvious to anyone using BPF at this point, as it allows all the BTF-powered features to be used with kernel modules: tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc. This patch set is a pre-requisite to adding split BTF support to pahole, which is a prerequisite to integrating split BTF into the Linux kernel build setup to generate BTF for kernel modules. The latter will come as a follow-up patch series once this series makes it to the libbpf and pahole makes use of it. Patch #4 introduces necessary basic support for split BTF into libbpf APIs. Patch #8 implements minimal changes to BTF dedup algorithm to allow deduplicating split BTFs. Patch #11 adds extra -B flag to bpftool to allow to specify the path to base BTF for cases when one wants to dump or inspect split BTF. All the rest are refactorings, clean ups, bug fixes and selftests. v1->v2: - addressed Song's feedback. Andrii Nakryiko (11): libbpf: factor out common operations in BTF writing APIs selftest/bpf: relax btf_dedup test checks libbpf: unify and speed up BTF string deduplication libbpf: implement basic split BTF support selftests/bpf: add split BTF basic test selftests/bpf: add checking of raw type dump in BTF writer APIs selftests libbpf: fix BTF data layout checks and allow empty BTF libbpf: support BTF dedup of split BTFs libbpf: accomodate DWARF/compiler bug with duplicated identical arrays selftests/bpf: add split BTF dedup selftests tools/bpftool: add bpftool support for split BTF tools/bpf/bpftool/btf.c | 9 +- tools/bpf/bpftool/main.c | 15 +- tools/bpf/bpftool/main.h | 1 + tools/lib/bpf/btf.c | 807 ++++++++++-------- tools/lib/bpf/btf.h | 8 + tools/lib/bpf/libbpf.map | 9 + tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/btf_helpers.c | 259 ++++++ tools/testing/selftests/bpf/btf_helpers.h | 19 + tools/testing/selftests/bpf/prog_tests/btf.c | 40 +- .../bpf/prog_tests/btf_dedup_split.c | 325 +++++++ .../selftests/bpf/prog_tests/btf_split.c | 99 +++ .../selftests/bpf/prog_tests/btf_write.c | 43 + tools/testing/selftests/bpf/test_progs.h | 11 + 14 files changed, 1292 insertions(+), 355 deletions(-) create mode 100644 tools/testing/selftests/bpf/btf_helpers.c create mode 100644 tools/testing/selftests/bpf/btf_helpers.h create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_split.c -- 2.24.1