From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8838CC43331 for ; Fri, 27 Mar 2020 20:17:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6962F20714 for ; Fri, 27 Mar 2020 20:17:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727322AbgC0URB (ORCPT ); Fri, 27 Mar 2020 16:17:01 -0400 Received: from www62.your-server.de ([213.133.104.62]:37466 "EHLO www62.your-server.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726959AbgC0URB (ORCPT ); Fri, 27 Mar 2020 16:17:01 -0400 Received: from sslproxy01.your-server.de ([78.46.139.224]) by www62.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89_1) (envelope-from ) id 1jHvPR-0006Dt-JP; Fri, 27 Mar 2020 21:16:53 +0100 Received: from [178.195.186.98] (helo=pc-9.home) by sslproxy01.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jHvPR-000N0M-7Z; Fri, 27 Mar 2020 21:16:53 +0100 Subject: Re: call for bpf progs. Re: [PATCHv2 bpf-next 5/5] selftests: bpf: add test for sk_assign To: Joe Stringer , Alexei Starovoitov Cc: Lorenz Bauer , Yonghong Song , bpf , Networking , Alexei Starovoitov , Eric Dumazet , Martin Lau , john fastabend References: <20200325055745.10710-1-joe@wand.net.nz> <20200325055745.10710-6-joe@wand.net.nz> <82e8d147-b334-3d29-0312-7b087ac908f3@fb.com> <20200326210719.den5isqxntnoqhmv@ast-mbp> From: Daniel Borkmann Message-ID: Date: Fri, 27 Mar 2020 21:16:52 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Authenticated-Sender: daniel@iogearbox.net X-Virus-Scanned: Clear (ClamAV 0.102.2/25764/Fri Mar 27 14:11:26 2020) Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org On 3/27/20 8:06 PM, Joe Stringer wrote: > On Thu, Mar 26, 2020 at 2:07 PM Alexei Starovoitov > wrote: >> The second concern was pruning, but iirc the experiments were inconclusive. >> selftests/bpf only has old fb progs. Hence, I think, the step zero is for >> everyone to contribute their bpf programs written in C. If we have both >> cilium and cloudflare progs as selftests it will help a lot to guide such long >> lasting verifier decisions. > > How would you like to handle program changes over time for this? > > In Cilium community we periodically rebuild bpf-next VM images for > testing, and run every pull request against those images[0]. We also > test against specific older kernels, currently 4.9 and 4.19. This > allows us to get some sense for the impact of upstream changes while > developing Cilium features, but unfortunately doesn't allow everyone > using kernel selftests to get that feedback at least from the kernel > tree. We also have a verifier complexity test script where we compile > with the maximum number of features (to ideally generate the most > complex programs possible) then attempt to load all of the various > programs, and output the complexity count that the kernel reports[1,2] > which we can track over time. > > However Cilium BPF programs are actively developing and even if we > merge these programs into the kernel tree, they will get out-of-date > quickly. Up until recently everything was verifying fine compiling > with LLVM7 and loading into bpf-next. Over the past month we started > noticing new issues not with the existing implementation, but in *new* > BPF features. As we increased complexity, our CI started failing > against bpf-next[3] while they loaded fine on older kernels. We ended > up mitigating by upgrading to LLVM-10. Long story short, there's > several moving parts; changing BPF program implementations, changing > the compiler toolchain, changing the kernel verifier. So my question > is basically, where's the line of responsibility for what the kernel > selftests are responsible for vs integration tests? How do we maintain > those over time as the BPF programs and compiler changes? I wonder whether it would make sense to create test cases which are based on our cilium/cilium:latest docker image. Those would be run as part of BPF kernel selftests as well as part of our own CI for every PR. I think it could be some basic connectivity test, service mapping, etc with a bunch of application containers. One nice thing that just came to mind is that it would actually allow for easy testing of latest clang/llvm git by rerunning the test script and remapping them as a volume, e.g.: docker run -it -v /root/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-18.04/bin/clang:/bin/clang \ -v /root/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-18.04/bin/llc:/bin/llc \ cilium/cilium:latest /bin/sh Perhaps that would be more useful and always up to date than a copy of the code base that would get stale next day? In the end in this context kernel changes and/or llvm changes might be of interest to check whether anything potentially blows up, so having a self-contained packaging might be useful. Thoughts? > Do we just parachute the ~11K LoC of Cilium datapath into the kernel > tree once per cycle? Or should Cilium autobuild a verifier-test docker > image that kernel testing scripts can pull & run? Or would it be > helpful to have a separate GitHub project similar to libbpf that pulls > out kernel selftests, Cilium progs, fb progs, cloudflare progs, etc > automatically and centralizes a generic suite of BPF verifier > integration tests? Some other option? > > [0] https://github.com/cilium/packer-ci-build > [1] https://github.com/cilium/cilium/blob/master/test/bpf/check-complexity.sh > [2] https://github.com/cilium/cilium/blob/master/test/bpf/verifier-test.sh > [3] https://github.com/cilium/cilium/issues/10517 >