From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C41C0C4332F for ; Sat, 10 Dec 2022 01:12:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229892AbiLJBMT (ORCPT ); Fri, 9 Dec 2022 20:12:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229545AbiLJBMS (ORCPT ); Fri, 9 Dec 2022 20:12:18 -0500 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CC62F010; Fri, 9 Dec 2022 17:12:16 -0800 (PST) Received: by mail-ej1-x630.google.com with SMTP id fc4so15211590ejc.12; Fri, 09 Dec 2022 17:12:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=w4ADGd8gswVi8TomaFcdrrYZPaXY9TBBtkDp4DMvfpE=; b=FhYRYTfh2eMILidzjM694/ryObUQPSo+rBTbKNRcEgkG0KoU1Ef+6/M3CdKkb/U0su RZMELU22J+MbnsrMTYEg0qlAPIDY/cBWMJiXFQP5ibBCJafyHdu0AHE3VBIbVA7DUOie ykA4Szl+o8KGVKp5bjA5uH+9ZNsNfptqfeqBaX8yotccGoLRbnmRqpUyR8mL620ol0QU 4z6WCPZnhNRRY+PiNpIMtw5CZfZDAJ9uvuV4UXimn0kp7FdPE1QxZKG+/eHM+o8DDm5/ ctsBABnqrGtVTrRZTwB738FJm/MhrNf+Sgd3scdYTP/qOUW+8/+hvZP+hEid6bMcfMEx 8FcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=w4ADGd8gswVi8TomaFcdrrYZPaXY9TBBtkDp4DMvfpE=; b=zbDwTbn9+4GuSACfTFtzBZz4VmXLkkw92KtnJL7+v0rYQE3IQ7vE0JYUp7qmDEmcFe TdWcRgop+dIuX6HkCTvfC9YkY3U4Ci4LChFrTIfPM9U05fOi2IXh8JivMToWSnj6iECD 5KNdPNj7dwGLU9weKenLkZvyx/R9xgmmIqPgkzPmxXX11zcTI5UUlUVl6Y7fARWK7Mex wkOVN9URIDhBWV0tjO33n1gKB3ZehQYrXKCsLlWXA+QaHwdBMTlSh+DgzZJcRayBLRnt SrL7m2cozXf6eP9t04sHXAXMySwYbRujfHp6QlSeUvz9VzzcpDI5b9K/XTk5dKsZl2L6 8Uww== X-Gm-Message-State: ANoB5pltXJZ7+Uyc09GGv2EOXcPSqKMdYH9s3TKZjXR3KjDm0iS/54mJ L5D2ml2jizfC9qo7WHaOJMk/A71PaCP8ogG4R34= X-Google-Smtp-Source: AA0mqf5aTKuKb/ygizN4hu5aKxFW2tN/RJZfTxdEPi61WR/NUIe7drXP/RIDZQsLAVsSj3am6WKF+wPcEUwhXbtDmOk= X-Received: by 2002:a17:906:4351:b0:78d:513d:f447 with SMTP id z17-20020a170906435100b0078d513df447mr69635532ejm.708.1670634734892; Fri, 09 Dec 2022 17:12:14 -0800 (PST) MIME-Version: 1.0 References: <5c9d77bf-75f5-954a-c691-39869bb22127@meta.com> <96b0d9d8-02a7-ce70-de1e-b275a01f5ff3@iogearbox.net> <20221209153445.22182ca5@kernel.org> In-Reply-To: From: Alexei Starovoitov Date: Fri, 9 Dec 2022 17:12:03 -0800 Message-ID: Subject: Re: BUG: unable to handle kernel paging request in bpf_dispatcher_xdp To: Jiri Olsa Cc: Jakub Kicinski , "Paul E. McKenney" , Daniel Borkmann , Yonghong Song , Song Liu , Hao Sun , Peter Zijlstra , bpf , Alexei Starovoitov , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , KP Singh , Stanislav Fomichev , Hao Luo , David Miller , Jesper Dangaard Brouer , Linux Kernel Mailing List , netdev , Thorsten Leemhuis Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Dec 9, 2022 at 4:06 PM Jiri Olsa wrote: > > On Fri, Dec 09, 2022 at 03:34:45PM -0800, Jakub Kicinski wrote: > > On Sat, 10 Dec 2022 00:32:07 +0100 Daniel Borkmann wrote: > > > fwiw, these should not be necessary, Documentation/RCU/checklist.rst : > > > > > > [...] One example of non-obvious pairing is the XDP feature in networking, > > > which calls BPF programs from network-driver NAPI (softirq) context. BPF > > > relies heavily on RCU protection for its data structures, but because the > > > BPF program invocation happens entirely within a single local_bh_disable() > > > section in a NAPI poll cycle, this usage is safe. The reason that this usage > > > is safe is that readers can use anything that disables BH when updaters use > > > call_rcu() or synchronize_rcu(). [...] > > > > FWIW I sent a link to the thread to Paul and he confirmed > > the RCU will wait for just the BH. > > so IIUC we can omit the rcu_read_lock/unlock on bpf_prog_run_xdp side > > Paul, > any thoughts on what we can use in here to synchronize bpf_dispatcher_change_prog > with bpf_prog_run_xdp callers? > > with synchronize_rcu_tasks I'm getting splats like: > https://lore.kernel.org/bpf/20221209153445.22182ca5@kernel.org/T/#m0a869f93404a2744884d922bc96d497ffe8f579f > > synchronize_rcu_tasks_rude seems to work (patch below), but it also sounds special ;-) Jiri, I haven't tried to repro this yet, but I feel you're on the wrong path here. The splat has this: ? bpf_prog_run_xdp include/linux/filter.h:775 [inline] ? bpf_test_run+0x2ce/0x990 net/bpf/test_run.c:400 that test_run logic takes rcu_read_lock. See bpf_test_timer_enter. I suspect the addition of synchronize_rcu_tasks_rude only slows down the race. The synchronize_rcu_tasks_trace also behaves like synchronize_rcu. See our new and fancy rcu_trace_implies_rcu_gp(), but I'm not sure it applies to synchronize_rcu_tasks_rude. Have you tried with just synchronize_rcu() ? If your theory about the race is correct then the vanila sync_rcu should help. If not, the issue is some place else.