From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E94F5C43217 for ; Tue, 29 Mar 2022 20:11:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230234AbiC2UMt (ORCPT ); Tue, 29 Mar 2022 16:12:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230035AbiC2UMs (ORCPT ); Tue, 29 Mar 2022 16:12:48 -0400 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EA677B717B; Tue, 29 Mar 2022 13:11:03 -0700 (PDT) Received: from kbox (c-73-140-2-214.hsd1.wa.comcast.net [73.140.2.214]) by linux.microsoft.com (Postfix) with ESMTPSA id 5FB5F20DEDDC; Tue, 29 Mar 2022 13:11:03 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 5FB5F20DEDDC DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1648584663; bh=zHcnNu/YGM+GoY1MvB/irgfNbLk1NiTLk3aItql2YyU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Un4xZKfi9rZzz+TfjZAPywux3T4fExp4njC9SgfKhsQS6ouP5UbORmGW0HkUeIQGh QScxNRWUEcZprvu1GRQxysuUy5knLLxuMC8+0ei98M07NjpUe42vA7TPlCd6l7kNci LpvIu65Q80sgfP+wsp2k+WJsGRtAkX3G5gKXPrfo= Date: Tue, 29 Mar 2022 13:10:57 -0700 From: Beau Belgrave To: Alexei Starovoitov Cc: Steven Rostedt , Masami Hiramatsu , linux-trace-devel , LKML , bpf , Network Development , linux-arch , Mathieu Desnoyers Subject: Re: [PATCH] tracing/user_events: Add eBPF interface for user_event created events Message-ID: <20220329201057.GA2549@kbox> References: <20220329181935.2183-1-beaub@linux.microsoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org On Tue, Mar 29, 2022 at 12:50:40PM -0700, Alexei Starovoitov wrote: > On Tue, Mar 29, 2022 at 11:19 AM Beau Belgrave > wrote: > > > > Send user_event data to attached eBPF programs for user_event based perf > > events. > > > > Add BPF_ITER flag to allow user_event data to have a zero copy path into > > eBPF programs if required. > > > > Update documentation to describe new flags and structures for eBPF > > integration. > > > > Signed-off-by: Beau Belgrave > > The commit describes _what_ it does, but says nothing about _why_. > At present I see no use out of bpf and user_events connection. > The whole user_events feature looks redundant to me. > We have uprobes and usdt. It doesn't look to me that > user_events provide anything new that wasn't available earlier. A lot of the why, in general, for user_events is covered in the first change in the series. Link: https://lore.kernel.org/all/20220118204326.2169-1-beaub@linux.microsoft.com/ The why was also covered in Linux Plumbers Conference 2021 within the tracing microconference. An example of why we want user_events: Managed code running that emits data out via Open Telemetry. Since it's managed there isn't a stub location to patch, it moves. We watch the Open Telemetry spans in an eBPF program, when a span takes too long we collect stack data and perform other actions. With user_events and perf we can monitor the entire system from the root container without having to have relay agents within each cgroup/namespace taking up resources. We do not need to enter each cgroup mnt space and determine the correct patch location or the right version of each binary for processes that use user_events. An example of why we want eBPF integration: We also have scenarios where we are live decoding the data quickly. Having user_data fed directly to eBPF lets us cast the data coming in to a struct and decode very very quickly to determine if something is wrong. We can take that data quickly and put it into maps to perform further aggregation as required. We have scenarios that have "skid" problems, where we need to grab further data exactly when the process that had the problem was running. eBPF lets us do all of this that we cannot easily do otherwise. Another benefit from user_events is the tracing is much faster than uprobes or others using int 3 traps. This is critical to us to enable on production systems. Thanks, -Beau