From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83C8AC4338F for ; Fri, 23 Jul 2021 11:29:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6B415608FE for ; Fri, 23 Jul 2021 11:29:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234480AbhGWKsy (ORCPT ); Fri, 23 Jul 2021 06:48:54 -0400 Received: from mail.kernel.org ([198.145.29.99]:35810 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231703AbhGWKsx (ORCPT ); Fri, 23 Jul 2021 06:48:53 -0400 Received: from gandalf.local.home (cpe-66-24-58-225.stny.res.rr.com [66.24.58.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6BF4D60E75; Fri, 23 Jul 2021 11:29:24 +0000 (UTC) Date: Fri, 23 Jul 2021 07:29:06 -0400 From: Steven Rostedt To: Stefan Metzmacher Cc: Ingo Molnar , linux-trace-devel@vger.kernel.org, io-uring , "linux-kernel@vger.kernel.org" Subject: Re: sched_waking vs. set_event_pid crash (Re: Tracing busy processes/threads freezes/stalls the whole machine) Message-ID: <20210723072906.4f4e7bd5@gandalf.local.home> In-Reply-To: References: <293cfb1d-8a53-21e1-83c1-cdb6e2f32c65@samba.org> <20210504092404.6b12aba4@gandalf.local.home> <20210504093550.5719d4bd@gandalf.local.home> <8bb757fb-a83b-0ed5-5247-8273be3925c5@samba.org> <90c806a0-8a2f-1257-7337-6761100217c9@samba.org> <4ebea8f0-58c9-e571-fd30-0ce4f6f09c70@samba.org> <20210722225124.6d7d7153@rorschach.local.home> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 23 Jul 2021 12:35:09 +0200 Stefan Metzmacher wrote: > > Assuming this does fix your issue, I sent out a real patch with the > > explanation of what happened in the change log, so that you can see why > > that change was your issue. > > Yes, it does the trick, thanks very much! Can I get a "Tested-by" from you? > Now I can finally use: > > trace-cmd record -e all -P $(pidof io_uring-cp-forever) > > But that doesn't include the iou-wrk-* threads > and the '-c' option seems to only work with forking. Have you tried it? It should work for threads as well. It hooks to the sched_process_fork tracepoint, which should be triggered even when a new thread is created. Or do you mean that you want that process and all its threads too that are already running? I could probably have it try to add it via the /proc file system in that case. Can you start the task via trace-cmd? trace-cmd record -e all -F -c io_uring-cp-forever ... > > Is there a way to specify "trace *all* threads of the given pid"? > (Note the threads are comming and going, so it's not possible to > specifiy -P more than once) Right, although, you could append tasks manually to the set_event_pid file from another terminal after starting trace-cmd. Once a pid is added to that file, all children it makes will also be added. That could be a work around until we have trace-cmd do it. Care to write a bugzilla report for this feature? https://bugzilla.kernel.org/buglist.cgi?component=Trace-cmd%2FKernelshark&list_id=1088173 -- Steve