From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C55ABC04AB4 for ; Tue, 14 May 2019 13:58:42 +0000 (UTC) Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5EFC12147A for ; Tue, 14 May 2019 13:58:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LlLpyE/v" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5EFC12147A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=fail smtp.mailfrom=kernelnewbies-bounces+kernelnewbies=archiver.kernel.org@kernelnewbies.org Received: from localhost ([::1] helo=shelob.surriel.com) by shelob.surriel.com with esmtp (Exim 4.91) (envelope-from ) id 1hQXx2-0007vL-BU for kernelnewbies@archiver.kernel.org; Tue, 14 May 2019 09:58:40 -0400 Received: from mail-io1-xd2d.google.com ([2607:f8b0:4864:20::d2d]) by shelob.surriel.com with esmtps (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from ) id 1hQXvL-0005md-1P for kernelnewbies@kernelnewbies.org; Tue, 14 May 2019 09:56:55 -0400 Received: by mail-io1-xd2d.google.com with SMTP id s20so13064162ioj.7 for ; Tue, 14 May 2019 06:56:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=/OKLYDB3rlbIiiQWL94EYCqEkY6Tspld4MmS8gbo1q0=; b=LlLpyE/vOLUYXx0hBj/I5EuIo2xq7SntZDDbdks+sB99SVVAmeFvmIkKAu34xlw2wN oPbOgv5LaTPXNuBKiQVSdM4exO0C2oP/vKwa7NfKje6WUpNKvbeyYorPW9KDLnKFp6GA IaWNXj93APiLJoTOlzWEsFWHBp8DjZoCGQ3Lax5EoAdyScHQxUD5jZLfVj15s8iO8CH8 lyEAiNs9pnTGdD2xcQ0CWp+qq2BVOTrk1cl7V7/6PULrGEwcIgVsY6r03kWozSCSWmQM 1/oYhEMOH354DLauFky/D1GZM9r9Mx4Oe0JWAhtIGXLqkHQX2SuQ6zdPaWLtYR0tM4YM 6SWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=/OKLYDB3rlbIiiQWL94EYCqEkY6Tspld4MmS8gbo1q0=; b=MSpHr7gb81w/4XQMRS+L7vIIVXqyC05tzSzGqStKSkj8ZLuDqUD08/tdGHYYzP6dsV 1qSH2SaKIg30ihySRFGodCksBPldA7cD9WFwcYS90GNKX+w1rYOkmMi8r4Dsk5rkCqlb UhNi0JzxzCILPzFPH0CLcWogv/84f+s/s1V10mLDUeAGQYRqJZHjbMG0wH5hbz0tehJj 5qFX//tt/i7VSBWyamGhcDaTXKAtI3HaL7SQYvr0GiOm/qk/E/LkyoyMANq3dwYt28d+ MAKYr7nPOoALI13ku6I3SSI575t//3NSDD8NYmJpRbSicwtU6tbzXDxKI0ptRnWiGqVY fNxQ== X-Gm-Message-State: APjAAAVsh+sBFX7MobmkZap35j4RBzWKdN7VnxLHOaUb/WO6eqc/OAuc oGifNaNrmWNhpeNpjvwRD3zEMjxY/bS7fnBjdSLy1zWp X-Google-Smtp-Source: APXvYqzWXQ3oQI8xTtMESse+GCkOfEgJ2C+0J5PmkayEoo3XXHFjTfiWKUDDIU05609KmXI3GtU6QFGJNOWW4lGgUWo= X-Received: by 2002:a5d:9548:: with SMTP id a8mr1858667ios.54.1557842152305; Tue, 14 May 2019 06:55:52 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Pedro Terra Delboni Date: Tue, 14 May 2019 10:55:40 -0300 Message-ID: Subject: Re: how to collect information regarding function calls in run time? To: kernelnewbies@kernelnewbies.org X-BeenThere: kernelnewbies@kernelnewbies.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Learn about the Linux kernel List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============5187342245885841652==" Errors-To: kernelnewbies-bounces+kernelnewbies=archiver.kernel.org@kernelnewbies.org --===============5187342245885841652== Content-Type: multipart/alternative; boundary="0000000000001abad70588d9642c" --0000000000001abad70588d9642c Content-Type: text/plain; charset="UTF-8" Sorry to revive this old thread, and thank you for the suggestions given, but I've been trying to make these work for my case with little success and I need a hand to understand what I'm doing wrong. Regarding bpftrace: This seemed like the best option since I could use it to count frames of the stack with depth 2, allowing me to know precisely the amount of times each specific call has been made. However, I could not use it because since I have to probe every function, it would raise an error related to open file limit. I've tried setting the open file limit to unlimited, but the command I used to do so said it was impossible, also the current limit is set to 1048576, so I'm guessing that probing every function isn't a viable solution. Regarding perf: I couldn't find out how to count each call with perf, but I could count how many times each function was called. Differently from bpftrace, I need to use a command for each function I would like to probe, so I created a script that would get all the functions that can be probed and call that command for each of them. Unfortunately this took too long, so I stopped after a certain number of probes were created and executed my test, at that time, the machine hanged. With a low number of functions being probed it worked fine though. Right now, the solution I can see is using bpftrace to probe a very limited number of calls, profile an execution, then repeat the profile for different set of probes, which can take a very long time. Is there an easier way (less time consuming) to achieve the same results (counting how many times each call instruction has been executed)? Thanks you for the attention, Pedro On Wed, Apr 3, 2019 at 6:23 PM Augusto Mecking Caringi < augustocaringi@gmail.com> wrote: > Hi, > > On Wed, Apr 3, 2019 at 9:26 PM Pedro Terra Delboni > wrote: > > > > Hello! > > > > I have a question about profiling, please, let me know if this is not > > the right mailing list to post these kind of questions. > > > > Is there a way for me to record how many times each specific direct > > call to a function happened? > > I've seen ways of profiling the kernel that calculate how much time > > the system spent in each function, but having a bit more information > > related to calls would be really good. > > > > I would like to know, for every function call that happens in run > > time, who called it (it's for a project in my University). > > > > I've also seen (I may be mistaken here) that by compiling the kernel > > with perf, each function will start with a stub call which can be used > > for profiling purposes. > > I was thinking in using this stub to plug a function to dump > > (somewhere) the return address before it's own (so I can collect the > > info about where the call came from). > > I wonder if changing every stub calls in all functions to dump its > > return address wouldn't create too much of a latency impact to the > > point of skewing the control flow of the execution, > > or even making it nonviable. > > > > Thanks in advance, any help would be great! > > If this is not the right place to post this question, I would > > appreciate if anyone could point me to the right place. > > There is also a new tool called bpftrace that can help you... > > https://github.com/iovisor/bpftrace > > Regards, > > -- > Augusto Mecking Caringi > --0000000000001abad70588d9642c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Sorry to revive thi= s old thread, and thank you for the suggestions given, but I've been tr= ying to make these work for my case with little success and I need a hand t= o understand what I'm doing wrong.

Regardi= ng bpftrace: This seemed like the best option since I could use it to count= frames of the stack with depth 2, allowing me to know precisely the amount= of times each specific call has been made. However, I could not use it bec= ause since I have to probe every function, it would raise an error related = to open file limit. I've tried setting the open file limit to unlimited= , but the command I used to do so said it was impossible, also the current = limit is set to 1048576, so I'm guessing that probing every function is= n't a viable solution.

Regarding perf: I could= n't find out how to count each call with perf, but I could count how ma= ny times each function was called. Differently from bpftrace, I need to use= a command for each function I would like to probe, so I created a script t= hat would get all the functions that can be probed and call that command fo= r each of them. Unfortunately this took too long, so I stopped after a cert= ain number of probes were created and executed my test, at that time, the m= achine hanged.
With a low number of functions being probed it wor= ked fine though.

Right now, the solution I can see= is using bpftrace to probe a very limited number of calls, profile an exec= ution, then repeat the profile for different set of probes, which can take = a very long time.
Is there an easier way (less time consuming) to= achieve the same results (counting how many times each call instruction ha= s been executed)?

Thanks you for the attention,
Pedro

On Wed, Apr 3, 2019 at 6:23 PM Augusto Mec= king Caringi <augustocaringi@gmail.com> wrote:
Hi,

On Wed, Apr 3, 2019 at 9:26 PM Pedro Terra Delboni <terra0009@gmail.com> wrote:
>
> Hello!
>
> I have a question about profiling, please, let me know if this is not<= br> > the right mailing list to post these kind of questions.
>
> Is there a way for me to record how many times each specific direct > call to a function happened?
> I've seen ways of profiling the kernel that calculate how much tim= e
> the system spent in each function, but having a bit more information > related to calls would be really good.
>
> I would like to know, for every function call that happens in run
> time, who called it (it's for a project in my University).
>
> I've also seen (I may be mistaken here) that by compiling the kern= el
> with perf, each function will start with a stub call which can be used=
> for profiling purposes.
> I was thinking in using this stub to plug a function to dump
> (somewhere) the return address before it's own (so I can collect t= he
> info about where the call came from).
> I wonder if changing every stub calls in all functions to dump its
> return address wouldn't create too much of a latency impact to the=
> point of skewing the control flow of the execution,
> or even making it nonviable.
>
> Thanks in advance, any help would be great!
> If this is not the right place to post this question, I would
> appreciate if anyone could point me to the right place.

There is also a new tool called bpftrace that can help you...

https://github.com/iovisor/bpftrace

Regards,

--
Augusto Mecking Caringi
--0000000000001abad70588d9642c-- --===============5187342245885841652== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies --===============5187342245885841652==--