linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com>
To: Ivan Babrou <ivan@cloudflare.com>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
	kernel-team <kernel-team@cloudflare.com>,
	Jiri Olsa <jolsa@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Namhyung Kim <namhyung@kernel.org>,
	sashal@kernel.org, Kenton Varda <kenton@cloudflare.com>
Subject: Re: perf not picking up symbols for namespaced processes
Date: Thu, 5 Dec 2019 09:33:02 -0300	[thread overview]
Message-ID: <20191205123302.GA25750@kernel.org> (raw)
In-Reply-To: <CABWYdi1ZKR=jmKnjoJTik08Q9uJBvyZ4W0C29iPiUJ5ef1obvw@mail.gmail.com>

Em Wed, Dec 04, 2019 at 07:46:10PM -0800, Ivan Babrou escreveu:
> We have a service that forks a child process in a namespace-based
> sandbox where the mount namespace is intentionally designed to reflect
> a totally empty filesystem. Our use case is very similar to Chrome's
> sandbox, for example, but on a server. Within the sandbox, not even
> the service's own binary is present in the mount namespace.
> 
> Process tree looks like this:
> 
> $ sudo pstree -psc 63989
> edgeworker(63989)─┬─edgeworker/sbox(255716)─┬─edgeworker/zygt(255718)
>                    │                         ├─{edgeworker/sbox}(255719)
>                    │                         ├─{edgeworker/sbox}(255720)
>                    │                         ├─{edgeworker/sbox}(255721)
>                    ├─edgeworker/stry(5803)
>                    ├─edgeworker/stry(63990)
>                    ├─edgeworker/stry(106218)
>                    ├─edgeworker/stry(191905)
>                    ├─edgeworker/stry(255695)
>                    ├─edgeworker/supr(255717)
> 
> Here sbox processes do actual work living in an empty mount namespaces
> and stry is a helper process for error reporting. All tasks come from
> the same binary that lives in the root mount namespace, launched by
> systemd.
> 
> During "perf script" run on a trace obtained from the system there are
> these possible outcomes:
> 
> 1. The first pid to be processed is a non-namespaced helper and
> symbols are present.
> 2. The first pid is not found and symbols are present.
> 3. The first pid is a sandboxed task and symbols are missing.
> 
> Symbols are missing, because "perf script" tries to jump into an empty
> sandbox and find a binary there, when in fact it lives outside:
> 
> getcwd("/state/home/ivan", 4096)        = 17
> open("/proc/self/ns/mnt", O_RDONLY)     = 5
> open("/proc/255719/ns/mnt", O_RDONLY)   = 6
> setns(6, CLONE_NEWNS)                   = 0
> stat("/usr/local/bin/edgeworker", 0x7ffedb9b3ca0) = -1 ENOENT (No such
> file or directory)
> 
> In the second outcome we don't have a PID to figure out the namespace
> to jump into, so this doesn't happen. It's a good fallback, but it was
> a bit confusing during debugging.
> 
> It's not entirely clear to me why sometimes a helper PID is picked,
> even though it's not the first sample in the recorded trace (at least
> not in the output). This happens deterministically, or at least
> appears so. In my process tree it's 255695.
> 
> I think perf should try to fallback to the default namespace to look
> up symbols if they are not found inside to cover our case. Relevant
> piece of logic is here:

That should work for your use case, as you're sure that looking up by
pathname only will find, outside the namespace, the binary you want.

Even with pathname based looukups being fragile, it works for your
usecase, so please consider providing a patch for such fallback,
together with a pr_debug() or even pr_warning() if this don't get too
noisy, to warn the user.

- Arnaldo
 
> * https://elixir.free-electrons.com/linux/v5.4.1/source/tools/perf/util/dso.c#L520

-- 

- Arnaldo

  reply	other threads:[~2019-12-05 12:33 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-05  3:46 perf not picking up symbols for namespaced processes Ivan Babrou
2019-12-05 12:33 ` Arnaldo Carvalho de Melo [this message]
2019-12-06  2:17   ` Ivan Babrou
2020-02-04 15:09     ` Marek Majkowski
2020-02-04 19:26       ` Jiri Olsa
2020-02-11 10:06         ` Marek Majkowski
2020-02-11 13:46           ` Arnaldo Carvalho de Melo
2020-02-11 13:54             ` Marek Majkowski
2020-02-11 14:28               ` Arnaldo Carvalho de Melo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191205123302.GA25750@kernel.org \
    --to=arnaldo.melo@gmail.com \
    --cc=alexander.shishkin@linux.intel.com \
    --cc=ivan@cloudflare.com \
    --cc=jolsa@redhat.com \
    --cc=kenton@cloudflare.com \
    --cc=kernel-team@cloudflare.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=namhyung@kernel.org \
    --cc=peterz@infradead.org \
    --cc=sashal@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).