lttng-dev.lists.lttng.org archive mirror
 help / color / mirror / Atom feed
From: Jonathan Rajotte-Julien via lttng-dev <lttng-dev@lists.lttng.org>
To: Rocky Dunlap <dunlap@ucar.edu>
Cc: lttng-dev@lists.lttng.org
Subject: Re: Large number of stream files in CTF trace -- too many file handles
Date: Mon, 16 Mar 2020 10:51:25 -0400	[thread overview]
Message-ID: <20200316145125.GC14185@joraj-alpa> (raw)
In-Reply-To: <CADtyBo77-N3SscU7K_HCTwVq7+5f_d9yb19buve66p+SrHxVHQ@mail.gmail.com>

Hi,

> If this is not the right approach, how should I proceed?  E.g., should the
> source-ctf-fs manage a limited pool of file handles?  I would think this
> would be pretty inefficient as you would need to constantly open/close
> files--expensive.

I would probably start looking at the soft and hardlimit for the babeltrace2
process in terms of open file:

On my machine:

joraj@~[]$ ulimit -Sn
1024

joraj@~[]$ ulimit -Hn
1048576

That is a lot of headspace.

I might have a setting somewhere increasing the base hardlimit but
in any case you will see how much room you have.

Based on the number of streams you have, I would say that you will
need more than 2000 as a base soft limit for this trace.

We do have a FD pooling system in place in another project we maintain
(lttng-tools[1] GPL-2.0) that might be pertinent for babeltrace2 at some point
in time. As for the overhead that would occur in a scenario with not enough FD
available, I think it is a good compromise between either reading a trace or
not reading it at all. A warning informing the user that we reached
the limit of the pool might be a good start in such case.

[1] https://github.com/lttng/lttng-tools/tree/master/src/common/fd-tracker

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

  reply	other threads:[~2020-03-16 14:51 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-13 21:55 Rocky Dunlap via lttng-dev
2020-03-16 14:51 ` Jonathan Rajotte-Julien via lttng-dev [this message]
2020-03-17  3:38   ` Rocky Dunlap via lttng-dev

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200316145125.GC14185@joraj-alpa \
    --to=lttng-dev@lists.lttng.org \
    --cc=dunlap@ucar.edu \
    --cc=jonathan.rajotte-julien@efficios.com \
    --subject='Re: Large number of stream files in CTF trace -- too many file handles' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).