From: Rocky Dunlap via lttng-dev <firstname.lastname@example.org> To: Jonathan Rajotte-Julien <email@example.com> Cc: firstname.lastname@example.org Subject: Re: Large number of stream files in CTF trace -- too many file handles Date: Mon, 16 Mar 2020 21:38:11 -0600 [thread overview] Message-ID: <CADtyBo7J-E0C_Ap-7tztdd-PpHZwDUXBVqkT_mp_O3x9h=Qy5g@mail.gmail.com> (raw) In-Reply-To: <20200316145125.GC14185@joraj-alpa> [-- Attachment #1.1: Type: text/plain, Size: 1576 bytes --] Jonathan, Increasing the soft FD limit worked great, both for the command line and the Python script. Thanks for the help! Rocky On Mon, Mar 16, 2020 at 8:51 AM Jonathan Rajotte-Julien < email@example.com> wrote: > Hi, > > > If this is not the right approach, how should I proceed? E.g., should > the > > source-ctf-fs manage a limited pool of file handles? I would think this > > would be pretty inefficient as you would need to constantly open/close > > files--expensive. > > I would probably start looking at the soft and hardlimit for the > babeltrace2 > process in terms of open file: > > On my machine: > > joraj@~$ ulimit -Sn > 1024 > > joraj@~$ ulimit -Hn > 1048576 > > That is a lot of headspace. > > I might have a setting somewhere increasing the base hardlimit but > in any case you will see how much room you have. > > Based on the number of streams you have, I would say that you will > need more than 2000 as a base soft limit for this trace. > > We do have a FD pooling system in place in another project we maintain > (lttng-tools GPL-2.0) that might be pertinent for babeltrace2 at some > point > in time. As for the overhead that would occur in a scenario with not > enough FD > available, I think it is a good compromise between either reading a trace > or > not reading it at all. A warning informing the user that we reached > the limit of the pool might be a good start in such case. > >  https://github.com/lttng/lttng-tools/tree/master/src/common/fd-tracker > > Cheers > > -- > Jonathan Rajotte-Julien > EfficiOS > [-- Attachment #1.2: Type: text/html, Size: 2174 bytes --] [-- Attachment #2: Type: text/plain, Size: 156 bytes --] _______________________________________________ lttng-dev mailing list firstname.lastname@example.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
prev parent reply other threads:[~2020-03-17 3:38 UTC|newest] Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-03-13 21:55 Rocky Dunlap via lttng-dev 2020-03-16 14:51 ` Jonathan Rajotte-Julien via lttng-dev 2020-03-17 3:38 ` Rocky Dunlap via lttng-dev [this message]
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='CADtyBo7J-E0C_Ap-7tztdd-PpHZwDUXBVqkT_mp_O3x9h=Qy5g@mail.gmail.com' \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: Large number of stream files in CTF trace -- too many file handles' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).