From: Jens Axboe <axboe@kernel.dk>
To: Jinpu Wang <jinpu.wang@ionos.com>,
Pavel Begunkov <asml.silence@gmail.com>
Cc: Md Haris Iqbal <haris.iqbal@ionos.com>,
linux-block <linux-block@vger.kernel.org>,
Danil Kipnis <danil.kipnis@ionos.com>
Subject: Re: [PATCH V6 0/3] block: add two statistic tables
Date: Wed, 21 Apr 2021 11:02:54 -0600 [thread overview]
Message-ID: <f79aca16-bc22-0c27-5a55-f85ac64ccba4@kernel.dk> (raw)
In-Reply-To: <CAMGffEmFsvv-0OkxycG-8CXA1N42VN66QLWFgtC05pH3Dkv-BQ@mail.gmail.com>
On 4/21/21 5:57 AM, Jinpu Wang wrote:
> On Wed, Apr 21, 2021 at 1:55 PM Pavel Begunkov <asml.silence@gmail.com> wrote:
>>
>> On 4/21/21 12:50 PM, Jinpu Wang wrote:
>> [snip]
>>>>> Hi Jens,
>>>>> The problem with using blktrace at production may cause a performance
>>>>> drop ~30%. while with the block stats here, we only see ~3% when
>>>>> enabled.
>>>>
>>>> It's probably was asked before, but let's refresh as the discussion
>>>> erupted again.
>>>>
>>>> I get your problem with blktrace(8), IIRC it definitely can deteriorate
>>>> performance if run constantly, but did you try to write a bpf program
>>>> that does smarter accumulation in the kernel? Like making bpf to collect
>>>> a latency table (right as in your patches do) and flushing it to the
>>>> disk periodically?
>>> Hi Pavel,
>>>
>>> Thanks for the suggestion.
>>>
>>> We did test with ebpf with kprobe in the past (~kernel 4.4/4.14), we
>>> saw 10% performance drop, that's the reason we develop this
>>> stats patches.
>>>
>>> But I just did another test with bpftrace on k 5.10.30, I do not see
>>> performance lost.
>>> It must be ebpf is improving very much since then.
>>>
>>> So to summarize, we can use bpftrace to do the drop in latest kernel,
>>> there is no need to have it build into the kernel.
>>
>> Perfect, and I'm sure it will be even more convenient for you, for
>> instance to gather other stats or do it somehow differently
>
> Yeah, agree.
> Thanks again!
Perfect, thanks! I'll drop the series.
--
Jens Axboe
prev parent reply other threads:[~2021-04-21 17:02 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-09 16:03 [PATCH V6 0/3] block: add two statistic tables Md Haris Iqbal
2021-04-09 16:03 ` [PATCH V6 1/3] block: add io_extra_stats node Md Haris Iqbal
2021-04-09 16:03 ` [PATCH V6 2/3] block: add a statistic table for io latency Md Haris Iqbal
2021-04-09 16:03 ` [PATCH V6 3/3] block: add a statistic table for io sector Md Haris Iqbal
2021-04-09 21:03 ` [PATCH V6 0/3] block: add two statistic tables Jens Axboe
2021-04-12 5:35 ` Jinpu Wang
2021-04-19 6:37 ` Jinpu Wang
2021-04-19 17:57 ` Jens Axboe
2021-04-21 7:49 ` Jinpu Wang
2021-04-21 10:20 ` Pavel Begunkov
2021-04-21 11:50 ` Jinpu Wang
2021-04-21 11:55 ` Pavel Begunkov
2021-04-21 11:57 ` Jinpu Wang
2021-04-21 17:02 ` Jens Axboe [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f79aca16-bc22-0c27-5a55-f85ac64ccba4@kernel.dk \
--to=axboe@kernel.dk \
--cc=asml.silence@gmail.com \
--cc=danil.kipnis@ionos.com \
--cc=haris.iqbal@ionos.com \
--cc=jinpu.wang@ionos.com \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).