Linux-Block Archive on lore.kernel.org
 help / color / Atom feed
From: Jens Axboe <axboe@kernel.dk>
To: Jinpu Wang <jinpu.wang@ionos.com>,
	Pavel Begunkov <asml.silence@gmail.com>
Cc: Md Haris Iqbal <haris.iqbal@ionos.com>,
	linux-block <linux-block@vger.kernel.org>,
	Danil Kipnis <danil.kipnis@ionos.com>
Subject: Re: [PATCH V6 0/3] block: add two statistic tables
Date: Wed, 21 Apr 2021 11:02:54 -0600
Message-ID: <f79aca16-bc22-0c27-5a55-f85ac64ccba4@kernel.dk> (raw)
In-Reply-To: <CAMGffEmFsvv-0OkxycG-8CXA1N42VN66QLWFgtC05pH3Dkv-BQ@mail.gmail.com>

On 4/21/21 5:57 AM, Jinpu Wang wrote:
> On Wed, Apr 21, 2021 at 1:55 PM Pavel Begunkov <asml.silence@gmail.com> wrote:
>>
>> On 4/21/21 12:50 PM, Jinpu Wang wrote:
>> [snip]
>>>>> Hi Jens,
>>>>> The problem with using blktrace at production may cause a performance
>>>>> drop ~30%. while with the block stats here, we only see ~3% when
>>>>> enabled.
>>>>
>>>> It's probably was asked before, but let's refresh as the discussion
>>>> erupted again.
>>>>
>>>> I get your problem with blktrace(8), IIRC it definitely can deteriorate
>>>> performance if run constantly, but did you try to write a bpf program
>>>> that does smarter accumulation in the kernel? Like making bpf to collect
>>>> a latency table (right as in your patches do) and flushing it to the
>>>> disk periodically?
>>> Hi Pavel,
>>>
>>> Thanks for the suggestion.
>>>
>>> We did test with ebpf with kprobe in the past (~kernel 4.4/4.14), we
>>> saw 10% performance drop, that's the reason we develop this
>>> stats patches.
>>>
>>> But I just did another test with bpftrace on k 5.10.30, I do not see
>>> performance lost.
>>> It must be ebpf is improving very much since then.
>>>
>>> So to summarize, we can use bpftrace to do the drop in latest kernel,
>>> there is no need to have it build into the kernel.
>>
>> Perfect, and I'm sure it will be even more convenient for you, for
>> instance to gather other stats or do it somehow differently
> 
> Yeah, agree.
> Thanks again!

Perfect, thanks! I'll drop the series.

-- 
Jens Axboe


      reply index

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-09 16:03 Md Haris Iqbal
2021-04-09 16:03 ` [PATCH V6 1/3] block: add io_extra_stats node Md Haris Iqbal
2021-04-09 16:03 ` [PATCH V6 2/3] block: add a statistic table for io latency Md Haris Iqbal
2021-04-09 16:03 ` [PATCH V6 3/3] block: add a statistic table for io sector Md Haris Iqbal
2021-04-09 21:03 ` [PATCH V6 0/3] block: add two statistic tables Jens Axboe
2021-04-12  5:35   ` Jinpu Wang
2021-04-19  6:37     ` Jinpu Wang
2021-04-19 17:57     ` Jens Axboe
2021-04-21  7:49       ` Jinpu Wang
2021-04-21 10:20         ` Pavel Begunkov
2021-04-21 11:50           ` Jinpu Wang
2021-04-21 11:55             ` Pavel Begunkov
2021-04-21 11:57               ` Jinpu Wang
2021-04-21 17:02                 ` Jens Axboe [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f79aca16-bc22-0c27-5a55-f85ac64ccba4@kernel.dk \
    --to=axboe@kernel.dk \
    --cc=asml.silence@gmail.com \
    --cc=danil.kipnis@ionos.com \
    --cc=haris.iqbal@ionos.com \
    --cc=jinpu.wang@ionos.com \
    --cc=linux-block@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Block Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-block/0 linux-block/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-block linux-block/ https://lore.kernel.org/linux-block \
		linux-block@vger.kernel.org
	public-inbox-index linux-block

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-block


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git