linux-lvm.redhat.com archive mirror
 help / color / mirror / Atom feed
From: Peter Rajnoha <prajnoha@redhat.com>
To: Martin Wilck <martin.wilck@suse.com>
Cc: "bmarzins@redhat.com" <bmarzins@redhat.com>,
	"zkabelac@redhat.com" <zkabelac@redhat.com>,
	"teigland@redhat.com" <teigland@redhat.com>,
	"linux-lvm@redhat.com" <linux-lvm@redhat.com>
Subject: Re: [linux-lvm] Discussion: performance issue on event activation mode
Date: Fri, 1 Oct 2021 10:08:25 +0200	[thread overview]
Message-ID: <20211001080825.s6scrvujgbufhaiu@alatyr-rpi.brq.redhat.com> (raw)
In-Reply-To: <074e7d61fac100ef3bdb49edb2a29b4b6a666ef3.camel@suse.com>

On Fri 01 Oct 2021 07:41, Martin Wilck wrote:
> On Thu, 2021-09-30 at 23:32 +0800, heming.zhao@suse.com wrote:
> > > I just want to say that some of the issues might simply be
> > > regressions/issues with systemd/udev that could be fixed. We as
> > > providers of block device abstractions where we need to handle,
> > > sometimes, thousands of devices, might be the first ones to hit these
> > > issues.
> > > 
> > 
> > The rhel8 callgrind picture
> > (https://prajnoha.fedorapeople.org/bz1986158/rhel8_libudev_critical_cost.png
> > )
> > responds to my analysis:
> > https://listman.redhat.com/archives/linux-lvm/2021-June/msg00022.html
> > handle_db_line took too much time and become the hotspot.
> 
> I missed that post. You wrote
> 
> > the dev_cache_scan doesn't have direct disk IOs, but libudev will
> scan/read
> > udev db which issue real disk IOs (location is /run/udev/data).
> > ...
> > 2. scans/reads udev db (/run/udev/data). may O(n)
> >  udev will call device_read_db => handle_db_line to handle every
> >    line of a db file.
> > ...
> > I didn't test the related udev code, and guess the <2> takes too much
> time.
> 
> ... but note that /run/udev is on tmpfs, not on a real disk. So  the
> accesses should be very fast unless there's some locking happening.

Yes, indeed! I think this is a regression.

The results/graphs show that lots of time is spent on some internal
hashmap handling. I don't see this in older versions of udev like the
one bundled with systemd v219 (I compared RHEL7 and 8, haven't done
detailed bisection yet). My suspicion is that some of the code in udev
got more shared with native systemd code, like that hash usage, so this
might be the clue, but someone from systemd/udev should look more closer
into this.

-- 
Peter

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


  reply	other threads:[~2021-10-01  8:08 UTC|newest]

Thread overview: 86+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-06  6:15 [linux-lvm] Discussion: performance issue on event activation mode heming.zhao
2021-06-06 16:35 ` Roger Heflin
2021-06-07 10:27   ` Martin Wilck
2021-06-07 15:30     ` heming.zhao
2021-06-07 15:45       ` Martin Wilck
2021-06-07 20:52       ` Roger Heflin
2021-06-07 21:30     ` David Teigland
2021-06-08  8:26       ` Martin Wilck
2021-06-08 15:39         ` David Teigland
2021-06-08 15:47           ` Martin Wilck
2021-06-08 16:02             ` Zdenek Kabelac
2021-06-08 16:05               ` Martin Wilck
2021-06-08 16:03             ` David Teigland
2021-06-08 16:07               ` Martin Wilck
2021-06-15 17:03           ` David Teigland
2021-06-15 18:21             ` Zdenek Kabelac
2021-06-16 16:18             ` heming.zhao
2021-06-16 16:38               ` David Teigland
2021-06-17  3:46                 ` heming.zhao
2021-06-17 15:27                   ` David Teigland
2021-06-08 16:49         ` heming.zhao
2021-06-08 16:18       ` heming.zhao
2021-06-09  4:01         ` heming.zhao
2021-06-09  5:37           ` Heming Zhao
2021-06-09 18:59             ` David Teigland
2021-06-10 17:23               ` heming.zhao
2021-06-07 15:48 ` Martin Wilck
2021-06-07 16:31   ` Zdenek Kabelac
2021-06-07 21:48   ` David Teigland
2021-06-08 12:29     ` Peter Rajnoha
2021-06-08 13:23       ` Martin Wilck
2021-06-08 13:41         ` Peter Rajnoha
2021-06-08 13:46           ` Zdenek Kabelac
2021-06-08 13:56             ` Peter Rajnoha
2021-06-08 14:23               ` Zdenek Kabelac
2021-06-08 14:48               ` Martin Wilck
2021-06-08 15:19                 ` Peter Rajnoha
2021-06-08 15:39                   ` Martin Wilck
2021-09-09 19:44         ` David Teigland
2021-09-10 17:38           ` Martin Wilck
2021-09-12 16:51             ` heming.zhao
2021-09-27 10:00           ` Peter Rajnoha
2021-09-27 15:38             ` David Teigland
2021-09-28  6:34               ` Martin Wilck
2021-09-28 14:42                 ` David Teigland
2021-09-28 15:16                   ` Martin Wilck
2021-09-28 15:31                     ` Martin Wilck
2021-09-28 15:56                     ` David Teigland
2021-09-28 18:03                       ` Benjamin Marzinski
2021-09-28 17:42                     ` Benjamin Marzinski
2021-09-28 19:15                       ` Martin Wilck
2021-09-29 22:06                       ` Peter Rajnoha
2021-09-30  7:51                         ` Martin Wilck
2021-09-30  8:07                           ` heming.zhao
2021-09-30  9:31                             ` Martin Wilck
2021-09-30 11:41                             ` Peter Rajnoha
2021-09-30 15:32                               ` heming.zhao
2021-10-01  7:41                                 ` Martin Wilck
2021-10-01  8:08                                   ` Peter Rajnoha [this message]
2021-09-30 11:29                           ` Peter Rajnoha
2021-09-30 16:04                             ` David Teigland
2021-09-30 14:41                           ` Benjamin Marzinski
2021-10-01  7:42                             ` Martin Wilck
2021-09-29 21:53                 ` Peter Rajnoha
2021-09-30  7:45                   ` Martin Wilck
2021-09-29 21:39               ` Peter Rajnoha
2021-09-30  7:22                 ` Martin Wilck
2021-09-30 14:26                   ` David Teigland
2021-09-30 15:55                 ` David Teigland
2021-10-01  8:00                   ` Peter Rajnoha
2021-10-18  6:24                   ` Martin Wilck
2021-10-18 15:04                     ` David Teigland
2021-10-18 16:56                       ` heming.zhao
2021-10-18 21:51                       ` Zdenek Kabelac
2021-10-19 17:18                         ` David Teigland
2021-10-20 14:40                       ` Martin Wilck
2021-10-20 14:50                         ` David Teigland
2021-10-20 14:54                           ` Martin Wilck
2021-10-20 15:12                             ` David Teigland
2021-06-07 16:40 ` David Teigland
2021-07-02 21:09 ` David Teigland
2021-07-02 21:22   ` Martin Wilck
2021-07-02 22:02     ` David Teigland
2021-07-03 11:49       ` heming.zhao
2021-07-08 10:10         ` Tom Yan
2021-07-02 21:31   ` Tom Yan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20211001080825.s6scrvujgbufhaiu@alatyr-rpi.brq.redhat.com \
    --to=prajnoha@redhat.com \
    --cc=bmarzins@redhat.com \
    --cc=linux-lvm@redhat.com \
    --cc=martin.wilck@suse.com \
    --cc=teigland@redhat.com \
    --cc=zkabelac@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).