linux-lvm.redhat.com archive mirror
 help / color / mirror / Atom feed
From: Daniel Janzon <daniel.janzon@edgeware.tv>
To: "linux-lvm@redhat.com" <linux-lvm@redhat.com>
Subject: Re: [linux-lvm] Best way to run LVM over multiple SW RAIDs?
Date: Mon, 9 Dec 2019 10:26:55 +0000	[thread overview]
Message-ID: <A8A53B0E4B7ACF4BAD5C77D3671AC8976ABE90D8@EX01.edgeware.tv> (raw)


> From: "John Stoffel" <john@stoffel.org>

> Stuart> The mdadm layer already does the striping.  So doing it again
> Stuart> in the LVM layer completely screws it up.  You want plain JBOD
> Stuart> (Just a Bunch Of Disks).

> Umm... not really.  The problem here is more the MD layer not being
> able to run RAID5 across multiple cores at the same time, which is why
> he split things the way he did.

Exactly. The md driver executes on a single core, but with a bunch of RAID5s
I can distribute the load over many cores. That's also why I cannot join the
bunch of RAID5's with a RAID0 (as someone suggested) because then again
all data is pulled through a single core.

> But we don't know the Kernel version, the LVM version, or the OS
> release so as to give better ideas of what to do.

It is Redhat 7, kernel 3.10, scheduler seems to be "[none] mq-deadline kyber"
according to /sys/block/nvme0n1/queue/scheduler. LVM version 2.02.185(2)-RHEL7.

But I wonder if fine-tuning e.g. io scheduler is going to cut it, since I am
looking for something like a 10x improvement.

> The biggest harm to performance here is really the RAID5, and if you
> can instead move to RAID 10 (mirror then stripe across mirrors) then
> you should be a performance boost.

The origin of my problem is indeed the poor performance of RAID5,
which maxes out the single core the driver runs on. But if I accept that
as a given, the next problem is LVM striping. Since I do get 10x better
performance with linear allocation.

> As Daniel says, he's got lots of disk load, but plenty of CPU, so the
> single thread for RAID5 is a big bottleneck.

Yes. That should be fixed since NVMe SSDs now outperform a single
CPU core. But that's a topic for another mailing list I suppose.

> I assume he wants to use LVM so he can create volume(s) larger than
> individual RAID5 volumes, so in that case, I'd probably just build a
> regular non-striped LVM VG holding all your RAID5 disks.  Hopefully
> the Parity disk is spread across all the partitions, though NVMe
> drives should have enough IOPs capacity to mask the RMW cost of RAID5
> to a degree.

The problem is the linear allocation of LVM. It will tend to fill the first
RAID5 first, then the next, etc. The access pattern is such that files
written close in time will be read close in time. We have live video
streams being written and read 24/7. What I want to avoid is that at
some point a majority of all reads end up on a single RAID5 which
will then fail to perform. Bound to happen in an always-on system.

> In any case, I'd just build it like:
>
>  pvcreate /dev/md#     (do for each of 8 RAID5 MD devices)
>  vgcreate datavg /dev/md[#-#]   (give all 8 RAID5 MD devices here.
>  lvcreate -n "name" -L <size> datavg

I think this is basically what I did, what I refer to as a "linearly allocated"
as compared to a striped group. It does indeed perform well most of
the time, but has, I believe, a poor guarantee for the worst case.


> If you can, I'd get more SSDs and move to RAID1+0 (RAID10) instead,
> though you do have the problem where a double disk failure could kill
> your data if it happens to both halves of a mirror.

Yes throwing money on the problem is a good way to solve it. I was 
hoping to avoid that for this application since I thought I just did something
wrong with the stripes.

Kind Regards,
Daniel

             reply	other threads:[~2019-12-09 10:27 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-09 10:26 Daniel Janzon [this message]
2019-12-09 14:26 ` [linux-lvm] Best way to run LVM over multiple SW RAIDs? Marian Csontos
2019-12-10 11:23 ` Gionatan Danti
2019-12-10 21:29   ` John Stoffel
  -- strict thread matches above, loose matches on Subject: below --
2019-12-16  8:22 Daniel Janzon
2019-10-29  8:47 Daniel Janzon
2019-12-07 16:16 ` Anatoly Pugachev
2019-12-07 17:37   ` Roberto Fastec
2019-12-07 20:34     ` Stuart D. Gathman
2019-12-07 22:44       ` John Stoffel
2019-12-07 23:14         ` Stuart D. Gathman
2019-12-08 11:57           ` Gionatan Danti
2019-12-08 22:51           ` John Stoffel
2019-12-09 10:40         ` Guoqing Jiang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=A8A53B0E4B7ACF4BAD5C77D3671AC8976ABE90D8@EX01.edgeware.tv \
    --to=daniel.janzon@edgeware.tv \
    --cc=linux-lvm@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).