From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9F129308BDA1 for ; Sun, 7 Oct 2018 03:02:27 +0000 (UTC) Received: from mail-vs1-f54.google.com (mail-vs1-f54.google.com [209.85.217.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3E9C63082A2A for ; Sun, 7 Oct 2018 03:02:26 +0000 (UTC) Received: by mail-vs1-f54.google.com with SMTP id w85so9582275vsa.10 for ; Sat, 06 Oct 2018 20:02:26 -0700 (PDT) MIME-Version: 1.0 From: Sherpa Sherpa Date: Sun, 7 Oct 2018 08:46:48 +0545 Message-ID: Content-Type: multipart/alternative; boundary="000000000000ec385e05779abbcd" Subject: [linux-lvm] what is the IOPS behavior when partitions of single disk are used in an LVM? Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: linux-lvm@redhat.com --000000000000ec385e05779abbcd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I have an ubuntu 14.04.1 LTS server which have LVM with logical volume and a volume group named "dbstore-lv" and "dbstore-vg" which have sdb1 sdb2 sdb3 created from same sdb disk. The system as 42 cores and about 128G memory. Although i dont see CPU spikes in htop the load average output from uptime is ~43+ as well as vmstat shows constant iowait of 20-40 where the context switches is constantly around 80,000-150000 and even more at peak hours, the cpu idle time is also hovers around 70-85. Below is output of iostat -xp 1 where the %util is constantly 100% avg-cpu: %user %nice %system %iowait %steal %idle 8.91 0.00 1.31 10.98 0.00 78.80 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 264.00 0.00 58.00 0.00 1428.00 49.24 0.02 0.28 0.00 0.28 0.21 1.20 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 264.00 0.00 58.00 0.00 1428.00 49.24 0.02 0.28 0.00 0.28 0.21 1.20 sdb 0.00 316.00 4.00 86.00 512.00 1608.00 47.11 36.02 0.27 5.00 0.05 11.11 100.00 sdb1 0.00 312.00 4.00 63.00 3512.00 4500.00 60.06 34.02 100.00 5.00 0.00 14.93 100.00 sdb2 0.00 0.00 0.00 821.00 450.00 84.00 8.00 82.00 99.19 0.00 0.19 47.62 100.00 sdb3 0.00 4.00 0.00 2.00 0.00 24.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 6.00 0.00 24.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 4.00 396.00 512.00 1584.00 10.48 36.02 8180.00 5.00 8180.00 2.50 100.00 dm-2 0.00 0.00 0.00 329.00 0.00 3896.00 23.68 0.85 2.58 0.00 2.58 0.05 1.60 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Similarly the TPS/iops is around 600-1000 most of the time(eg. iostat outptu below) avg-cpu: %user %nice %system %iowait %steal %idle 22.24 0.35 2.56 32.08 0.00 42.77 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 527.00 3828.00 1536.00 3828 1536 sdb 576.00 8532.00 2804.00 8532 2804 sdc 42.00 280.00 156.00 280 156 dm-0 0.00 0.00 0.00 0 0 dm-1 956.00 8400.00 2804.00 8400 2804 dm-2 569.00 4108.00 1692.00 4108 1692 dm-3 0.00 0.00 0.00 0 0 Below is excerpt of lsblk which shows lvm associated to disks sdb 8:16 0 19.7T 0 disk =E2=94=9C=E2=94=80sdb1 8:17 0 7.7T 0 part =E2=94=82 =E2=94=94=E2=94=80dbstore-lv (dm-1) 252:1 0 9.4= T 0 lvm /var/db/st01 =E2=94=9C=E2=94=80sdb2 8:18 0 1.7T 0 part =E2=94=82 =E2=94=94=E2=94=80dbstore-lv (dm-1) 252:1 0 9.4= T 0 lvm /var/db/st01 =E2=94=94=E2=94=80sdb3 8:19 0 10.3T 0 part =E2=94=94=E2=94=80archive--archivedbstore--lv (dm-0) 252:0 0 10.3= T 0 lvm /opt/archive/ I am assuming this is due to disk seek problem as the same disk partitions are used for same LVM or may be its due to saturation of the disks(i dont have the vendor provided IOPS data of this disk yet). As initial tuning i have set vm.dirty_ratio to 5 and dirty_background_ratio to 2 + tried deadline scheduler (currently noop) but this doesnt seem to help to reduce the iowait. Any suggestions please ? Is this high IOwait and high load average(40+) possibly due to multiple PVs(made from same sdb disk partition= s) in same volume group +same LV ? Warm Regards Sherpa --000000000000ec385e05779abbcd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

I have an ubuntu= 14.04.1 LTS server which have LVM with logical volume and a volume group n= amed "dbstore-lv" and "dbstore-vg" which have sdb1 sdb2= sdb3 created from same sdb disk. The system as 42 cores and about 128G mem= ory. Although i dont see CPU spikes in=C2=A0htop= =C2=A0the load average output from=C2=A0uptime=C2= =A0is ~43+ as well as vmstat shows constant iowait of 20-40 where the=C2=A0= context switches=C2=A0is constantly around 80,000-= 150000 and even more at peak hours, the cpu idle time is also hovers around= 70-85. Below is output of=C2=A0iostat -xp 1=C2=A0= where the=C2=A0%util=C2=A0is constantly 100%

avg-cpu: %user %nice %system %iowait %steal %idle 8.91 0.00 1.31 10.98 0.00 78.80 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz = avgqu-sz await r_await w_await svctm %util sda 0.00 264.00 0.00 58.00 0.00 1428.00 49.24 = 0.02 0.28 0.00 0.28 0.21 1.20 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 = 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 = 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 264.00 0.00 58.00 0.00 1428.00 49.24 = 0.02 0.28 0.00 0.28 0.21 1.20 sdb 0.00 316.00 4.00 86.00 512.00 1608.00 47.11 = 36.02 0.27 5.00 0.05 11.11 100.00 sdb1 0.00 312.00 4.00 63.00 3512.00 4500.00 60.06= 34.02 100.00 5.00 0.00 14.93 100.00 sdb2 0.00 0.00 0.00 821.00 450.00 84.00 8.= 00 82.00 99.19 0.00 0.19 47.62 100.00 sdb3 0.00 4.00 0.00 2.00 0.00 24.00 24.00 = 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 6.00 0.00 24.00 8.00 = 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 4.00 396.00 512.00 1584.00 10.48 = 36.02 8180.00 5.00 8180.00 2.50 100.00 dm-2 0.00 0.00 0.00 329.00 0.00 3896.00 23.68 = 0.85 2.58 0.00 2.58 0.05 1.60 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 = 0.00 0.00 0.00 0.00 0.00 0.00

Similarly the TPS/iops is around 600-1000 most of the tim= e(eg. iostat outptu below)

avg-cpu:  %user   %nice %=
system %iowait  %steal   %idle
          22.24    0.35    2.56   32.08    0.00   42.77

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             527.00      3828.00      1536.00       3828       1536
sdb             576.00      8532.00      2804.00       8532       2804
sdc              42.00       280.00       156.00        280        156
dm-0              0.00         0.00         0.00          0          0
dm-1            956.00      8400.00      2804.00       8400       2804
dm-2            569.00      4108.00      1692.00       4108       1692
dm-3              0.00         0.00         0.00          0          0

Below is excerpt of=C2=A0lsblk= =C2=A0which shows lvm associated to disks

sdb       =
                         8:16   0  19.7T  0 disk
=E2=94=9C=E2=94=80sdb1                             8:17   0   7.7T  0 part
=E2=94=82 =E2=94=94=E2=94=80dbstore-lv (dm-1)              252:1    0   9.4=
T  0 lvm  /var/db/st01
=E2=94=9C=E2=94=80sdb2                             8:18   0   1.7T  0 part
=E2=94=82 =E2=94=94=E2=94=80dbstore-lv (dm-1)              252:1    0   9.4=
T  0 lvm  /var/db/st01
=E2=94=94=E2=94=80sdb3                             8:19   0  10.3T  0 part
  =E2=94=94=E2=94=80archive--archivedbstore--lv (dm-0)     252:0    0  10.3=
T  0 lvm  /opt/archive/

I am assuming this is due to disk seek problem as the sam= e disk partitions are used for same LVM or may be its due to saturation of = the disks(i dont have the vendor provided IOPS data of this disk yet). As i= nitial tuning i have set vm.dirty_ratio to 5 and dirty_background_ratio to = 2 + tried deadline scheduler (currently noop) but this doesnt seem to help = to reduce the iowait. Any suggestions please ? Is this high IOwait and high= load average(40+) possibly due to multiple PVs(made=C2=A0from same sdb disk=C2=A0partitions) in same vo= lume group=C2=A0+same LV ?=C2=A0

Warm Re= gards
Sherpa
--000000000000ec385e05779abbcd--