High write iops

High write iops

[Игорь Наумов]

Hello, 

using Ubuntu server 18.04.2 btrfs on /

10 absolutely identical servers (Dell M630), with the same configuration, updated 2/10 and encountered this problem on both.
After upgrading kernel from version 4.15.0-36-generic # 39-Ubuntu SMP observing at the same software configuration an increased load on the disk (write) - MySQL previously loaded the disk at 20-50 mB/s, after the update it started loading at 150-250 mB/s.

MySQL configuration and package version were not changed.
If I boot into the old kernel version (4.15.0-36), the problem disappears. Also tested kernels:
4.15.0-52 - problem persists
4.18.0-25 (hwe) - problem persists, load became more 200-250 mB/s
5.2-rc7 (vanilla) - problem persists
Changing /var/lib/mysql partition to ext4 fixes the problem.

With the help of Kai-Heng Feng in bugs.launchpad.net
founded that this commit causes the regression:
commit 31d11b83b96faaee4bb514d375a09489117c3e8d
Author: Filipe Manana < fdmanana@suse.com >
Date: Wed May 9 16:01:46 2018 +0100
Btrfs: fix duplicate extents after fsync of file with prealloc extents

All the details you can find in the bug/1835185 at bugs.launchpad.net
-- 
Igor Naumov

Re: High write iops

[Nikolay Borisov]

On 15.07.19 г. 10:19 ч., Игорь Наумов  wrote:
> 
> Hello, 
> 
> using Ubuntu server 18.04.2 btrfs on /
> 
> 10 absolutely identical servers (Dell M630), with the same configuration, updated 2/10 and encountered this problem on both.
> After upgrading kernel from version 4.15.0-36-generic # 39-Ubuntu SMP observing at the same software configuration an increased load on the disk (write) - MySQL previously loaded the disk at 20-50 mB/s, after the update it started loading at 150-250 mB/s.
> 
> MySQL configuration and package version were not changed.
> If I boot into the old kernel version (4.15.0-36), the problem disappears. Also tested kernels:
> 4.15.0-52 - problem persists
> 4.18.0-25 (hwe) - problem persists, load became more 200-250 mB/s
> 5.2-rc7 (vanilla) - problem persists
> Changing /var/lib/mysql partition to ext4 fixes the problem.
> 
> With the help of Kai-Heng Feng in bugs.launchpad.net
> founded that this commit causes the regression:
> commit 31d11b83b96faaee4bb514d375a09489117c3e8d
> Author: Filipe Manana < fdmanana@suse.com >
> Date: Wed May 9 16:01:46 2018 +0100
> Btrfs: fix duplicate extents after fsync of file with prealloc extents
> 
> All the details you can find in the bug/1835185 at bugs.launchpad.net


Have you set nodatacow on MySQL data filess? Also have you seen actual
increase of latency of your mysql queries because it's possible that
this commit improved what fsync does and allowed your filesystem to
write more data per unit of time? Have you seen drop in IOPS?

>