FUSE: write operations trigger balance_dirty_pages when using writeback cache

FUSE: write operations trigger balance_dirty_pages when using writeback cache

[刘硕然]

Dear Miklos,

Recently I've been testing FUSE and libfuse example passthrough_ll with writeback cache on, and found out that the performance drops significantly compared to that in local filesystem. As I can see from trace, balance_dirty_pages is triggered very frequently even if there not enough pages that shall be sent to libfuse. I'm not sure if this is a known fact or the FUSE writeback feature requires some specific configurations. Trace log is attached.

dd-19067 [001] .... 195295.568097: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=32 dirty_ratelimit=32 task_ratelimit=0 dirtied=32 dirtied_pause=32 paused=0 pause=33 period=33 think=0 cgroup_ino=1
dd-19067 [001] .... 195295.602029: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=33 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=34 period=34 think=1 cgroup_ino=1
dd-19067 [001] .... 195295.637026: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=34 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=35 period=35 think=1 cgroup_ino=1

BTW, I'm using Linux kernel 4.17.12 and libfuse 3.2.5. Look forward to hearing from you. Thanks in advance.

Regards,
Shuoran Liu

Re: FUSE: write operations trigger balance_dirty_pages when using writeback cache

[Miklos Szeredi]

On Thu, Aug 9, 2018 at 5:37 AM, 刘硕然 <liushuoran@jd.com> wrote:
> Dear Miklos,
>
> Recently I've been testing FUSE and libfuse example passthrough_ll with writeback cache on, and found out that the performance drops significantly compared to that in local filesystem. As I can see from trace, balance_dirty_pages is triggered very frequently even if there not enough pages that shall be sent to libfuse. I'm not sure if this is a known fact or the FUSE writeback feature requires some specific configurations. Trace log is attached.
>
> dd-19067 [001] .... 195295.568097: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=32 dirty_ratelimit=32 task_ratelimit=0 dirtied=32 dirtied_pause=32 paused=0 pause=33 period=33 think=0 cgroup_ino=1
> dd-19067 [001] .... 195295.602029: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=33 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=34 period=34 think=1 cgroup_ino=1
> dd-19067 [001] .... 195295.637026: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=34 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=35 period=35 think=1 cgroup_ino=1
>
> BTW, I'm using Linux kernel 4.17.12 and libfuse 3.2.5. Look forward to hearing from you. Thanks in advance.

Try

   echo 20 > /sys/devices/virtual/bdi/0:`stat -c %d /mnt/fuse/`/max_ratio

where you replace /mnt/fuse with your mountpoint.

Thanks,
Miklos

Re: FUSE: write operations trigger balance_dirty_pages when using writeback cache

[Miklos Szeredi]

On Thu, Aug 9, 2018 at 2:08 PM, 刘硕然 <liushuoran@jd.com> wrote:
> Thanks for the advice. I tried removing BDI_CAP_STRICTLIMIT, and it works. There is no balance_dirty_pages() triggered, and the performance improves a lot.
>
> Tested by libfuse passthrough_ll example and fio:
> ./passthrough_ll -o writeback /mnt/fuse/
> fio --name=test --ioengine=psync --directory=/mnt/fuse/home/test --bs=4k --direct=0 --size=64M --rw=write --fallocate=0 --numjobs=1
>
> performance with BDI_CAP_STRICTLIMIT:
> WRITE: bw=158MiB/s (165MB/s), 158MiB/s-158MiB/s (165MB/s-165MB/s), io=64.0MiB (67.1MB), run=406-406msec
>
> Performance without BDI_CAP_STRICTLIMIT:
> WRITE: bw=1561MiB/s (1637MB/s), 1561MiB/s-1561MiB/s (1637MB/s-1637MB/s), io=64.0MiB (67.1MB), run=41-41msec
>
> However, I wonder if there are some side-effects to remove it? Since it seems that the original purpose of this feature is to prevent FUSE from consuming too much memory.

Yes.  So if BDI_CAP_STRICTLIMIT is causing a serious performance
bottleneck, then we need to think about solving this without losing
the benefits.  Simply removing it is definitely not a proper solution.

Thanks,
Miklos

Re: FUSE: write operations trigger balance_dirty_pages when using writeback cache

[刘硕然]

Thanks for the advice. I tried removing BDI_CAP_STRICTLIMIT, and it works. There is no balance_dirty_pages() triggered, and the performance improves a lot.

Tested by libfuse passthrough_ll example and fio:
./passthrough_ll -o writeback /mnt/fuse/
fio --name=test --ioengine=psync --directory=/mnt/fuse/home/test --bs=4k --direct=0 --size=64M --rw=write --fallocate=0 --numjobs=1

performance with BDI_CAP_STRICTLIMIT:
WRITE: bw=158MiB/s (165MB/s), 158MiB/s-158MiB/s (165MB/s-165MB/s), io=64.0MiB (67.1MB), run=406-406msec

Performance without BDI_CAP_STRICTLIMIT:
WRITE: bw=1561MiB/s (1637MB/s), 1561MiB/s-1561MiB/s (1637MB/s-1637MB/s), io=64.0MiB (67.1MB), run=41-41msec

However, I wonder if there are some side-effects to remove it? Since it seems that the original purpose of this feature is to prevent FUSE from consuming too much memory. Please correct me if I am mistaken. Thanks in advance.


Regards,
Shuoran


-----邮件原件-----
发件人: Miklos Szeredi [mailto:miklos@szeredi.hu] 
发送时间: 2018年8月9日 16:30
收件人: 刘硕然 <liushuoran@jd.com>
抄送: linux-fsdevel@vger.kernel.org; linux-kernel@vger.kernel.org
主题: Re: FUSE: write operations trigger balance_dirty_pages when using writeback cache

On Thu, Aug 9, 2018 at 9:31 AM, 刘硕然 <liushuoran@jd.com> wrote:
> Thank you for the prompt reply.
>
> I tried this config, but still can get balance_dirty_pages triggered.

I think it may be due to BDI_CAP_STRICTLIMIT used by fuse.  If you remove that setting from fuse in the kernel you should not be getting the balance_dirty_pages() as often.

Not sure if that's the realproblem, though, that depends on how much time is spent in balance_dirty_pages().  You can try profiling the kernel to find that out.

My guess is that the real cause of the slowdown is some other place.
There's for example a known issue with selinux related getxattr thrashing.  Disabling getxattr on your filesystem may significantly improve performance.

Thanks,
Miklos

Re: FUSE: write operations trigger balance_dirty_pages when using writeback cache

[Miklos Szeredi]

On Thu, Aug 9, 2018 at 9:31 AM, 刘硕然 <liushuoran@jd.com> wrote:
> Thank you for the prompt reply.
>
> I tried this config, but still can get balance_dirty_pages triggered.

I think it may be due to BDI_CAP_STRICTLIMIT used by fuse.  If you
remove that setting from fuse in the kernel you should not be getting
the balance_dirty_pages() as often.

Not sure if that's the realproblem, though, that depends on how much
time is spent in balance_dirty_pages().  You can try profiling the
kernel to find that out.

My guess is that the real cause of the slowdown is some other place.
There's for example a known issue with selinux related getxattr
thrashing.  Disabling getxattr on your filesystem may significantly
improve performance.

Thanks,
Miklos

Re: FUSE: write operations trigger balance_dirty_pages when using writeback cache

[刘硕然]

Thank you for the prompt reply.

I tried this config, but still can get balance_dirty_pages triggered.

[root@A01-R20-I31-77-8S5FKM2 example]# stat -c %d /mnt/fuse/
42
[root@A01-R20-I31-77-8S5FKM2 example]# echo 20 > /sys/devices/virtual/bdi/0:`stat -c %d /mnt/fuse/`/max_ratio
[root@A01-R20-I31-77-8S5FKM2 example]# cat /sys/devices/virtual/bdi/0\:42/max_ratio 
20
[root@A01-R20-I31-77-8S5FKM2 example]# dd if=/dev/zero of=/mnt/fuse/tmp/test001 bs=4k count=20000

...
dd-633   [001] .... 273499.083059: balance_dirty_pages: bdi 0:42: limit=3180556 setpoint=2782501 dirty=310 bdi_setpoint=0 bdi_dirty=13 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=14 period=14 think=1 cgroup_ino=1
dd-633   [001] .... 273499.098089: balance_dirty_pages: bdi 0:42: limit=3180556 setpoint=2782501 dirty=311 bdi_setpoint=0 bdi_dirty=14 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=15 period=15 think=1 cgroup_ino=1
dd-633   [001] .... 273499.114082: balance_dirty_pages: bdi 0:42: limit=3180556 setpoint=2782501 dirty=312 bdi_setpoint=0 bdi_dirty=15 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=16 period=16 think=1 cgroup_ino=1
...


Regards,
Shuoran

-----邮件原件-----
发件人: Miklos Szeredi [mailto:miklos@szeredi.hu] 
发送时间: 2018年8月9日 15:14
收件人: 刘硕然 <liushuoran@jd.com>
抄送: linux-fsdevel@vger.kernel.org; linux-kernel@vger.kernel.org
主题: Re: FUSE: write operations trigger balance_dirty_pages when using writeback cache

On Thu, Aug 9, 2018 at 5:37 AM, 刘硕然 <liushuoran@jd.com> wrote:
> Dear Miklos,
>
> Recently I've been testing FUSE and libfuse example passthrough_ll with writeback cache on, and found out that the performance drops significantly compared to that in local filesystem. As I can see from trace, balance_dirty_pages is triggered very frequently even if there not enough pages that shall be sent to libfuse. I'm not sure if this is a known fact or the FUSE writeback feature requires some specific configurations. Trace log is attached.
>
> dd-19067 [001] .... 195295.568097: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=32 dirty_ratelimit=32 task_ratelimit=0 dirtied=32 dirtied_pause=32 paused=0 pause=33 period=33 think=0 cgroup_ino=1
> dd-19067 [001] .... 195295.602029: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=33 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=34 period=34 think=1 cgroup_ino=1
> dd-19067 [001] .... 195295.637026: balance_dirty_pages: bdi 0:42: limit=3180390 setpoint=2782421 dirty=5 bdi_setpoint=0 bdi_dirty=34 dirty_ratelimit=32 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=35 period=35 think=1 cgroup_ino=1
>
> BTW, I'm using Linux kernel 4.17.12 and libfuse 3.2.5. Look forward to hearing from you. Thanks in advance.

Try

   echo 20 > /sys/devices/virtual/bdi/0:`stat -c %d /mnt/fuse/`/max_ratio

where you replace /mnt/fuse with your mountpoint.

Thanks,
Miklos