Re: xfs fstrim and quota

Re: xfs fstrim and quota

[michael.arndt]

Hello *

Will fstrim operations for a thin provisioning storage and xfs quota 
conflict with each other ?

If i understand fstrim code correctly, in case of xfs / thin 
provisioning  storage it tells xfs to release unused blocks.

I have read indications that blocks are marked to the underlying storage 
as freed by zeroing them out.

Is the "write zeros" correct information, or will be commands like scsi 
unmap or FITRIM be sent to the storage ?

I found many exact references to fstrim on SSD, but no technical 
description on which operation is implemented for thin provisioning 
storages.

TIA
Micha

Re: xfs fstrim and quota

[Brian Foster]

On Tue, Apr 24, 2018 at 10:28:34AM +0200, michael.arndt@berlin.de wrote:
> Hello *
> 
> Will fstrim operations for a thin provisioning storage and xfs quota
> conflict with each other ?
> 

How so? fstrim should basically inform the underlying device of blocks
in the filesystem that are free. Free blocks in the fs aren't accounted
to any quota by definition, so there shouldn't be a conflict.

> If i understand fstrim code correctly, in case of xfs / thin provisioning
> storage it tells xfs to release unused blocks.
> 

Right... so an underlying thin block device can release blocks that are
unused in the fs.

> I have read indications that blocks are marked to the underlying storage as
> freed by zeroing them out.
> 
> Is the "write zeros" correct information, or will be commands like scsi
> unmap or FITRIM be sent to the storage ?
> 

The fs invokes the block layer discard mechanism. I can't really speak
to how this translates into device commands. My understanding is that a
dm-thin device would/could act on this to release associated blocks to
the pool, otherwise the behavior may depend on the physical
characteristics of the underlying device (i.e., SSD, non-dm thin
devices, etc.), supported commands or whatnot. Perhaps somebody else can
chime in on that or otherwise this might be a better question for the
block layer folks.

Brian

> I found many exact references to fstrim on SSD, but no technical description
> on which operation is implemented for thin provisioning storages.
> 
> TIA
> Micha
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: xfs fstrim and quota

[Eric Sandeen]

On 4/24/18 2:28 AM, michael.arndt@berlin.de wrote:
> Hello *
> 
> Will fstrim operations for a thin provisioning storage and xfs quota conflict with each other ?
> 
> If i understand fstrim code correctly, in case of xfs / thin provisioning  storage it tells xfs to release unused blocks.
> 
> I have read indications that blocks are marked to the underlying storage as freed by zeroing them out.

Brian addressed most of this, I think, but the short answer is that no, fstrim
will in no way affect xfs quota.  Everything happening as a result of fstrim
happens well below what the quota subsystem even knows about.  Even if discard
happens via WRITE_SAME, that's not anything quota is aware of.

Short answer is: There is no conflict.

Thanks,
-Eric
 
> Is the "write zeros" correct information, or will be commands like scsi unmap or FITRIM be sent to the storage ?
> 
> I found many exact references to fstrim on SSD, but no technical description on which operation is implemented for thin provisioning storages.
> 
> TIA
> Micha

xfs remove / unlink extremely slow ?

[Michael Arndt]

Hello *,

Problem: /bin/rm extremely slow on a major xfs (SSD based) HPC Storage
               slow == 90 seconds for unlink of an empty file without any extents
               strace says: time completely used for unlink call

Question; Is there any issue resolution ?
Information re XFS Version and OS at end of this Post

Example of issue:

[root@atgrzsl3150 DOM_0]# xfs_bmap -a .AN_720.0000122.fl3step_0.lock 
.AN_720.0000122.fl3step_0.lock: no extents

[root@atgrzsl3150 DOM_0]# ls -laFtr .AN_720.0000122.fl3step_0.lock 
-rw-rw-r-- 1 user group 0 Oct 22 14:14 .AN_720.0000122.fl3step_0.lock

strace -T -tt /bin/rm .AN_720.0000122.fl3step_0.lock 
11:41:11.621005 execve("/bin/rm", ["/bin/rm", ".AN_720.0000122.fl3step_0.lock"], [/* 31 vars */]) = 0 <0.000169>
11:41:11.621312 brk(NULL)               = 0x6f5000 <0.000023>
11:41:11.621378 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4d96017000 <0.000058>
…….
11:41:11.622485 newfstatat(AT_FDCWD, ".AN_720.0000122.fl3step_0.lock", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 <0.000009>
11:41:11.622522 geteuid()               = 0 <0.000009>
-> 11:41:11.622546 unlinkat(AT_FDCWD, ".AN_720.0000122.fl3step_0.lock", 0) = 0 <89.612833>
-> 11:42:41.235428 lseek(0, 0, SEEK_CUR)   = -1 ESPIPE (Illegal seek) <0.000065>
11:42:41.235548 close(0)                = 0 <0.000052>
11:42:41.235689 close(1)                = 0 <0.000011>
11:42:41.235738 close(2)                = 0 <0.000055>
11:42:41.235830 exit_group(0)           = ?
11:42:41.235941 +++ exited with 0 +++

xfs_info /dev/mapper/vg_calc2-calc2
meta-data=/dev/mapper/vg_calc2-calc2 isize=512    agcount=50, agsize=268435448 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=13421711360, imaxpct=20
         =                       sunit=8      swidth=40 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


Issue on:

xfsprogs-4.5.0-18.el7.x86_64
xfsdump-3.1.7-1.el7.x86_64
Red Hat Enterprise Linux Server release 7.4 (Maipo)
 df -kh .
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vg_calc2        50T   20T   31T  40% /calc2

Layers: 

SSD based commercial Storage exports many small LUN’s -> LUN#s striped via LVM2 for speed, xfs with default opts on top of LVM
Currently no discard Option for mount and no fstrim manually called

Mount Options used
/dev/mapper/vg_calc2-calc2 /calc2 xfs noatime,delaylog,nobarrier,nodiratime,logbsize=256k,logbufs=8 0 0 


thanks for any tip / hint / question
Micha

Re: xfs remove / unlink extremely slow ?

[Brian Foster]

On Thu, Nov 15, 2018 at 06:07:46PM +0100, Michael Arndt wrote:
> Brian,
> 
> Info, supporting your remark about VFS layer: 
> 
> I did a reboot of cluster headnode and after that the remove was „fast“ meanig the typical millsecond /single file  to seconds /  removal of whole trees.
> 
> What does your intuition say, what could be suspected in the VFS layer / lvm2 layer or below to trigger this problem 
> ( just to collect ideas, where to search). ?
> 

If the problem only occurs after a period of time like this, I suppose
that could suggest some kind of cache effect is contributing to it. You
could see if an 'echo 2 > /proc/sys/vm/drop_caches' clears the problem
once it occurs, for example, but note that will clear all cached
dentries/inodes on the system.

I'm not familiar enough with the VFS code and associated caching to
speculate much beyond that. Other things that might be interesting are
whether other operations on said file take the same amount of time
(i.e., an mv before an rm?), whether this is a common filename pattern
that sees a lot of create/delete operations (perhaps polluting a lookup
cache with unlinked entries? i.e., does changing the filename have any
effect?), whether a relocation of the file to another directory changes
anything, whether other operations in the same dir on unrelated files
show the same problem, etc.

Your best bet is probably to collect as much trace data as you can and
then possibly report this to the general linux-fsdevel mailing list.
Note that if you're using a distro kernel, it might be more appropriate
to report an issue with your distro vendor than an upstream list.

Brian

> Caused by the fact, that the problem was seen also two times before, the issue will reappear after some time.
> 
> Off topic: is the mount option delaylog still functional or obsoleted, because already default behaviour ?
> 
> best
> Micha
> thx for extremely friendly help
> 

Re: xfs remove / unlink extremely slow ?

[Michael Arndt]

Brian,

Info, supporting your remark about VFS layer: 

I did a reboot of cluster headnode and after that the remove was „fast“ meanig the typical millsecond /single file  to seconds /  removal of whole trees.

What does your intuition say, what could be suspected in the VFS layer / lvm2 layer or below to trigger this problem 
( just to collect ideas, where to search). ?

Caused by the fact, that the problem was seen also two times before, the issue will reappear after some time.

Off topic: is the mount option delaylog still functional or obsoleted, because already default behaviour ?

best
Micha
thx for extremely friendly help

Re: xfs remove / unlink extremely slow ?

[Michael Arndt]

Brian,

Info, supporting your remark about VFS layer: 

I did a reboot of cluster headnode and after that the remove was „fast“ meanig the typical millsecond /single file  to seconds /  removal of whole trees.

What does your intuition say, what could be suspected in the VFS layer / lvm2 layer or below to trigger this problem 
( just to collect ideas, where to search). ?

Caused by the fact, that the problem was seen also two times before, the issue will reappear after some time.

Off topic: is the mount option delaylog still functional or obsoleted, because already default behaviour ?

best
Micha
thx for extremely friendly help