linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* xfstest failure with xfs, dax and v4.4-rc3
@ 2015-12-01 22:43 Ross Zwisler
  2015-12-01 23:51 ` Dave Chinner
  0 siblings, 1 reply; 5+ messages in thread
From: Ross Zwisler @ 2015-12-01 22:43 UTC (permalink / raw)
  To: Dave Chinner, xfs, Brian Foster, linux-fsdevel; +Cc: linux-kernel

I'm seeing a new intermittent xfstests failure with v4.4-rc3 using XFS and
DAX.  The test that is failing is generic/083, and it fails about 50% of the
time in my setup.  Here's the output:

FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 lorwyn 4.4.0-rc3
MKFS_OPTIONS  -- -f -bsize=4096 /dev/pmem0p2
MOUNT_OPTIONS -- -o dax -o context=system_u:object_r:nfs_t:s0 /dev/pmem0p2
/mnt/xfstests_scratch

generic/083 2s ... 2s
_check_xfs_filesystem: filesystem on /dev/pmem0p2 is inconsistent (r) (see
/root/xfstests/results//generic/083.full)
Ran: generic/083
Failures: generic/083
Failed 1 of 1 tests

The 083.full file is pretty short, and can be found at the end of this email.

My test setup is a pair of 4GiB PMEM partitions in a KVM virtual machine.

This test passes 100% of the time with v4.3 using DAX, and it passes 100% of
the time in my setup with DAX disabled using v4.4-rc3.

Please let me know if you have any other questions about how to reproduce the
failure.  I'm also happy to test patches.

Thanks,
- Ross

---
*** mkfs -dsize=104857600,agcount=6

meta-data=/dev/pmem0p2           isize=512    agcount=6, agsize=4267 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0
data     =                       bsize=4096   blocks=25600, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=1605, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
seed = 1448253989
_check_xfs_filesystem: filesystem on /dev/pmem0p2 is inconsistent
*** xfs_repair -n output ***
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
agi unlinked bucket 4 is 19780 in ag 3 (inode=216388)
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected dir inode 216388, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 216388 nlinks from 0 to 2
No modify flag set, skipping filesystem flush and exiting.
*** end xfs_repair output
*** mount output ***
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3752992k,nr_inodes=938248,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/fedora-root on / type ext4 (rw,relatime,seclabel,data=ordered)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
tmpfs on /tmp type tmpfs (rw,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/vda1 on /boot type ext4 (rw,relatime,seclabel,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.122.1:/home/rzwisler/project on /root/project type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.101,local_lock=none,addr=192.168.122.1)
tmpfs on /run/user/42 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=814668k,mode=700,uid=42,gid=42)
gvfsd-fuse on /run/user/42/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=42,group_id=42)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=814668k,mode=700)
/dev/pmem0p1 on /mnt/xfstests_test type xfs (rw,relatime,seclabel,attr2,dax,inode64,noquota)
*** end mount output

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: xfstest failure with xfs, dax and v4.4-rc3
  2015-12-01 22:43 xfstest failure with xfs, dax and v4.4-rc3 Ross Zwisler
@ 2015-12-01 23:51 ` Dave Chinner
  2015-12-02 15:12   ` Jeff Moyer
  0 siblings, 1 reply; 5+ messages in thread
From: Dave Chinner @ 2015-12-01 23:51 UTC (permalink / raw)
  To: Ross Zwisler, xfs, Brian Foster, linux-fsdevel, linux-kernel

On Tue, Dec 01, 2015 at 03:43:46PM -0700, Ross Zwisler wrote:
> I'm seeing a new intermittent xfstests failure with v4.4-rc3 using XFS and
> DAX.  The test that is failing is generic/083, and it fails about 50% of the
> time in my setup.  Here's the output:
> 
> FSTYP         -- xfs (debug)
> PLATFORM      -- Linux/x86_64 lorwyn 4.4.0-rc3
> MKFS_OPTIONS  -- -f -bsize=4096 /dev/pmem0p2
> MOUNT_OPTIONS -- -o dax -o context=system_u:object_r:nfs_t:s0 /dev/pmem0p2
> /mnt/xfstests_scratch
> 
> generic/083 2s ... 2s
> _check_xfs_filesystem: filesystem on /dev/pmem0p2 is inconsistent (r) (see
> /root/xfstests/results//generic/083.full)
> Ran: generic/083
> Failures: generic/083
> Failed 1 of 1 tests

$ grep 083  tests/xfs/group
083 dangerous_fuzzers
$

Yup, it's expected to fail, and given that it's "dangerous" likely
to hang the filesystem or crash the kernel.

Tests that are expected to pass and act as regression tests are in
the auto group. If a test is not in the auto group, it's usually
because the test is either broken or will not complete due
hanging/crashing. i.e use:

# MOUNT_OPTIONS="-o dax" ./check -g auto

For regression testing.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: xfstest failure with xfs, dax and v4.4-rc3
  2015-12-01 23:51 ` Dave Chinner
@ 2015-12-02 15:12   ` Jeff Moyer
  2015-12-03  6:51     ` Dave Chinner
  0 siblings, 1 reply; 5+ messages in thread
From: Jeff Moyer @ 2015-12-02 15:12 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Ross Zwisler, xfs, Brian Foster, linux-fsdevel, linux-kernel

Dave Chinner <david@fromorbit.com> writes:

> On Tue, Dec 01, 2015 at 03:43:46PM -0700, Ross Zwisler wrote:
>> I'm seeing a new intermittent xfstests failure with v4.4-rc3 using XFS and
>> DAX.  The test that is failing is generic/083, and it fails about 50% of the
>> time in my setup.  Here's the output:
>> 
>> FSTYP         -- xfs (debug)
>> PLATFORM      -- Linux/x86_64 lorwyn 4.4.0-rc3
>> MKFS_OPTIONS  -- -f -bsize=4096 /dev/pmem0p2
>> MOUNT_OPTIONS -- -o dax -o context=system_u:object_r:nfs_t:s0 /dev/pmem0p2
>> /mnt/xfstests_scratch
>> 
>> generic/083 2s ... 2s
   ^^^^^^^
>> _check_xfs_filesystem: filesystem on /dev/pmem0p2 is inconsistent (r) (see
>> /root/xfstests/results//generic/083.full)
>> Ran: generic/083
>> Failures: generic/083
>> Failed 1 of 1 tests
>
> $ grep 083  tests/xfs/group
> 083 dangerous_fuzzers
> $

That's great, but Ross reported an issues with *generic*/083.  ;-)

/me hands Dave more coffee.

-Jeff

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: xfstest failure with xfs, dax and v4.4-rc3
  2015-12-02 15:12   ` Jeff Moyer
@ 2015-12-03  6:51     ` Dave Chinner
  2015-12-03 16:43       ` Ross Zwisler
  0 siblings, 1 reply; 5+ messages in thread
From: Dave Chinner @ 2015-12-03  6:51 UTC (permalink / raw)
  To: Jeff Moyer; +Cc: Ross Zwisler, xfs, Brian Foster, linux-fsdevel, linux-kernel

On Wed, Dec 02, 2015 at 10:12:29AM -0500, Jeff Moyer wrote:
> Dave Chinner <david@fromorbit.com> writes:
> 
> > On Tue, Dec 01, 2015 at 03:43:46PM -0700, Ross Zwisler wrote:
> >> I'm seeing a new intermittent xfstests failure with v4.4-rc3 using XFS and
> >> DAX.  The test that is failing is generic/083, and it fails about 50% of the
> >> time in my setup.  Here's the output:
> >> 
> >> FSTYP         -- xfs (debug)
> >> PLATFORM      -- Linux/x86_64 lorwyn 4.4.0-rc3
> >> MKFS_OPTIONS  -- -f -bsize=4096 /dev/pmem0p2
> >> MOUNT_OPTIONS -- -o dax -o context=system_u:object_r:nfs_t:s0 /dev/pmem0p2
> >> /mnt/xfstests_scratch
> >> 
> >> generic/083 2s ... 2s
>    ^^^^^^^
> >> _check_xfs_filesystem: filesystem on /dev/pmem0p2 is inconsistent (r) (see
> >> /root/xfstests/results//generic/083.full)
> >> Ran: generic/083
> >> Failures: generic/083
> >> Failed 1 of 1 tests
> >
> > $ grep 083  tests/xfs/group
> > 083 dangerous_fuzzers
> > $
> 
> That's great, but Ross reported an issues with *generic*/083.  ;-)
> 
> /me hands Dave more coffee.

Ah, some days I need more coffee than others :/

As it is, that test does not fail on my DAX testing on RAM disks.
ISTR it failing up until recently, though. Yeah:

Last login: Tue Nov 17 08:45:55 2015 from 192.168.1.103
$ cd ~/src/xfstests-dev; sudo mkfs.xfs -f /dev/ram0 ; sudo xfs_admin -U generate /dev/ram0 ; sudo MOUNT_OPTIONS="-o dax" ./check -g auto
SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test4 4.3.0-dgc+
MKFS_OPTIONS  -- -f -bsize=4096 /dev/ram1
MOUNT_OPTIONS -- -o dax /dev/ram1 /mnt/scratch
....
generic/083 2s ... 1s
_check_xfs_filesystem: filesystem on /dev/ram1 is inconsistent (c) (see /home/dave/src/xfstests-dev/results//xfs/generic/083.full)
_check_xfs_filesystem: filesystem on /dev/ram1 is inconsistent (r) (see /home/dave/src/xfstests-dev/results//xfs/generic/083.full)
....

But as I ran earlier today when testing the ENOSPC fix:

PLATFORM      -- Linux/x86_64 test4 4.4.0-rc2-dgc+
MKFS_OPTIONS  -- -f -bsize=4096 /dev/ram1
MOUNT_OPTIONS -- -o dax /dev/ram1 /mnt/scratch
....
generic/083 1s ... 2s
....

So I'm seeing that the current 4.4-rc2 + my local dev tree patches
appear to have fixed whatever was causing generic/083 to fail
here...

What version of xfsprogs are you using, Ross?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: xfstest failure with xfs, dax and v4.4-rc3
  2015-12-03  6:51     ` Dave Chinner
@ 2015-12-03 16:43       ` Ross Zwisler
  0 siblings, 0 replies; 5+ messages in thread
From: Ross Zwisler @ 2015-12-03 16:43 UTC (permalink / raw)
  To: Dave Chinner
  Cc: Jeff Moyer, Ross Zwisler, xfs, Brian Foster, linux-fsdevel, linux-kernel

On Thu, Dec 03, 2015 at 05:51:27PM +1100, Dave Chinner wrote:
> As it is, that test does not fail on my DAX testing on RAM disks.
> ISTR it failing up until recently, though. Yeah:
> 
> Last login: Tue Nov 17 08:45:55 2015 from 192.168.1.103
> $ cd ~/src/xfstests-dev; sudo mkfs.xfs -f /dev/ram0 ; sudo xfs_admin -U generate /dev/ram0 ; sudo MOUNT_OPTIONS="-o dax" ./check -g auto
> SECTION       -- xfs
> FSTYP         -- xfs (debug)
> PLATFORM      -- Linux/x86_64 test4 4.3.0-dgc+
> MKFS_OPTIONS  -- -f -bsize=4096 /dev/ram1
> MOUNT_OPTIONS -- -o dax /dev/ram1 /mnt/scratch
> ....
> generic/083 2s ... 1s
> _check_xfs_filesystem: filesystem on /dev/ram1 is inconsistent (c) (see /home/dave/src/xfstests-dev/results//xfs/generic/083.full)
> _check_xfs_filesystem: filesystem on /dev/ram1 is inconsistent (r) (see /home/dave/src/xfstests-dev/results//xfs/generic/083.full)
> ....
> 
> But as I ran earlier today when testing the ENOSPC fix:
> 
> PLATFORM      -- Linux/x86_64 test4 4.4.0-rc2-dgc+
> MKFS_OPTIONS  -- -f -bsize=4096 /dev/ram1
> MOUNT_OPTIONS -- -o dax /dev/ram1 /mnt/scratch
> ....
> generic/083 1s ... 2s
> ....
> 
> So I'm seeing that the current 4.4-rc2 + my local dev tree patches
> appear to have fixed whatever was causing generic/083 to fail
> here...

Yea, interesting, the patch you sent out yesterday:

xfs: Don't use reserved blocks for data blocks with DAX

Makes this issue disappear in my system as well.  Testing with v4.4-rc3
generic/083 fails 50% of the time or so in my setup, but with just that patch
added to v4.4-rc3 I can't make it fail.

> What version of xfsprogs are you using, Ross?

I'm currently using xfsprogs v4.3.0, but the failure occurred when I was using
v4.2.0 as well.

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2015-12-03 16:43 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-01 22:43 xfstest failure with xfs, dax and v4.4-rc3 Ross Zwisler
2015-12-01 23:51 ` Dave Chinner
2015-12-02 15:12   ` Jeff Moyer
2015-12-03  6:51     ` Dave Chinner
2015-12-03 16:43       ` Ross Zwisler

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).