* mkfs.ext4 -D option fails to mount
@ 2017-06-29 15:35 Ross Zwisler
2017-06-29 15:57 ` Ross Zwisler
0 siblings, 1 reply; 6+ messages in thread
From: Ross Zwisler @ 2017-06-29 15:35 UTC (permalink / raw)
To: linux-ext4
Our validation team noticed that in some configurations mkfs.ext4 with the
-D option creates a filesystem that can't be mounted:
# mkfs.ext4 -D -F /dev/pmem5
mke2fs 1.43.3 (04-Sep-2016)
/dev/pmem5 contains a ext4 file system
last mounted on Tue Jul 26 07:44:19 2016
Creating filesystem with 65027584 4k blocks and 16261120 inodes
Filesystem UUID: 6f95ece9-d4cb-4cfc-bc22-211119d5efe7
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done
# mount /dev/pmem5 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/pmem5,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
where dmesg says:
EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 1 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 1 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 1 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 2 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 2 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 2 overlaps superblock
...
EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 63 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 63 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 63 overlaps superblock
EXT4-fs (pmem5): no journal found
If we omit the "-D" option from mkfs.ext4, everything works. Note also that
this behavior is independent of the DAX mount option.
This isn't blocking us, I just thought you would want to know.
- Ross
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount
2017-06-29 15:35 mkfs.ext4 -D option fails to mount Ross Zwisler
@ 2017-06-29 15:57 ` Ross Zwisler
2017-06-29 17:53 ` Theodore Ts'o
0 siblings, 1 reply; 6+ messages in thread
From: Ross Zwisler @ 2017-06-29 15:57 UTC (permalink / raw)
To: Ross Zwisler; +Cc: linux-ext4
On Thu, Jun 29, 2017 at 09:35:38AM -0600, Ross Zwisler wrote:
> Our validation team noticed that in some configurations mkfs.ext4 with the
> -D option creates a filesystem that can't be mounted:
>
> # mkfs.ext4 -D -F /dev/pmem5
> mke2fs 1.43.3 (04-Sep-2016)
> /dev/pmem5 contains a ext4 file system
> last mounted on Tue Jul 26 07:44:19 2016
> Creating filesystem with 65027584 4k blocks and 16261120 inodes
> Filesystem UUID: 6f95ece9-d4cb-4cfc-bc22-211119d5efe7
> Superblock backups stored on blocks:
> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
> 4096000, 7962624, 11239424, 20480000, 23887872
>
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (262144 blocks): done
> Writing superblocks and filesystem accounting information: done
>
> # mount /dev/pmem5 /mnt
> mount: wrong fs type, bad option, bad superblock on /dev/pmem5,
> missing codepage or helper program, or other error
>
> In some cases useful info is found in syslog - try
> dmesg | tail or so.
>
> where dmesg says:
>
> EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 1 overlaps superblock
> EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 1 overlaps superblock
> EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 1 overlaps superblock
> EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 2 overlaps superblock
> EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 2 overlaps superblock
> EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 2 overlaps superblock
> ...
> EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 63 overlaps superblock
> EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 63 overlaps superblock
> EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 63 overlaps superblock
> EXT4-fs (pmem5): no journal found
>
> If we omit the "-D" option from mkfs.ext4, everything works. Note also that
> this behavior is independent of the DAX mount option.
>
> This isn't blocking us, I just thought you would want to know.
One more bit of info - this seems to be strongly tied to the size of the
block device. With a 32 GB block device it works fine, with 248 GB you get
overlap messages for groups 1 through 63, and with a 250 GB device you get
overlaps for groups 1 through 1999.
I've been varying my virtual NVDIMM namespace size via QEMU.
Here's are the relevant bits from my QEMU my command line to enable the
NVDIMM:
-m 8G,slots=3,maxmem=512G -machine pc,accel=kvm,nvdimm
-object memory-backend-file,id=mem1,share,mem-path=/tmp/nvdimm-8,size=250G
-device nvdimm,memdev=mem1,id=nv1
Here's my full QEMU command line, in case that's interesting:
sudo /usr/bin/qemu-system-x86_64 /home/rzwisler/vms/amonkhet-8.qcow2 -m
8G,slots=3,maxmem=512G -smp 6 -machine pc,accel=kvm,nvdimm -enable-kvm -netdev
tap,id=hostnet0,ifname=tap8,script=no,downscript=no -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:94:b8,bus=pci.0,addr=0x8
-rtc base=localtime -serial stdio -display none -monitor
unix:/tmp/amonkhet-8.monitor,server,nowait -object
memory-backend-file,id=mem1,share,mem-path=/tmp/nvdimm-8,size=250G -device
nvdimm,memdev=mem1,id=nv1
- Ross
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount
2017-06-29 15:57 ` Ross Zwisler
@ 2017-06-29 17:53 ` Theodore Ts'o
2017-06-29 23:06 ` Ross Zwisler
0 siblings, 1 reply; 6+ messages in thread
From: Theodore Ts'o @ 2017-06-29 17:53 UTC (permalink / raw)
To: Ross Zwisler; +Cc: linux-ext4
On Thu, Jun 29, 2017 at 09:57:27AM -0600, Ross Zwisler wrote:
> On Thu, Jun 29, 2017 at 09:35:38AM -0600, Ross Zwisler wrote:
> > Our validation team noticed that in some configurations mkfs.ext4 with the
> > -D option creates a filesystem that can't be mounted:
The -D option just means that we're doing the I/O using Direct I/O (as
opposed to buffered I/O). It shouldn't make any difference to what
gets written, so this very much smells like a bug in how /dev/pmem
supports Direct I/O...
> One more bit of info - this seems to be strongly tied to the size of the
> block device. With a 32 GB block device it works fine, with 248 GB you get
> overlap messages for groups 1 through 63, and with a 250 GB device you get
> overlaps for groups 1 through 1999.
This very much sounds like Direct I/O is just getting completely
botched for the pmem device, and writes to a block group descriptor
block is affecting the wrong place on the storage device.
- Ted
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount
2017-06-29 17:53 ` Theodore Ts'o
@ 2017-06-29 23:06 ` Ross Zwisler
2017-06-29 23:48 ` Theodore Ts'o
0 siblings, 1 reply; 6+ messages in thread
From: Ross Zwisler @ 2017-06-29 23:06 UTC (permalink / raw)
To: Theodore Ts'o; +Cc: Ross Zwisler, linux-ext4
On Thu, Jun 29, 2017 at 01:53:19PM -0400, Theodore Ts'o wrote:
> On Thu, Jun 29, 2017 at 09:57:27AM -0600, Ross Zwisler wrote:
> > On Thu, Jun 29, 2017 at 09:35:38AM -0600, Ross Zwisler wrote:
> > > Our validation team noticed that in some configurations mkfs.ext4 with the
> > > -D option creates a filesystem that can't be mounted:
>
> The -D option just means that we're doing the I/O using Direct I/O (as
> opposed to buffered I/O). It shouldn't make any difference to what
> gets written, so this very much smells like a bug in how /dev/pmem
> supports Direct I/O...
>
> > One more bit of info - this seems to be strongly tied to the size of the
> > block device. With a 32 GB block device it works fine, with 248 GB you get
> > overlap messages for groups 1 through 63, and with a 250 GB device you get
> > overlaps for groups 1 through 1999.
>
> This very much sounds like Direct I/O is just getting completely
> botched for the pmem device, and writes to a block group descriptor
> block is affecting the wrong place on the storage device.
This also reproduces with brd or loop as our block device:
# modprobe brd rd_size=$((1024*1024*248))
# mkfs.ext4 /dev/ram0 -F -D
mke2fs 1.43.3 (04-Sep-2016)
Discarding device blocks: done
Creating filesystem with 65011712 4k blocks and 16252928 inodes
Filesystem UUID: 1632baa4-7260-4cc1-9558-0cb3aa2f213e
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done
# mount /dev/ram0 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/ram0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
where dmesg says:
EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (ram0): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 65 overlaps superblock
EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 65 overlaps superblock
...
EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 1983 overlaps superblock
EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 1983 overlaps superblock
EXT4-fs (ram0): ext4_check_descriptors: Inode table for group 1983 overlaps superblock
EXT4-fs (ram0): no journal found
or
# truncate -s 248G loop_fs
# losetup /dev/loop0 ./loop_fs
# mkfs.ext4 /dev/loop0 -F -D
mke2fs 1.43.3 (04-Sep-2016)
/dev/loop0 contains a ext4 file system
last mounted on Thu Jun 29 17:03:07 2017
Discarding device blocks: done
Creating filesystem with 65011712 4k blocks and 16252928 inodes
Filesystem UUID: 00912ac9-599a-4396-9ef3-a353bdec69ea
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done
# mount /dev/loop0 /tmp
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
with similar messages.
- Ross
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount
2017-06-29 23:06 ` Ross Zwisler
@ 2017-06-29 23:48 ` Theodore Ts'o
2017-06-30 17:47 ` Ross Zwisler
0 siblings, 1 reply; 6+ messages in thread
From: Theodore Ts'o @ 2017-06-29 23:48 UTC (permalink / raw)
To: Ross Zwisler; +Cc: linux-ext4
I'm not able to reproduce it using your reproduction receipe using
e2fsprogs 1.43.4. I suspect you might be hitting a bug which was
fixed by this commit:
commit d6cad379eb6c86ca58bf5b83a586577de412a2e6
Author: Theodore Ts'o <tytso@mit.edu>
Date: Sun Sep 11 00:25:48 2016 -0400
libext2fs: fix unaligned, multiblock writes in the unix_io handler
The read-modify-write code for the unaligned fallback code wasn't
working for multi-block writes. This was unmasked by FreeBSD 11-rc2,
since its malloc() is returning unaligned memory regions for large
memory regions.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Can you retry with e2fsprogs 1.43.4 and see if it fixes your issue?
- Ted
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount
2017-06-29 23:48 ` Theodore Ts'o
@ 2017-06-30 17:47 ` Ross Zwisler
0 siblings, 0 replies; 6+ messages in thread
From: Ross Zwisler @ 2017-06-30 17:47 UTC (permalink / raw)
To: Theodore Ts'o; +Cc: Ross Zwisler, linux-ext4
On Thu, Jun 29, 2017 at 07:48:16PM -0400, Theodore Ts'o wrote:
> I'm not able to reproduce it using your reproduction receipe using
> e2fsprogs 1.43.4. I suspect you might be hitting a bug which was
> fixed by this commit:
>
> commit d6cad379eb6c86ca58bf5b83a586577de412a2e6
> Author: Theodore Ts'o <tytso@mit.edu>
> Date: Sun Sep 11 00:25:48 2016 -0400
>
> libext2fs: fix unaligned, multiblock writes in the unix_io handler
>
> The read-modify-write code for the unaligned fallback code wasn't
> working for multi-block writes. This was unmasked by FreeBSD 11-rc2,
> since its malloc() is returning unaligned memory regions for large
> memory regions.
>
> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
>
> Can you retry with e2fsprogs 1.43.4 and see if it fixes your issue?
Yep, confirmed that it works with e2fsprogs v1.43.4. Cool, thanks.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-06-30 17:47 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-29 15:35 mkfs.ext4 -D option fails to mount Ross Zwisler
2017-06-29 15:57 ` Ross Zwisler
2017-06-29 17:53 ` Theodore Ts'o
2017-06-29 23:06 ` Ross Zwisler
2017-06-29 23:48 ` Theodore Ts'o
2017-06-30 17:47 ` Ross Zwisler
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.