* mkfs.ext4 -D option fails to mount
@ 2017-06-29 15:35 Ross Zwisler
2017-06-29 15:57 ` Ross Zwisler
0 siblings, 1 reply; 6+ messages in thread
From: Ross Zwisler @ 2017-06-29 15:35 UTC (permalink / raw)
To: linux-ext4
Our validation team noticed that in some configurations mkfs.ext4 with the
-D option creates a filesystem that can't be mounted:
# mkfs.ext4 -D -F /dev/pmem5
mke2fs 1.43.3 (04-Sep-2016)
/dev/pmem5 contains a ext4 file system
last mounted on Tue Jul 26 07:44:19 2016
Creating filesystem with 65027584 4k blocks and 16261120 inodes
Filesystem UUID: 6f95ece9-d4cb-4cfc-bc22-211119d5efe7
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done
# mount /dev/pmem5 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/pmem5,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
where dmesg says:
EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 1 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 1 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 1 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 2 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 2 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 2 overlaps superblock
...
EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 63 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 63 overlaps superblock
EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 63 overlaps superblock
EXT4-fs (pmem5): no journal found
If we omit the "-D" option from mkfs.ext4, everything works. Note also that
this behavior is independent of the DAX mount option.
This isn't blocking us, I just thought you would want to know.
- Ross
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: mkfs.ext4 -D option fails to mount 2017-06-29 15:35 mkfs.ext4 -D option fails to mount Ross Zwisler @ 2017-06-29 15:57 ` Ross Zwisler 2017-06-29 17:53 ` Theodore Ts'o 0 siblings, 1 reply; 6+ messages in thread From: Ross Zwisler @ 2017-06-29 15:57 UTC (permalink / raw) To: Ross Zwisler; +Cc: linux-ext4 On Thu, Jun 29, 2017 at 09:35:38AM -0600, Ross Zwisler wrote: > Our validation team noticed that in some configurations mkfs.ext4 with the > -D option creates a filesystem that can't be mounted: > > # mkfs.ext4 -D -F /dev/pmem5 > mke2fs 1.43.3 (04-Sep-2016) > /dev/pmem5 contains a ext4 file system > last mounted on Tue Jul 26 07:44:19 2016 > Creating filesystem with 65027584 4k blocks and 16261120 inodes > Filesystem UUID: 6f95ece9-d4cb-4cfc-bc22-211119d5efe7 > Superblock backups stored on blocks: > 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, > 4096000, 7962624, 11239424, 20480000, 23887872 > > Allocating group tables: done > Writing inode tables: done > Creating journal (262144 blocks): done > Writing superblocks and filesystem accounting information: done > > # mount /dev/pmem5 /mnt > mount: wrong fs type, bad option, bad superblock on /dev/pmem5, > missing codepage or helper program, or other error > > In some cases useful info is found in syslog - try > dmesg | tail or so. > > where dmesg says: > > EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 1 overlaps superblock > EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 1 overlaps superblock > EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 1 overlaps superblock > EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 2 overlaps superblock > EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 2 overlaps superblock > EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 2 overlaps superblock > ... > EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 63 overlaps superblock > EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 63 overlaps superblock > EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 63 overlaps superblock > EXT4-fs (pmem5): no journal found > > If we omit the "-D" option from mkfs.ext4, everything works. Note also that > this behavior is independent of the DAX mount option. > > This isn't blocking us, I just thought you would want to know. One more bit of info - this seems to be strongly tied to the size of the block device. With a 32 GB block device it works fine, with 248 GB you get overlap messages for groups 1 through 63, and with a 250 GB device you get overlaps for groups 1 through 1999. I've been varying my virtual NVDIMM namespace size via QEMU. Here's are the relevant bits from my QEMU my command line to enable the NVDIMM: -m 8G,slots=3,maxmem=512G -machine pc,accel=kvm,nvdimm -object memory-backend-file,id=mem1,share,mem-path=/tmp/nvdimm-8,size=250G -device nvdimm,memdev=mem1,id=nv1 Here's my full QEMU command line, in case that's interesting: sudo /usr/bin/qemu-system-x86_64 /home/rzwisler/vms/amonkhet-8.qcow2 -m 8G,slots=3,maxmem=512G -smp 6 -machine pc,accel=kvm,nvdimm -enable-kvm -netdev tap,id=hostnet0,ifname=tap8,script=no,downscript=no -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:94:b8,bus=pci.0,addr=0x8 -rtc base=localtime -serial stdio -display none -monitor unix:/tmp/amonkhet-8.monitor,server,nowait -object memory-backend-file,id=mem1,share,mem-path=/tmp/nvdimm-8,size=250G -device nvdimm,memdev=mem1,id=nv1 - Ross ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount 2017-06-29 15:57 ` Ross Zwisler @ 2017-06-29 17:53 ` Theodore Ts'o 2017-06-29 23:06 ` Ross Zwisler 0 siblings, 1 reply; 6+ messages in thread From: Theodore Ts'o @ 2017-06-29 17:53 UTC (permalink / raw) To: Ross Zwisler; +Cc: linux-ext4 On Thu, Jun 29, 2017 at 09:57:27AM -0600, Ross Zwisler wrote: > On Thu, Jun 29, 2017 at 09:35:38AM -0600, Ross Zwisler wrote: > > Our validation team noticed that in some configurations mkfs.ext4 with the > > -D option creates a filesystem that can't be mounted: The -D option just means that we're doing the I/O using Direct I/O (as opposed to buffered I/O). It shouldn't make any difference to what gets written, so this very much smells like a bug in how /dev/pmem supports Direct I/O... > One more bit of info - this seems to be strongly tied to the size of the > block device. With a 32 GB block device it works fine, with 248 GB you get > overlap messages for groups 1 through 63, and with a 250 GB device you get > overlaps for groups 1 through 1999. This very much sounds like Direct I/O is just getting completely botched for the pmem device, and writes to a block group descriptor block is affecting the wrong place on the storage device. - Ted ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount 2017-06-29 17:53 ` Theodore Ts'o @ 2017-06-29 23:06 ` Ross Zwisler 2017-06-29 23:48 ` Theodore Ts'o 0 siblings, 1 reply; 6+ messages in thread From: Ross Zwisler @ 2017-06-29 23:06 UTC (permalink / raw) To: Theodore Ts'o; +Cc: Ross Zwisler, linux-ext4 On Thu, Jun 29, 2017 at 01:53:19PM -0400, Theodore Ts'o wrote: > On Thu, Jun 29, 2017 at 09:57:27AM -0600, Ross Zwisler wrote: > > On Thu, Jun 29, 2017 at 09:35:38AM -0600, Ross Zwisler wrote: > > > Our validation team noticed that in some configurations mkfs.ext4 with the > > > -D option creates a filesystem that can't be mounted: > > The -D option just means that we're doing the I/O using Direct I/O (as > opposed to buffered I/O). It shouldn't make any difference to what > gets written, so this very much smells like a bug in how /dev/pmem > supports Direct I/O... > > > One more bit of info - this seems to be strongly tied to the size of the > > block device. With a 32 GB block device it works fine, with 248 GB you get > > overlap messages for groups 1 through 63, and with a 250 GB device you get > > overlaps for groups 1 through 1999. > > This very much sounds like Direct I/O is just getting completely > botched for the pmem device, and writes to a block group descriptor > block is affecting the wrong place on the storage device. This also reproduces with brd or loop as our block device: # modprobe brd rd_size=$((1024*1024*248)) # mkfs.ext4 /dev/ram0 -F -D mke2fs 1.43.3 (04-Sep-2016) Discarding device blocks: done Creating filesystem with 65011712 4k blocks and 16252928 inodes Filesystem UUID: 1632baa4-7260-4cc1-9558-0cb3aa2f213e Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: done # mount /dev/ram0 /mnt mount: wrong fs type, bad option, bad superblock on /dev/ram0, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. where dmesg says: EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode table for group 64 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 65 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 65 overlaps superblock ... EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 1983 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 1983 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode table for group 1983 overlaps superblock EXT4-fs (ram0): no journal found or # truncate -s 248G loop_fs # losetup /dev/loop0 ./loop_fs # mkfs.ext4 /dev/loop0 -F -D mke2fs 1.43.3 (04-Sep-2016) /dev/loop0 contains a ext4 file system last mounted on Thu Jun 29 17:03:07 2017 Discarding device blocks: done Creating filesystem with 65011712 4k blocks and 16252928 inodes Filesystem UUID: 00912ac9-599a-4396-9ef3-a353bdec69ea Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: done # mount /dev/loop0 /tmp mount: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. with similar messages. - Ross ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount 2017-06-29 23:06 ` Ross Zwisler @ 2017-06-29 23:48 ` Theodore Ts'o 2017-06-30 17:47 ` Ross Zwisler 0 siblings, 1 reply; 6+ messages in thread From: Theodore Ts'o @ 2017-06-29 23:48 UTC (permalink / raw) To: Ross Zwisler; +Cc: linux-ext4 I'm not able to reproduce it using your reproduction receipe using e2fsprogs 1.43.4. I suspect you might be hitting a bug which was fixed by this commit: commit d6cad379eb6c86ca58bf5b83a586577de412a2e6 Author: Theodore Ts'o <tytso@mit.edu> Date: Sun Sep 11 00:25:48 2016 -0400 libext2fs: fix unaligned, multiblock writes in the unix_io handler The read-modify-write code for the unaligned fallback code wasn't working for multi-block writes. This was unmasked by FreeBSD 11-rc2, since its malloc() is returning unaligned memory regions for large memory regions. Signed-off-by: Theodore Ts'o <tytso@mit.edu> Can you retry with e2fsprogs 1.43.4 and see if it fixes your issue? - Ted ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mkfs.ext4 -D option fails to mount 2017-06-29 23:48 ` Theodore Ts'o @ 2017-06-30 17:47 ` Ross Zwisler 0 siblings, 0 replies; 6+ messages in thread From: Ross Zwisler @ 2017-06-30 17:47 UTC (permalink / raw) To: Theodore Ts'o; +Cc: Ross Zwisler, linux-ext4 On Thu, Jun 29, 2017 at 07:48:16PM -0400, Theodore Ts'o wrote: > I'm not able to reproduce it using your reproduction receipe using > e2fsprogs 1.43.4. I suspect you might be hitting a bug which was > fixed by this commit: > > commit d6cad379eb6c86ca58bf5b83a586577de412a2e6 > Author: Theodore Ts'o <tytso@mit.edu> > Date: Sun Sep 11 00:25:48 2016 -0400 > > libext2fs: fix unaligned, multiblock writes in the unix_io handler > > The read-modify-write code for the unaligned fallback code wasn't > working for multi-block writes. This was unmasked by FreeBSD 11-rc2, > since its malloc() is returning unaligned memory regions for large > memory regions. > > Signed-off-by: Theodore Ts'o <tytso@mit.edu> > > Can you retry with e2fsprogs 1.43.4 and see if it fixes your issue? Yep, confirmed that it works with e2fsprogs v1.43.4. Cool, thanks. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-06-30 17:47 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-06-29 15:35 mkfs.ext4 -D option fails to mount Ross Zwisler 2017-06-29 15:57 ` Ross Zwisler 2017-06-29 17:53 ` Theodore Ts'o 2017-06-29 23:06 ` Ross Zwisler 2017-06-29 23:48 ` Theodore Ts'o 2017-06-30 17:47 ` Ross Zwisler
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.