Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

Hi all... I usually run my system the zen kernel (zen-kernel.org), but
I've tried this with the default Archlinux kernel (pretty much a
vanilla build) as well and for some reason, the root drive won't mount
with the options I desire on anything above .29 The drive is an SSD,
and this is really getting on my nerves. I've looked at the
initscripts, fstab, everything else, and I just can't get any of this
to work properly. It was mounting with all the options I specified in
2.6.39, but since some change to the kernel (or something else?) I
can't mount with my options anymore. On top of this all, I've tried
using both /dev/root and /dev/sdb1, but for some reason the kernel's
only wanting to use /dev/root - I'm wondering if there's some kind of
conflict due to that symlink...

I know this isn't my initscripts, as I've also tried this using
openrc, aside from the default arch init system, and because upon
downgrading to 2.6.39 everything is working normally again, I know it
is the kernel. Information:

grub.cfg:

linux	/boot/linux-zen root=/dev/sdb1 ro rootfstype=ext4
rootflags=data=writeback noatime barrier=0 discard noinitrd

fstab:

/dev/sdb1 / ext4 data=writeback,noatime,nodiratime,barrier=0,discard 0 1

Upon attempting to remount manually, I only get the following when I
try to use the options I specified above:

EXT4-fs (sdb1): Cannot change data mode on remount

Interestingly, this also happened and worked, which I think is a bug,
how can barrier=0 and 1 at the same time?

EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0

I'm currently using a .39 kernel and can live with it for now, but I
thought this was serious enough that the ext4 devs should hear about
it. If you need any more information, just let me know.

Thanks!

Matt Parnell

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Christian Kujau]

On Tue, 13 Sep 2011 at 04:01, Matt Parnell wrote:
> I know this isn't my initscripts, as I've also tried this using
> openrc, aside from the default arch init system, and because upon
> downgrading to 2.6.39 everything is working normally again, I know it

Did you try booting with init=/bin/bash, i.e. bypassing all initscripts? 
Also, try booting w/o an initrd image if you can, thus bypassing all the 
early boot magic.

Shot in the dark: if it worked with 2.6.39 but not with 3.x and the 
initscript have not changed, maybe some initscript went mad when it 
could not parse the new kernel version number?

Christian.
-- 
BOFH excuse #141:

disks spinning backwards - toggle the hemisphere jumper.

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Christian Kujau]

[ Cc'ing linux-ext4 again ]

On Tue, 13 Sep 2011 at 05:44, Matt Parnell wrote:
> > Did you try booting with init=/bin/bash, i.e. bypassing all initscripts?
> > Also, try booting w/o an initrd image if you can, thus bypassing all the
> > early boot magic.
> 
> Not exactly, but I had it drop out to perform an fsck, and the mtab
> indicated it was mounted according to my options, but this was when
> the disk was mounted read only from grub. Also, I don't use an
> initrd...so that's not it either.

Yeah, mtab is no good when your rootfs is readonly. What did /proc/mounts 
say?

C.

> > Shot in the dark: if it worked with 2.6.39 but not with 3.x and the
> > initscript have not changed, maybe some initscript went mad when it
> > could not parse the new kernel version number?
> >
> > Christian.
> 
> It's not that either... Arch has been using 3.0+ as it's default for a
> few weeks now, so they would've updated their scripts for that.
> Furthermore, I tried using OpenRC (the gentoo init system) and got the
> same results.
> 

-- 
BOFH excuse #359:

YOU HAVE AN I/O ERROR -> Incompetent Operator error

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Eric Sandeen]

On 9/13/11 4:01 AM, Matt Parnell wrote:

> Interestingly, this also happened and worked, which I think is a bug,
> how can barrier=0 and 1 at the same time?
> 
> EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0

Fairly certain that the parser will just take the last value specified
for the option.

/proc/mounts likely says "nobarrier" when you specify it this way?

-Eric

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

[-- Attachment #1: Type: text/plain, Size: 3035 bytes --]

FYI this behavior still occurs with every revision up to the current
3.0.6 working tree. I do not think it's my distro's initscripts,
they've been rolling 3.0.X kernels for some time and those behave the
same way, so it's not some sort of parsing issues - Arch's initscripts
are all bash based for the most part anyway, see the attached from my
system - that's the entire boot initscript minus my daemons.

There's no alternative init system for Archlinux at this point, I'm
working on porting OpenRC from gentoo as I have time to write
initscripts from scratch, but that's a slow moving project.

Whatever the case, the following are from my most recent attempt with
a 3.0.6 build. Manually attempting to remount rw still fails, and no
matter whether I use /dev/sdb1 or /dev/root, I get the same behavior.

/proc/mounts (as you can see it's not being remounted rw for some
reason - /dev/sdc1 is the flash drive I used to pipe this info onto):

rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0
devtmpfs /dev devtmpfs rw,relatime,size=4059824k,nr_inodes=1014956,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
/sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
/run /run tmpfs rw,nosuid,nodev,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/sda4 /media/disk ext3
rw,nosuid,nodev,noatime,nodiratime,errors=continue,barrier=0,data=writeback
0 0
/dev/sdc1 /mnt/tmp vfat
rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
0 0

/etc/mtab (not useful, I know, but for reference):

/dev/root / ext4
rw,noatime,nodiratime,data=writeback,barrier=0,discard,commit=0 0 0
devtmpfs /dev devtmpfs rw,relatime,size=4059656k,nr_inodes=1014914,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
/sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
/run /run tmpfs rw,nosuid,nodev,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev 0 0


/etc/fstab:

#
# /etc/fstab: static file system information
#
devpts                 /dev/pts      devpts    defaults            0      0
shm                    /dev/shm      tmpfs     nodev,nosuid        0      0
/dev/sdb1 / ext4
defaults,noatime,nodiratime,data=writeback,barrier=0,discard 0 0
/dev/sda4 /media/disk ext3
defaults,data=writeback,noatime,nodiratime,user,exec 0 0
tmpfs		/tmp	tmpfs	nodev,nosuid	0	0



>On 09/13/2011 09:49 AM, Eric Sandeen wrote:
>
>On 9/13/11 4:01 AM, Matt Parnell wrote:
>
>Interestingly, this also happened and worked, which I think is a bug,
>how can barrier=0 and 1 at the same time?
>
>EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0
>
>Fairly certain that the parser will just take the last value specified
>for the option.
>
>/proc/mounts likely says "nobarrier" when you specify it this way?
>
>-Eric

[-- Attachment #2: rc.sysinit --]
[-- Type: application/octet-stream, Size: 8742 bytes --]

#!/bin/bash
#
# /etc/rc.sysinit
#

. /etc/rc.conf
. /etc/rc.d/functions

echo " "
printhl "Arch Linux\n"
printhl "${C_H2}http://www.archlinux.org"
printsep

# mount /proc, /sys, /run, /dev, /run/lock, /dev/pts, /dev/shm (the api filesystems)
mountpoint -q /proc    || mount -n -t proc proc /proc -o nosuid,noexec,nodev
mountpoint -q /sys     || mount -n -t sysfs sys /sys -o nosuid,noexec,nodev
mountpoint -q /run     || mount -n -t tmpfs run /run -o mode=0755,size=10M,nosuid,nodev
mountpoint -q /dev     || mount -n -t devtmpfs udev /dev -o mode=0755,size=10M,nosuid \
	|| mount -n -t tmpfs udev /dev -o mode=0755,size=10M,nosuid
mkdir -p -m 1777 /run/lock
mkdir -p /dev/{pts,shm}
mountpoint -q /dev/pts || mount -n /dev/pts &>/dev/null \
	|| mount -n -t devpts devpts /dev/pts -o mode=0620,gid=5,nosuid,noexec
mountpoint -q /dev/shm || mount -n /dev/shm &>/dev/null \
	|| mount -n -t tmpfs shm /dev/shm -o mode=1777,nosuid,nodev

# remount root ro to allow for fsck later on, we remount now to
# make sure nothing can open files rw on root which would block a remount
findmnt / --options ro &>/dev/null ||
	status "Mounting Root Read-Only" mount -n -o remount,ro /

run_hook sysinit_start

# start up our mini logger until syslog takes over
minilogd
bootlogd -p /run/bootlogd.pid

if [[ ! -a /usr/lib ]] ; then
  printf "${C_FAIL}/usr is not mounted. This is not supported.${C_OTHER}\n"
fi

HWCLOCK_PARAMS="--systz"
case $HARDWARECLOCK in
	"") ;;
	UTC) HWCLOCK_PARAMS+=" --utc --noadjfile";;
	localtime) HWCLOCK_PARAMS+=" --localtime --noadjfile";;
	*) HWCLOCK_PARAMS="";;
esac

if [[ $HWCLOCK_PARAMS ]]; then
	stat_busy "Adjusting system time and setting kernel timezone"
		# enable rtc access
		modprobe -q -a rtc-cmos rtc genrtc
		# If devtmpfs is used, the required RTC device already exists now
		# Otherwise, create whatever device is available
		if ! [[ -c /dev/rtc || -c /dev/rtc0 ]]; then
			for dev in /sys/class/rtc/rtc0/dev /sys/class/misc/rtc/dev; do
				[[ -e $dev ]] || continue
				IFS=: read -r major minor < "$dev"
				mknod /dev/rtc c $major $minor
			done
		fi

		# Adjust the system time for timezone offset if rtc is not in UTC
		# 1. Make creation time on udev nodes sane (FS#8665)
		# 2. Filesystem checks can depend on system time
		# 3. This also sets the kernel time zone, used by e.g. vfat
		# If TIMEZONE is not set in rc.conf, the timezone stored in /etc/localtime
		# is used. If HARDWARECLOCK is not set in rc.conf, the value in
		# /var/lib/hwclock/adjfile is used (in this case /var can not be a separate
		# partition).
	TZ=$TIMEZONE hwclock $HWCLOCK_PARAMS && stat_done || stat_fail
fi

# Start/trigger UDev, load MODULES and settle UDev
udevd_modprobe sysinit

# bring up the loopback interface
[[ -d /sys/class/net/lo ]] &&
	status "Bringing up loopback interface" ip link set up dev lo

# FakeRAID devices detection
[[ $USEDMRAID = [Yy][Ee][Ss] && -x $(type -P dmraid) ]] &&
	status "Activating FakeRAID arrays" dmraid -i -ay

# BTRFS devices detection
[[ $USEBTRFS = [Yy][Ee][Ss] && -x $(type -P btrfs) ]] &&
	status "Activating BTRFS volumes" btrfs device scan

# Activate LVM2 groups if any
activate_vgs

# Set up non-root encrypted partition mappings
if [[ -f /etc/crypttab && $CS ]] && grep -q ^[^#] /etc/crypttab; then
	stat_busy "Unlocking encrypted volumes:"
		modprobe -q dm-crypt 2>/dev/null
		do_unlock() {
			# $1 = requested name
			# $2 = source device
			# $3 = password
			# $4 = options
			stat_append "${1}.."
			local open=create a=$1 b=$2 failed=0
			# Ordering of options is different if you are using LUKS vs. not.
			# Use ugly swizzling to deal with it.
			# isLuks only gives an exit code but no output to stdout or stderr.
			if $CS isLuks "$2" 2>/dev/null; then
				open=luksOpen
				a=$2
				b=$1
			fi
			case $3 in
				SWAP)
					local _overwriteokay=0
					if [[ -b $2 && -r $2 ]]; then
						# This is DANGEROUS! If there is any known file system,
						# partition table, RAID or LVM volume on the device
						# we don't overwrite it.
						#
						# 'blkid' returns 2 if no valid signature has been found.
						# Only in this case we should allow overwriting the device.
						#
						# This sanity check _should_ be sufficient, but it might not.
						# This may cause dataloss if it is not used carefully.
						blkid -p "$2" &>/dev/null
						(( $? == 2 )) && _overwriteokay=1
					fi
					if (( _overwriteokay == 0 )); then
						false
					elif $CS -d /dev/urandom $4 $open "$a" "$b" >/dev/null; then
						stat_append "creating swapspace.."
						mkswap -f -L $1 /dev/mapper/$1 >/dev/null
					fi;;
				ASK)
					printf "\nOpening '$1' volume:\n"
					$CS $4 $open "$a" "$b" < /dev/console;;
				/dev*)
					local ckdev=${3%%:*}
					local cka=${3#*:}
					local ckb=${cka#*:}
					local cka=${cka%:*}
					local ckfile=/dev/ckfile
					local ckdir=/dev/ckdir
					case ${cka} in
						*[!0-9]*)
							# Use a file on the device
							# cka is not numeric: cka=filesystem, ckb=path
							mkdir ${ckdir}
							mount -r -t ${cka} ${ckdev} ${ckdir}
							dd if=${ckdir}/${ckb} of=${ckfile} >/dev/null 2>&1
							umount ${ckdir}
							rmdir ${ckdir};;
						*)
							# Read raw data from the block device
							# cka is numeric: cka=offset, ckb=length
							dd if=${ckdev} of=${ckfile} bs=1 skip=${cka} count=${ckb} >/dev/null 2>&1;;
					esac
					$CS -d ${ckfile} $4 $open "$a" "$b" >/dev/null
					dd if=/dev/urandom of=${ckfile} bs=1 count=$(stat -c %s ${ckfile}) conv=notrunc >/dev/null 2>&1
					rm ${ckfile};;
				/*)
					$CS -d "$3" $4 $open "$a" "$b" >/dev/null;;
				*)
					echo "$3" | $CS $4 $open "$a" "$b" >/dev/null;;
			esac
			if (( $? )); then
				failed=1
				stat_append "failed "
			else
				stat_append "ok "
			fi
			return $failed
		}
	crypto_unlocked=0
	read_crypttab do_unlock && stat_done || stat_fail
	# Maybe someone has LVM on an encrypted block device
	(( crypto_unlocked == 1 )) && activate_vgs
fi

# Check filesystems
[[ -f /forcefsck ]] || in_array forcefsck $(< /proc/cmdline) && FORCEFSCK="-- -f"
declare -r FORCEFSCK
run_hook sysinit_prefsck
if [[ -x $(type -P fsck) ]]; then
	stat_busy "Checking Filesystems"
		fsck_all >|"${FSCK_OUT:-/dev/stdout}" 2>|"${FSCK_ERR:-/dev/stdout}"
	declare -r fsckret=$?
	(( fsckret <= 1 )) && stat_done || stat_fail
else
	declare -r fsckret=0
fi
run_hook sysinit_postfsck

# Single-user login and/or automatic reboot if needed
fsck_reboot $fsckret

status "Remounting Root Read/Write" \
	mount -n -o remount,rw /

# don't touch /etc/mtab if it is a symlink to /proc/self/mounts
if [[ ! -L /etc/mtab ]]; then
	stat_busy "Creating mtab"
		if [[ -x $(type -P findmnt) && -e /proc/self/mountinfo ]]; then
			findmnt -rnu -o SOURCE,TARGET,FSTYPE,OPTIONS >| /etc/mtab
		else
			cat /proc/mounts >| /etc/mtab
		fi
	(( $? == 0 )) && stat_done || stat_fail
fi

# now mount all the local filesystems
run_hook sysinit_premount
status "Mounting Local Filesystems" \
	mount_all
run_hook sysinit_postmount

# enable monitoring of lvm2 groups, now that the filesystems are mounted rw
[[ $USELVM = [Yy][Ee][Ss] && -x $(type -P lvm) && -d /sys/block ]] &&
	status "Activating monitoring of LVM2 groups" \
		vgchange --monitor y >/dev/null

status "Activating Swap" swapon -a

[[ $TIMEZONE ]] && status "Configuring Time Zone" set_timezone "$TIMEZONE"

RANDOM_SEED=/var/lib/misc/random-seed
[[ -f $RANDOM_SEED ]] &&
	status "Initializing Random Seed" \
		cp $RANDOM_SEED /dev/urandom

# Remove leftover files
remove_leftover

if [[ $HOSTNAME ]]; then
	stat_busy "Setting Hostname: $HOSTNAME"
	echo "$HOSTNAME" >| /proc/sys/kernel/hostname && stat_done || stat_fail
fi

if [[ ${LOCALE,,} =~ utf ]]; then
	stat_busy "Setting Consoles to UTF-8 mode"
		# UTF-8 consoles are default since 2.6.24 kernel
		# this code is needed not only for older kernels,
		# but also when user has set vt.default_utf8=0 but LOCALE is *.UTF-8.
		for i in /dev/tty[0-9]*; do
			kbd_mode -u < ${i}
			printf "\e%%G" > ${i}
		done
		echo 1 >| /sys/module/vt/parameters/default_utf8
	stat_done
else
	stat_busy "Setting Consoles to legacy mode"
		# make non-UTF-8 consoles work on 2.6.24 and newer kernels
		for i in /dev/tty[0-9]*; do
			kbd_mode -a < ${i}
			printf "\e%%@" > ${i}
		done
		echo 0 >| /sys/module/vt/parameters/default_utf8
	stat_done
fi
[[ $KEYMAP ]] &&
	status "Loading Keyboard Map: $KEYMAP" loadkeys -q $KEYMAP

# Set console font if required
set_consolefont

stat_busy "Saving dmesg Log"
	if [[ -e /proc/sys/kernel/dmesg_restrict ]] &&
		(( $(< /proc/sys/kernel/dmesg_restrict) == 1 )); then
		install -Tm 0600 <( dmesg ) /var/log/dmesg.log
	else
		install -Tm 0644 <( dmesg ) /var/log/dmesg.log
	fi
(( $? == 0 )) && stat_done || stat_fail

run_hook sysinit_end

# End of file
# vim: set ts=2 sw=2 noet:

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Ted Ts'o]

On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
> 
> /proc/mounts (as you can see it's not being remounted rw for some
> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
> 
> rootfs / rootfs rw 0 0
> /dev/root / ext4 ro,relatime,barrier=1 0 0

Can you send me the output of /proc/cmdline?  It looks like the
rootflags command line option isn't getting passed to the kernel for
some reason.  That would explain why data=writeback isn't showing up
in /proc/mounts.

						- Ted

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

The output of /proc/cmdline on 3.0.7:

BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4 
rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd


On 10/09/2011 06:44 PM, Ted Ts'o wrote:
> On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>> /proc/mounts (as you can see it's not being remounted rw for some
>> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>>
>> rootfs / rootfs rw 0 0
>> /dev/root / ext4 ro,relatime,barrier=1 0 0
> Can you send me the output of /proc/cmdline?  It looks like the
> rootflags command line option isn't getting passed to the kernel for
> some reason.  That would explain why data=writeback isn't showing up
> in /proc/mounts.
>
> 						- Ted

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

Also, note that the image is named vmlinuz26-zen there by mistake - I 
fixed that in my build script, but even so, it makes no difference. In 
3.0.7, I still can't remount root rw on ext4.

On 10/20/2011 10:29 PM, Matt Parnell wrote:
> The output of /proc/cmdline on 3.0.7:
>
> BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4 
> rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd
>
>
> On 10/09/2011 06:44 PM, Ted Ts'o wrote:
>> On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>>> /proc/mounts (as you can see it's not being remounted rw for some
>>> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>>>
>>> rootfs / rootfs rw 0 0
>>> /dev/root / ext4 ro,relatime,barrier=1 0 0
>> Can you send me the output of /proc/cmdline?  It looks like the
>> rootflags command line option isn't getting passed to the kernel for
>> some reason.  That would explain why data=writeback isn't showing up
>> in /proc/mounts.
>>
>>                         - Ted

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Ted Ts'o]

On Thu, Oct 20, 2011 at 10:29:49PM -0500, Matt Parnell wrote:
> The output of /proc/cmdline on 3.0.7:
> 
> BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4
> rootflags=data=writeback noatime nodiratime barrier=0 discard
> noinitrd
> 
> 
> On 10/09/2011 06:44 PM, Ted Ts'o wrote:
> >On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
> >>/proc/mounts (as you can see it's not being remounted rw for some
> >>reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
> >>
> >>rootfs / rootfs rw 0 0
> >>/dev/root / ext4 ro,relatime,barrier=1 0 0

I haven't had a chance to check on v3.0 (I'm currently travelling and
penning this from an airplane), but on v3.1-rc3, it works for me under
KVM.  I fire up kvm-qemu this way:

$QEMU -enable-kvm -boot order=c $NET \
      -drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
      -drive file=$VDB,cache=none,if=virtio \
      -drive file=$VDC,cache=none,if=virtio \
      -drive file=$VDD,cache=none,if=virtio \
      -nographic -smp $NR_CPU -m $MEM \
      --kernel $KERNEL \
      --append "root=/dev/vda console=ttyS0,115200 rootflags=data=writeback" |\
tee $LOGFILE

with this in /proc/cmdline:

candygram:~# cat /proc/cmdline 
root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint

... and this is what I have in /proc/mounts:

candygram:~# head -2 /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0

... and what I have in /etc/fstab:

UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02	/	ext4    noatime,data=writeback 0 1

The kvm image I'm using is using Debian unstable image generated using
debootstrap, and uses no modules or initrd's --- which makes it easier
for me to boot since $KERNEL is set to:

    /tyt/linux/ext4/arch/x86/boot/bzImage

This allows me to fire up test kernel right from my build tree,
without needing to set up any kind of initrd nonsense.  :-)

		       	      	      	     		- Ted

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

That doesn't really help me at all, it's not rootflags=data=writeback 
causing this, it's starting to make me think that arch's init may be to 
blame, although I previously ruled it out...

This is all a big wtf though. I'm going to go complain at the arch devs 
now... I've not finished porting openrc to it yet.

On 10/22/2011 12:00 AM, Ted Ts'o wrote:
> On Thu, Oct 20, 2011 at 10:29:49PM -0500, Matt Parnell wrote:
>> The output of /proc/cmdline on 3.0.7:
>>
>> BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4
>> rootflags=data=writeback noatime nodiratime barrier=0 discard
>> noinitrd
>>
>>
>> On 10/09/2011 06:44 PM, Ted Ts'o wrote:
>>> On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>>>> /proc/mounts (as you can see it's not being remounted rw for some
>>>> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>>>>
>>>> rootfs / rootfs rw 0 0
>>>> /dev/root / ext4 ro,relatime,barrier=1 0 0
> I haven't had a chance to check on v3.0 (I'm currently travelling and
> penning this from an airplane), but on v3.1-rc3, it works for me under
> KVM.  I fire up kvm-qemu this way:
>
> $QEMU -enable-kvm -boot order=c $NET \
>        -drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
>        -drive file=$VDB,cache=none,if=virtio \
>        -drive file=$VDC,cache=none,if=virtio \
>        -drive file=$VDD,cache=none,if=virtio \
>        -nographic -smp $NR_CPU -m $MEM \
>        --kernel $KERNEL \
>        --append "root=/dev/vda console=ttyS0,115200 rootflags=data=writeback" |\
> tee $LOGFILE
>
> with this in /proc/cmdline:
>
> candygram:~# cat /proc/cmdline
> root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint
>
> ... and this is what I have in /proc/mounts:
>
> candygram:~# head -2 /proc/mounts
> rootfs / rootfs rw 0 0
> /dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0
>
> ... and what I have in /etc/fstab:
>
> UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02	/	ext4    noatime,data=writeback 0 1
>
> The kvm image I'm using is using Debian unstable image generated using
> debootstrap, and uses no modules or initrd's --- which makes it easier
> for me to boot since $KERNEL is set to:
>
>      /tyt/linux/ext4/arch/x86/boot/bzImage
>
> This allows me to fire up test kernel right from my build tree,
> without needing to set up any kind of initrd nonsense.  :-)
>
> 		       	      	      	     		- Ted

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

Also, I didn't intend to come across cross there... I appreciate the 
efforts you are taking there.

It's just, I don't know what could've changed between .29 and 3.0 that 
would break an entire init system so that it won't remount root rw...

On 10/22/2011 02:51 AM, Matt Parnell wrote:
> That doesn't really help me at all, it's not rootflags=data=writeback 
> causing this, it's starting to make me think that arch's init may be 
> to blame, although I previously ruled it out...
>
> This is all a big wtf though. I'm going to go complain at the arch 
> devs now... I've not finished porting openrc to it yet.
>
> On 10/22/2011 12:00 AM, Ted Ts'o wrote:
>> On Thu, Oct 20, 2011 at 10:29:49PM -0500, Matt Parnell wrote:
>>> The output of /proc/cmdline on 3.0.7:
>>>
>>> BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4
>>> rootflags=data=writeback noatime nodiratime barrier=0 discard
>>> noinitrd
>>>
>>>
>>> On 10/09/2011 06:44 PM, Ted Ts'o wrote:
>>>> On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>>>>> /proc/mounts (as you can see it's not being remounted rw for some
>>>>> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>>>>>
>>>>> rootfs / rootfs rw 0 0
>>>>> /dev/root / ext4 ro,relatime,barrier=1 0 0
>> I haven't had a chance to check on v3.0 (I'm currently travelling and
>> penning this from an airplane), but on v3.1-rc3, it works for me under
>> KVM.  I fire up kvm-qemu this way:
>>
>> $QEMU -enable-kvm -boot order=c $NET \
>>        -drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
>>        -drive file=$VDB,cache=none,if=virtio \
>>        -drive file=$VDC,cache=none,if=virtio \
>>        -drive file=$VDD,cache=none,if=virtio \
>>        -nographic -smp $NR_CPU -m $MEM \
>>        --kernel $KERNEL \
>>        --append "root=/dev/vda console=ttyS0,115200 
>> rootflags=data=writeback" |\
>> tee $LOGFILE
>>
>> with this in /proc/cmdline:
>>
>> candygram:~# cat /proc/cmdline
>> root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint
>>
>> ... and this is what I have in /proc/mounts:
>>
>> candygram:~# head -2 /proc/mounts
>> rootfs / rootfs rw 0 0
>> /dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0
>>
>> ... and what I have in /etc/fstab:
>>
>> UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02    /    ext4    
>> noatime,data=writeback 0 1
>>
>> The kvm image I'm using is using Debian unstable image generated using
>> debootstrap, and uses no modules or initrd's --- which makes it easier
>> for me to boot since $KERNEL is set to:
>>
>>      /tyt/linux/ext4/arch/x86/boot/bzImage
>>
>> This allows me to fire up test kernel right from my build tree,
>> without needing to set up any kind of initrd nonsense.  :-)
>>
>>                                                     - Ted

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Ted Ts'o]

On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
> That doesn't really help me at all, it's not
> rootflags=data=writeback causing this, it's starting to make me
> think that arch's init may be to blame, although I previously ruled
> it out...

Well, it looks like rootflags=data=writeback is not making it to the
file system.  That's why it's not showing up in /proc/mounts, from you
showed us.  Can you look at the kernel dmesg?

You should see something like this:

[    1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
[    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
[    1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
[    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.


The first line is the failure to mount the root file system as ext3.
The 2nd is the failure to mount the file system as ext2, using the
ext4 file system driver.  The last two lines show the options show the
mount as ext4.

What do those two lines look to you.  If you don't see "Opts:
data=writeback", then somehow the rootflags option isn't getting passed
down to the file system.  Then when you try to remount the file system
read/write, the fact that you have "data=writeback" in your /etc/fstab
causes the failure to remount.

If you simply remove that from /etc/fstab, things should work better.
The remount will preserve whatever data=journalling mode was in use
when the root file system was originallymounted as.  If rootflags is
non-functional, then the file system won't be mounted as
data=writeback, but at least the boot sequence will continue without
blowing out.

							- Ted

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

On 10/22/2011 02:32 AM, Ted Ts'o wrote:
> On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
>> That doesn't really help me at all, it's not
>> rootflags=data=writeback causing this, it's starting to make me
>> think that arch's init may be to blame, although I previously ruled
>> it out...
> Well, it looks like rootflags=data=writeback is not making it to the
> file system.  That's why it's not showing up in /proc/mounts, from you
> showed us.  Can you look at the kernel dmesg?
>
> You should see something like this:
>
> [    1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
> [    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
> [    1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
> [    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
>
>
> The first line is the failure to mount the root file system as ext3.
> The 2nd is the failure to mount the file system as ext2, using the
> ext4 file system driver.  The last two lines show the options show the
> mount as ext4.
>
> What do those two lines look to you.  If you don't see "Opts:
> data=writeback", then somehow the rootflags option isn't getting passed
> down to the file system.  Then when you try to remount the file system
> read/write, the fact that you have "data=writeback" in your /etc/fstab
> causes the failure to remount.
>
> If you simply remove that from /etc/fstab, things should work better.
> The remount will preserve whatever data=journalling mode was in use
> when the root file system was originallymounted as.  If rootflags is
> non-functional, then the file system won't be mounted as
> data=writeback, but at least the boot sequence will continue without
> blowing out.
>
> 							- Ted

Since then, I created a fresh partition and re rsynced my root over onto 
it after making it WITHOUT a journal using the current sysrescue64 cd 
(kernel 3.0.1).

Same thing is happening, and the drive still refuses to mount any other 
way except

  /dev/root / ext4 rw,relatime,user_xattr,acl,barrier=1 0 0

Despite my specifying these in fstab and grub.cfg ...it's like it's not 
even passing the options somewhere, either from grub2 to the kernel or 
from my initscripts...but since I've tried using openrc too and get this 
result, I'm not sure what to do. Note that using both /dev/sdb1 and 
/dev/root don't interchange between the fstab and the kernel...I think 
the /dev/root symlink may be breaking my initscripts.

/dev/sdb1 / ext4 
noatime,nodiratime,data=writeback,nouser_xattr,barrier=0,commit=10 0 0

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

On 10/22/2011 02:32 AM, Ted Ts'o wrote:
> On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
>> That doesn't really help me at all, it's not
>> rootflags=data=writeback causing this, it's starting to make me
>> think that arch's init may be to blame, although I previously ruled
>> it out...
> Well, it looks like rootflags=data=writeback is not making it to the
> file system.  That's why it's not showing up in /proc/mounts, from you
> showed us.  Can you look at the kernel dmesg?
>
> You should see something like this:
>
> [    1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
> [    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
> [    1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
> [    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
>
>
> The first line is the failure to mount the root file system as ext3.
> The 2nd is the failure to mount the file system as ext2, using the
> ext4 file system driver.  The last two lines show the options show the
> mount as ext4.
>
> What do those two lines look to you.  If you don't see "Opts:
> data=writeback", then somehow the rootflags option isn't getting passed
> down to the file system.  Then when you try to remount the file system
> read/write, the fact that you have "data=writeback" in your /etc/fstab
> causes the failure to remount.
>
> If you simply remove that from /etc/fstab, things should work better.
> The remount will preserve whatever data=journalling mode was in use
> when the root file system was originallymounted as.  If rootflags is
> non-functional, then the file system won't be mounted as
> data=writeback, but at least the boot sequence will continue without
> blowing out.
>
> 							- Ted
Looks like the kernel isn't creating the /dev/root link/block device, 
either all of the time or some of the time, I'm confused.

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

On 11/06/2011 06:24 PM, Matt Parnell wrote:
> On 10/22/2011 02:32 AM, Ted Ts'o wrote:
>> On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
>>> That doesn't really help me at all, it's not
>>> rootflags=data=writeback causing this, it's starting to make me
>>> think that arch's init may be to blame, although I previously ruled
>>> it out...
>> Well, it looks like rootflags=data=writeback is not making it to the
>> file system.  That's why it's not showing up in /proc/mounts, from you
>> showed us.  Can you look at the kernel dmesg?
>>
>> You should see something like this:
>>
>> [    1.421146] EXT3-fs (vda): error: couldn't mount because of 
>> unsupported optional features (240)
>> [    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature 
>> incompatibilities
>> [    1.454631] EXT4-fs (vda): mounted filesystem with writeback data 
>> mode. Opts: data=writeback
>> [    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 
>> 254:0.
>>
>>
>> The first line is the failure to mount the root file system as ext3.
>> The 2nd is the failure to mount the file system as ext2, using the
>> ext4 file system driver.  The last two lines show the options show the
>> mount as ext4.
>>
>> What do those two lines look to you.  If you don't see "Opts:
>> data=writeback", then somehow the rootflags option isn't getting passed
>> down to the file system.  Then when you try to remount the file system
>> read/write, the fact that you have "data=writeback" in your /etc/fstab
>> causes the failure to remount.
>>
>> If you simply remove that from /etc/fstab, things should work better.
>> The remount will preserve whatever data=journalling mode was in use
>> when the root file system was originallymounted as.  If rootflags is
>> non-functional, then the file system won't be mounted as
>> data=writeback, but at least the boot sequence will continue without
>> blowing out.
>>
>>                             - Ted
> Looks like the kernel isn't creating the /dev/root link/block device, 
> either all of the time or some of the time, I'm confused.
I figured it out. data=writeback isn't needed and is done by default if 
there's no journal, as per commit 
373cd5c53d5ea6622c319ecd84e29e2737d488bd 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd

...sorry about throwing you guys through loops over this, although I'm 
guessing there's still a bug in that the kernel should just warn the 
user about the option when it's used and otherwise ignore it instead of 
breaking if there's no journal.

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

On 11/06/2011 09:26 PM, Matt Parnell wrote:
> On 11/06/2011 06:24 PM, Matt Parnell wrote:
>> On 10/22/2011 02:32 AM, Ted Ts'o wrote:
>>> On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
>>>> That doesn't really help me at all, it's not
>>>> rootflags=data=writeback causing this, it's starting to make me
>>>> think that arch's init may be to blame, although I previously ruled
>>>> it out...
>>> Well, it looks like rootflags=data=writeback is not making it to the
>>> file system.  That's why it's not showing up in /proc/mounts, from you
>>> showed us.  Can you look at the kernel dmesg?
>>>
>>> You should see something like this:
>>>
>>> [    1.421146] EXT3-fs (vda): error: couldn't mount because of 
>>> unsupported optional features (240)
>>> [    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature 
>>> incompatibilities
>>> [    1.454631] EXT4-fs (vda): mounted filesystem with writeback data 
>>> mode. Opts: data=writeback
>>> [    1.455966] VFS: Mounted root (ext4 filesystem) readonly on 
>>> device 254:0.
>>>
>>>
>>> The first line is the failure to mount the root file system as ext3.
>>> The 2nd is the failure to mount the file system as ext2, using the
>>> ext4 file system driver.  The last two lines show the options show the
>>> mount as ext4.
>>>
>>> What do those two lines look to you.  If you don't see "Opts:
>>> data=writeback", then somehow the rootflags option isn't getting passed
>>> down to the file system.  Then when you try to remount the file system
>>> read/write, the fact that you have "data=writeback" in your /etc/fstab
>>> causes the failure to remount.
>>>
>>> If you simply remove that from /etc/fstab, things should work better.
>>> The remount will preserve whatever data=journalling mode was in use
>>> when the root file system was originallymounted as.  If rootflags is
>>> non-functional, then the file system won't be mounted as
>>> data=writeback, but at least the boot sequence will continue without
>>> blowing out.
>>>
>>>                             - Ted
>> Looks like the kernel isn't creating the /dev/root link/block device, 
>> either all of the time or some of the time, I'm confused.
> I figured it out. data=writeback isn't needed and is done by default 
> if there's no journal, as per commit 
> 373cd5c53d5ea6622c319ecd84e29e2737d488bd 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
>
> ...sorry about throwing you guys through loops over this, although I'm 
> guessing there's still a bug in that the kernel should just warn the 
> user about the option when it's used and otherwise ignore it instead 
> of breaking if there's no journal.
That is, you were right in removing rootflags=data=writeback from the 
kernel boot params, and removing data=writeback from fstab would fix it 
- that commit is where things break, though...

So really instead of breaking it should log that it's ignoring that flag 
becuase it's enabled by default when there's no journal.

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Ted Ts'o]

On Sun, Nov 06, 2011 at 09:26:57PM -0800, Matt Parnell wrote:
> I figured it out. data=writeback isn't needed and is done by default
> if there's no journal, as per commit
> 373cd5c53d5ea6622c319ecd84e29e2737d488bd http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
> 
> ...sorry about throwing you guys through loops over this, although
> I'm guessing there's still a bug in that the kernel should just warn
> the user about the option when it's used and otherwise ignore it
> instead of breaking if there's no journal.

Ah, OK.  It wasn't clear to me that you were running without a year.
You're right, we shouldn't complain in that case, since we ignore the
messages when mounting the file system.  Actually, we parse the
options when we are mounting the file system, but then we zero them
out when we discover that there's no journal; but in any case, we
should similarly ignore the pointless data=* mount options (for a file
system w/o a journal) on the remount operation as well.

     	     	      	     	     	       	  - Ted

commit eb513689c97e3e73bb9b4459d490a8e894b4a546
Author: Theodore Ts'o <tytso@mit.edu>
Date:   Mon Nov 7 10:47:42 2011 -0500

    ext4: ignore journalled data options on remount if fs has no journal
    
    This avoids a confusing failure in the init scripts when the
    /etc/fstab has data=writeback or data=journal but the file system does
    not have a journal.  So check for this case explicitly, and warn the
    user that we are ignoring the (pointless, since they have no journal)
    data=* mount option.
    
    Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>

diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 9953d80..0435013 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -1683,7 +1683,9 @@ static int parse_options(char *options, struct super_block *sb,
 			data_opt = EXT4_MOUNT_WRITEBACK_DATA;
 		datacheck:
 			if (is_remount) {
-				if (test_opt(sb, DATA_FLAGS) != data_opt) {
+				if (!sbi->s_journal)
+					ext4_msg(sb, KERN_WARNING, "Remounting file system with no journal so ignoring journalled data option");
+				else if (test_opt(sb, DATA_FLAGS) != data_opt) {
 					ext4_msg(sb, KERN_ERR,
 						"Cannot change data mode on remount");
 					return 0;

Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

[Matt Parnell]

On 11/07/2011 07:47 AM, Ted Ts'o wrote:
> On Sun, Nov 06, 2011 at 09:26:57PM -0800, Matt Parnell wrote:
>> I figured it out. data=writeback isn't needed and is done by default
>> if there's no journal, as per commit
>> 373cd5c53d5ea6622c319ecd84e29e2737d488bd http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
>>
>> ...sorry about throwing you guys through loops over this, although
>> I'm guessing there's still a bug in that the kernel should just warn
>> the user about the option when it's used and otherwise ignore it
>> instead of breaking if there's no journal.
> Ah, OK.  It wasn't clear to me that you were running without a year.
> You're right, we shouldn't complain in that case, since we ignore the
> messages when mounting the file system.  Actually, we parse the
> options when we are mounting the file system, but then we zero them
> out when we discover that there's no journal; but in any case, we
> should similarly ignore the pointless data=* mount options (for a file
> system w/o a journal) on the remount operation as well.
>
>       	     	      	     	     	       	  - Ted
>
> commit eb513689c97e3e73bb9b4459d490a8e894b4a546
> Author: Theodore Ts'o<tytso@mit.edu>
> Date:   Mon Nov 7 10:47:42 2011 -0500
>
>      ext4: ignore journalled data options on remount if fs has no journal
>
>      This avoids a confusing failure in the init scripts when the
>      /etc/fstab has data=writeback or data=journal but the file system does
>      not have a journal.  So check for this case explicitly, and warn the
>      user that we are ignoring the (pointless, since they have no journal)
>      data=* mount option.
>
>      Signed-off-by: "Theodore Ts'o"<tytso@mit.edu>
>
> diff --git a/fs/ext4/super.c b/fs/ext4/super.c
> index 9953d80..0435013 100644
> --- a/fs/ext4/super.c
> +++ b/fs/ext4/super.c
> @@ -1683,7 +1683,9 @@ static int parse_options(char *options, struct super_block *sb,
>   			data_opt = EXT4_MOUNT_WRITEBACK_DATA;
>   		datacheck:
>   			if (is_remount) {
> -				if (test_opt(sb, DATA_FLAGS) != data_opt) {
> +				if (!sbi->s_journal)
> +					ext4_msg(sb, KERN_WARNING, "Remounting file system with no journal so ignoring journalled data option");
> +				else if (test_opt(sb, DATA_FLAGS) != data_opt) {
>   					ext4_msg(sb, KERN_ERR,
>   						"Cannot change data mode on remount");
>   					return 0;
Great! I'm glad we managed to track this one down. I'm not using the 
flags anymore, but I'll be glad when your patch there makes it into ext4 
for other users.