* random i/o error without error in dmesg
@ 2015-10-26 11:23 Szalma László
2015-10-26 14:23 ` Marc Joliet
0 siblings, 1 reply; 16+ messages in thread
From: Szalma László @ 2015-10-26 11:23 UTC (permalink / raw)
To: linux-btrfs
Hi,
I have this error for a time, It's not easy to reproduce, i write
everything i know at the moment.
I maintain some servers running xen (4.5.1) and gentoo dom0 with recent
kernels (3.18.*, 4.1.6, 4.2.3, 4.2.4). I use gentoo-sources patchset.
Running xen domu s, for www and mysql.
I have mysql servers in domu with high load (lots of read write). These
systems are identical in term of configuration and kernel.
Sometimes I got mysql errors randomly (sometimes more than one at a day,
sometimes one at a week), but it is more frequent on high load.
The mysql errors are because the file cannot be read from the
filesystem. If i try to run md5sum on it it shows io error.
At this point mysql stop && umount && mount && mysql start solves the
problem.
calling
echo 3 > /proc/sys/vm/drop_caches
sometimes solves the io error, but not every time. The problem rarely
randomly fixed without remount.
The problem seems to have no connection to the dom0 kernel and the xen
version. I have this problem for example on these dom0 -s:
kernel: 3.19.3 xen 4.5.0
kernel: 4.2.3 xen 4.5.1
The problem seems to have started with the kernel 4.0 series, but I am
not sure. In the summer the load was low, and the problem occured very
rarely.
In this case of io error:
btrfs scrub finds no error.
no memory or hdd/ssd hardware error (smart, memtest, etc) (not only one
physical server is affected) and no errors in dmesg at all.
tried different kernel configs, but I don't think I have anything
extraordinary.
I use deadline scheduler.
I use these mount options:
/dev/xvdb1 on /mnt/mysql_naplo_b2 type btrfs
(rw,noatime,compress=zlib,nossd,noacl,space_cache,subvolid=5,subvol=/)
I tried to reformat the filesystem with recent btrfs-progs: (and olders
before)
btrfs-progs v4.2.2
I use default mkfs options (skinny extents)
After format the problem was disappeared for some days. (it seems
correlation with the age of the filesystem?)
I do manual defragment on the filesystem with a script simply
recursively check "filefrag" for count the fragmentation and defrag if
it is more than 50 and the file is larger than 64kbyte. (this sometimes
lowers the frequency of the problem)
The files unreadable are usually small files, for example:
filefrag:
/mnt/mysql_naplo_b2/mozanaplo_boly_altisk_2015/n_helyettesites.MYD: 2
extents found
ls -l:
-rw-rw---- 1 mysql mysql 8092 okt 22 08.24
/mnt/mysql_naplo_b2/mozanaplo_boly_altisk_2015/n_helyettesites.MYD
There is no error in dmesg, no io errors, no kernel panic, etc at all.
The (virtual) servers has 3-4GB of memory, and I use a 2GB tmpfs for the
temporary tables (this way the physical memory usage is somewhat hectic).
The filesystem has no snapshots, but sometimes (for rebuilding
replication) I take on, and delete it. (but the problem happens on
filesystems with no snapshot created ever)
I did not try downgrading the kernel (for 3.18), but I always try to
upgrade.
I guess this problem has some connection to the memory usage (but there
is no out of memory).
I am able to try any debug mode if you suggest one, but it's not
reproducable, it happens randomly. I think there should be some errors
in the dmesg if I encounter io errors, but I am not sure if this error
has direct connection for btrfs at all. I didn't try other filesystems.
The problem was occured with kernel versions: 4.0.1, 4.0.4, 4.1.6,
4.2.1, 4.2.3, 4.2.4.
I checked the bugzilla, and google for similar problem, but I couldn't
find any similar.
This problem sometimes (i think it is the same) happen on a www server
too, with apache log files (they are fragmented heavily), but very
rarely. I don't have any problem with this configuration on other
servers even mysql servers with lower load.
I welcome any suggestion:
László Szalma
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-26 11:23 random i/o error without error in dmesg Szalma László
@ 2015-10-26 14:23 ` Marc Joliet
2015-10-27 6:23 ` Duncan
2016-04-22 13:17 ` Marc Joliet
0 siblings, 2 replies; 16+ messages in thread
From: Marc Joliet @ 2015-10-26 14:23 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 3398 bytes --]
Hi
FWIW, this sounds like what I've been seeing with dovecot. In case it's
relevant, I'll try to explain.
After some uptime, I'll see log messages like this:
Okt 26 12:05:46 thetick dovecot[467]: imap(marcec): Error: pread() failed with
file /home/marcec/.mdbox/mailboxes/BTRFS/dbox-Mails/dovecot.index.log:
Input/output error
Occasionally they go away by themselves, but usually I have to reboot to make
them go away. This happens when getmail attempts to fetch mail, which fails
due to the above error. After the reboot getmail succeeds again.
As in Szalma's case, btrfs-scrub never reports anything wrong.
I use LZO compression on the relevant file system, so I wanted to wait until
kernel 4.1.11 before reporting this, but that hasn't hit Gentoo yet (and
neither has 4.1.10, for some reason). I don't use quotas.
According to the what I see in the systemd journal, the errors started on
2015-06-01 with kernel 3.19.8. Note that, strangely enough, I had been using
that same version since 2015-05-23, so for more than a week before the error
cropped up. I checked whether I made any changes to the configuration, and
found this:
diff --git a/kernels/kernel-config-3.19.8-gentoo b/kernels/kernel-
config-3.19.8-gentoo
index b061b31..8cf8eba 100644
--- a/kernels/kernel-config-3.19.8-gentoo
+++ b/kernels/kernel-config-3.19.8-gentoo
@@ -64,7 +64,7 @@ CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
-CONFIG_LOCALVERSION_AUTO=y
+# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
@@ -73,8 +73,8 @@ CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
# CONFIG_KERNEL_GZIP is not set
# CONFIG_KERNEL_BZIP2 is not set
-CONFIG_KERNEL_LZMA=y
-# CONFIG_KERNEL_XZ is not set
+# CONFIG_KERNEL_LZMA is not set
+CONFIG_KERNEL_XZ=y
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
@@ -132,7 +132,7 @@ CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_IRQ_TIME_ACCOUNTING is not set
CONFIG_BSD_PROCESS_ACCT=y
-# CONFIG_BSD_PROCESS_ACCT_V3 is not set
+CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
The only change I can think of that might affect anything is
CONFIG_BSD_PROCESS_ACCT_V3=y (I don't remember why exactly I set it). I can
try without it set, but maybe the kernel configuration is a red herring?
Anyway, the current state of the system is:
# uname -r
4.1.9-gentoo-r1
# btrfs filesystem show /
Label: 'MARCEC_ROOT' uuid: 0267d8b3-a074-460a-832d-5d5fd36bae64
Total devices 1 FS bytes used 74.40GiB
devid 1 size 107.79GiB used 105.97GiB path /dev/sda1
btrfs-progs v4.2.2
# btrfs filesystem df /
Data, single: total=98.94GiB, used=72.30GiB
System, single: total=32.00MiB, used=20.00KiB
Metadata, single: total=7.00GiB, used=2.10GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
The filesystem is mounted as (leaving out subvolume mounts which use the same
mount options):
/dev/sda1 on / type btrfs (rw,noatime,compress=lzo,ssd,discard,space_cache)
Greetings,
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-26 14:23 ` Marc Joliet
@ 2015-10-27 6:23 ` Duncan
2015-10-27 9:19 ` Marc Joliet
` (3 more replies)
2016-04-22 13:17 ` Marc Joliet
1 sibling, 4 replies; 16+ messages in thread
From: Duncan @ 2015-10-27 6:23 UTC (permalink / raw)
To: linux-btrfs
Marc Joliet posted on Mon, 26 Oct 2015 15:23:39 +0100 as excerpted:
> Occasionally they go away by themselves, but usually I have to reboot to
> make them go away. This happens when getmail attempts to fetch mail,
> which fails due to the above error. After the reboot getmail succeeds
> again.
Just out of curiosity, does a remount,ro, followed by a remount,rw, solve
the problem?
The ro/rw cycle should force anything in memory to device, so if that
eliminates the problem, it could well be some sort of sync issue. If it
doesn't, then it's more likely an in-memory filesystem state issue,
that's cleared by the reboot.
And if the ro/rw cycle doesn't clear the problem, what about a full
unmount/mount cycle, at least of that subvolume?
If you're running multiple subvolumes with root being one of them, you
can't of course unmount the entire filesystem, but you could go down to
emergency mode (systemctl emergency), try unmounting everything but /,
mounting / ro, and then switching back to normal mode (from emergency
mode, simply exiting should return you to normal multi-user or gui
target, remounting filesystems as necessary, etc).
IOW, does it take a full reboot to clear the problem, or is a simple ro/rw
mount cycle enough, or an unmount/remount?
Finally, assuming root itself isn't btrfs, if you have btrfs configured
as a module, you could try unmounting all btrfs and then unloading the
module, then reloading and remounting. That should entirely clear all in-
memory btrfs state, so if that doesn't solve the problem, while rebooting
does, then the problem's very possibly outside of btrfs scope. Of course
if root is btrfs, you can't really check that.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-27 6:23 ` Duncan
@ 2015-10-27 9:19 ` Marc Joliet
2015-10-27 14:57 ` Szalma László
` (2 subsequent siblings)
3 siblings, 0 replies; 16+ messages in thread
From: Marc Joliet @ 2015-10-27 9:19 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2126 bytes --]
On Tuesday 27 October 2015 06:23:06 Duncan wrote:
>Marc Joliet posted on Mon, 26 Oct 2015 15:23:39 +0100 as excerpted:
>> Occasionally they go away by themselves, but usually I have to reboot to
>> make them go away. This happens when getmail attempts to fetch mail,
>> which fails due to the above error. After the reboot getmail succeeds
>> again.
>
>Just out of curiosity, does a remount,ro, followed by a remount,rw, solve
>the problem?
>
>The ro/rw cycle should force anything in memory to device, so if that
>eliminates the problem, it could well be some sort of sync issue. If it
>doesn't, then it's more likely an in-memory filesystem state issue,
>that's cleared by the reboot.
>
>And if the ro/rw cycle doesn't clear the problem, what about a full
>unmount/mount cycle, at least of that subvolume?
>
>If you're running multiple subvolumes with root being one of them, you
>can't of course unmount the entire filesystem, but you could go down to
>emergency mode (systemctl emergency), try unmounting everything but /,
>mounting / ro, and then switching back to normal mode (from emergency
>mode, simply exiting should return you to normal multi-user or gui
>target, remounting filesystems as necessary, etc).
>
>IOW, does it take a full reboot to clear the problem, or is a simple ro/rw
>mount cycle enough, or an unmount/remount?
>
>Finally, assuming root itself isn't btrfs, if you have btrfs configured
>as a module, you could try unmounting all btrfs and then unloading the
>module, then reloading and remounting. That should entirely clear all in-
>memory btrfs state, so if that doesn't solve the problem, while rebooting
>does, then the problem's very possibly outside of btrfs scope. Of course
>if root is btrfs, you can't really check that.
Thanks for the hints. I just upgraded to gentoo-sources 4.1.11 and will see
if that changes anything. If not, I'll have to try unmounting /home from
emergency mode (it's a subvolume mount).
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-27 6:23 ` Duncan
2015-10-27 9:19 ` Marc Joliet
@ 2015-10-27 14:57 ` Szalma László
2015-10-27 20:54 ` Marc Joliet
2015-10-28 8:44 ` Szalma László
3 siblings, 0 replies; 16+ messages in thread
From: Szalma László @ 2015-10-27 14:57 UTC (permalink / raw)
To: linux-btrfs
I wait for the problem to occur, i will try the remount,ro remount,rw.
But I know for sure, in my case umount / mount solved the problem every
time!
Reboot was not needed. (I think reboot is simpler because the fs is
used, and the service has to be stopped etc.)
László Szalma
2015-10-27 07:23 keltezéssel, Duncan írta:
> Marc Joliet posted on Mon, 26 Oct 2015 15:23:39 +0100 as excerpted:
>
>> Occasionally they go away by themselves, but usually I have to reboot to
>> make them go away. This happens when getmail attempts to fetch mail,
>> which fails due to the above error. After the reboot getmail succeeds
>> again.
> Just out of curiosity, does a remount,ro, followed by a remount,rw, solve
> the problem?
>
> The ro/rw cycle should force anything in memory to device, so if that
> eliminates the problem, it could well be some sort of sync issue. If it
> doesn't, then it's more likely an in-memory filesystem state issue,
> that's cleared by the reboot.
>
> And if the ro/rw cycle doesn't clear the problem, what about a full
> unmount/mount cycle, at least of that subvolume?
>
> If you're running multiple subvolumes with root being one of them, you
> can't of course unmount the entire filesystem, but you could go down to
> emergency mode (systemctl emergency), try unmounting everything but /,
> mounting / ro, and then switching back to normal mode (from emergency
> mode, simply exiting should return you to normal multi-user or gui
> target, remounting filesystems as necessary, etc).
>
> IOW, does it take a full reboot to clear the problem, or is a simple ro/rw
> mount cycle enough, or an unmount/remount?
>
> Finally, assuming root itself isn't btrfs, if you have btrfs configured
> as a module, you could try unmounting all btrfs and then unloading the
> module, then reloading and remounting. That should entirely clear all in-
> memory btrfs state, so if that doesn't solve the problem, while rebooting
> does, then the problem's very possibly outside of btrfs scope. Of course
> if root is btrfs, you can't really check that.
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-27 6:23 ` Duncan
2015-10-27 9:19 ` Marc Joliet
2015-10-27 14:57 ` Szalma László
@ 2015-10-27 20:54 ` Marc Joliet
2015-10-28 5:21 ` Duncan
2015-10-28 8:44 ` Szalma László
3 siblings, 1 reply; 16+ messages in thread
From: Marc Joliet @ 2015-10-27 20:54 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2737 bytes --]
OK, upgrading to gentoo-sources 4.1.11 didn't help, so I tried these steps.
More inline below.
On Tuesday 27 October 2015 06:23:06 Duncan wrote:
>Marc Joliet posted on Mon, 26 Oct 2015 15:23:39 +0100 as excerpted:
>> Occasionally they go away by themselves, but usually I have to reboot to
>> make them go away. This happens when getmail attempts to fetch mail,
>> which fails due to the above error. After the reboot getmail succeeds
>> again.
>
>Just out of curiosity, does a remount,ro, followed by a remount,rw, solve
>the problem?
>
>The ro/rw cycle should force anything in memory to device, so if that
>eliminates the problem, it could well be some sort of sync issue. If it
>doesn't, then it's more likely an in-memory filesystem state issue,
>that's cleared by the reboot.
Didn't try this directly, but...
>And if the ro/rw cycle doesn't clear the problem, what about a full
>unmount/mount cycle, at least of that subvolume?
...this didn't work, after which...
>If you're running multiple subvolumes with root being one of them, you
>can't of course unmount the entire filesystem, but you could go down to
>emergency mode (systemctl emergency), try unmounting everything but /,
>mounting / ro, and then switching back to normal mode (from emergency
>mode, simply exiting should return you to normal multi-user or gui
>target, remounting filesystems as necessary, etc).
...I tried most of this. I unmounted everything except for /var and /,
neither of which I could mount read-only. It didn't help, either.
>IOW, does it take a full reboot to clear the problem, or is a simple ro/rw
>mount cycle enough, or an unmount/remount?
Seems that a full reboot is needed, but I would expect that it would have the
same effect if I were to pivot back into the initramfs, unmount / from there,
then boot back into the system. Because quite frankly, I can't think of any
reason why a power cycle to the SSD should make a difference here. I vaguely
remember that systemd can do that, so I'll see if I can find out how.
>Finally, assuming root itself isn't btrfs, if you have btrfs configured
>as a module, you could try unmounting all btrfs and then unloading the
>module, then reloading and remounting. That should entirely clear all in-
>memory btrfs state, so if that doesn't solve the problem, while rebooting
>does, then the problem's very possibly outside of btrfs scope. Of course
>if root is btrfs, you can't really check that.
Nope, btrfs is built-in (though it doesn't have to be, what with me using an
initramfs).
Thanks
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-27 20:54 ` Marc Joliet
@ 2015-10-28 5:21 ` Duncan
2015-10-28 11:23 ` Austin S Hemmelgarn
2015-10-29 21:10 ` Marc Joliet
0 siblings, 2 replies; 16+ messages in thread
From: Duncan @ 2015-10-28 5:21 UTC (permalink / raw)
To: linux-btrfs
Marc Joliet posted on Tue, 27 Oct 2015 21:54:40 +0100 as excerpted:
>>IOW, does it take a full reboot to clear the problem, or is a simple
>>ro/rw mount cycle enough, or an unmount/remount?
>
> Seems that a full reboot is needed, but I would expect that it would
> have the same effect if I were to pivot back into the initramfs, unmount
> / from there,
> then boot back into the system. Because quite frankly, I can't think of
> any reason why a power cycle to the SSD should make a difference here.
> I vaguely remember that systemd can do that, so I'll see if I can find
> out how.
Agree with both the systemd returning to the initr* point (which I
actually had in mind while writing the above but don't remember the
details either, so chose to omit in the interest of limiting the size of
the reply and research necessary to generate it), and the ssd power-cycle
point.
>>Finally, assuming root itself isn't btrfs, if you have btrfs configured
>>as a module, you could try unmounting all btrfs and then unloading the
>>module, then reloading and remounting. That should entirely clear all
>>in-memory btrfs state, so if that doesn't solve the problem, while
>>rebooting does, then the problem's very possibly outside of btrfs scope.
>> Of course if root is btrfs, you can't really check that.
>
> Nope, btrfs is built-in (though it doesn't have to be, what with me
> using an initramfs).
Same here, also gentoo as I guess you know from previous exchanges. But
unfortunately, if your initr* is anything like mine, and your kernel
monolithic as mine, making btrfs a module with a btrfs root isn't the
easy thing it might seem to those who run ordinary distro supplied binary
kernels with pretty much everything modularized, as doing so involves a
whole new set of research on how to get that module properly included in
the initr* and loaded there, as well as installing and building the whole
module-handling infrastructure (modprobe and friends) again, as it's not
actually installed on the system at all at this point, because with the
kernel entirely monolithic, module-handling tools are unnecessary and
thus just another unnecessary package to have to keep building updates
for, if they remain installed.
So I definitely sympathize with the feeling that such a stone is better
left unturned, if overturning it is at all a possibility that can be
avoided, as it is here, this whole exercise being simply one of better
pinning the bug down, not yet actually trying to solve it. And given
that unturned stone, there are certainly easier ways.
And one of those easier ways is investigating that whole systemd return
to initr* idea, since we both remember reading something about it, but
aren't familiar with the details. In addition to addressing the problem
headon if anyone offers a way to do so, that's the path I'd be looking at
right now.
Meanwhile, addressing something in the snipped content, if a mount
refuses to remount read-only, it's normally due to deleted but still open
files. Often, in many cases the overwhelming majority of the time,
that's due to updates that have been done, removing for instance old
library files that are still loaded into running executables. The
generalized solution is to quit or restart the affected executables, thus
releasing the last open references to the otherwise already deleted
library and other files, so the filesystem can finish deleting them.
Once this is done, the filesystem can normally be remounted ro (tho bugs
like the one of this thread may prevent that).
The problem lies in actually figuring out what still running executables
are still holding open file references to otherwise deleted files. /proc/
actually makes this sort of data available in its collection of files for
a running process, as there's one (as the saying goes, finding which one
is an exercise left for the reader, as I could look it up or check, but
so can you if you're /that/ interested) that tells which files the
process has open, and deleted files are clearly marked.
But it's still a hassle to go thru the appropriate /proc/ files for each
process, at least if it's done manually, so various helper tools have
been developed to automate the process. The one I use here is a python-
based script, packaged on gentoo as...
app-admin/lib_users
Run as root, it'll tell you which executables are running that still hold
references to deleted files, so you can restart them. Of course when a
library was updated for security reasons doing this restart is a good
idea from that perspective as well. Some tools actually have a table of
executables and the services they belong to, and will automate the
restart, but of course these sorts of tools tend to be somewhat complex
and distro specific, while lib_users simply tells you the executables and
lets you decide what to do with that information, that being both simpler
and without the distro-specifics of the lists that the fully automated
tools must use.
If you do your updates while an X-based session is running and run
lib_users before shutting it down, you'll often find a whole slew of x-
based executables listed, so much so that I don't even bother until I've
shut down X, these days. That leaves only services and longer running CLI
executables, perhaps including the shell itself, in the list.
If it's only a couple higher-level services listed, it's generally
simplest to just restart them. If it's many services or lower level
services such that restarting them will disturb the services depending on
them[1], from my experience, it's often easier to simply do a systemctl
emergency, perhaps even hitting ctrl-D straight from the resulting root
password prompt, to go immediately back to multi-user mode, as even
without entering the password, the drop to emergency mode will have
restarted all normal services.
What's left in the lib_user list after a dip to emergency mode, whether
you've returned to multi-user mode or not, is often only systemd and
perhaps its journald service, itself. Journald can be restarted manually
in the usual way (systemctl restart journald.service or whatever, I
usually use tab completion so don't pay all that much attention to the
specific name), if necessary, while systemd itself can be "reexecuted"
using the systemctl daemon-reexec (tab-completion again) command.
With a drop out of X mode, a dip to emergency mode and reexecing systemd
itself, at least in my experience with what I'm running here, lib_users
normally reports nothing further, and a remount read-only succeeds.
Of course, if lib_users reports nothing further still holding references
to deleted files, and a remount read-only STILL fails, that's a major
note of trouble and an important finding in itself.
Meanwhile, as explained in the systemd docs (specifically the systemd for
administrators series, IIRC), systemd dropping back to the initr* is
actually its way of automatically doing effectively the same thing we
were using lib_users and all those restarts to do, getting rid of all
possible still running on root executables, including systemd itself, by
reexecing systemd itself back in the initr*, as a way to help eliminate
*everything* running on root, so it can not only be remounted read-only,
but actually unmounted the same as any other filesystem, as userspace is
now actually running from the initr* once again. That's a far *FAR*
safer reboot after upgrade than traditional sysvinit solutions were able
to do. =:^)
So the whole systemd switchroot back to initr* thing happens to be
actually linked in here too. Tho in this case we'd be using it for more,
and indeed, there are systemd docs on how to get the reboot process to
stop in in the initr*[1]. While I don't recall actually seeing anything
about short-circuiting the reboot and restarting from the initr*, it
should certainly be possible to configure systemd and the initr* to allow
it.
And I might have it configured that way here too, to short-circuit the
reboot, were it not for the two facts that I (1) often use a dip to
emergency mode (and reexec of systemd if necessary) for many of the
reasons one might otherwise dip to initr*, and (2) generally track pre-
release kernels, so a lot of the time when I reboot, it's to start a
freshly built kernel, and the dip to initr* wouldn't help with that.
---
[1] Low-level services: Low-level services aren't so much a problem with
systemd, since it generally keeps thinks like sockets available and hooks
them back up to the restarted service, without disturbing the higher
level services that would otherwise need a restart too, at all. But the
shear number of otherwise manual restarts necessary can still make it
easier to simply dip to emergency mode temporarily, before going normal
multi-user again.
[2] Stopping the reboot process back in the initr*, before final
restart: I did actually try to get this working at one point, but due to
some particulars specific to my own rather strange layout, certain initr*
related systemd service file symlinks were orphaned at the time, and it
didn't work as it was designed and documented to work. I've since fixed
those symlink issues, but I've not tried again to stop in the initr* on
the way down, so I've never actually seen it do that, tho if I'm fast
enough and the reboot slow enough, I can at least see the return to initr*
stuff scrolling by now before it blinks out, and it wasn't doing that
before, so I expect I could actually get it to stop in in the initr* on
the way down, now, if I tried, by following the documentation. Anyway,
that's why I'm not offering specific help on that aspect, because it was
broken for me when I tried it, and while I believe it fixed now, I've not
tried it since the fix, so I don't have the actual benefit of experience
to offer.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-27 6:23 ` Duncan
` (2 preceding siblings ...)
2015-10-27 20:54 ` Marc Joliet
@ 2015-10-28 8:44 ` Szalma László
2015-10-28 12:46 ` Duncan
2015-11-02 18:26 ` Szalma László
3 siblings, 2 replies; 16+ messages in thread
From: Szalma László @ 2015-10-28 8:44 UTC (permalink / raw)
To: linux-btrfs
Ok, I had a chance to try some things.
1.: the error
md5sum xyz
md5sum: xyz: Input/output error
(no any errors in dmesg)
2.: mount -o remount,ro /mnt/x
(could not do, it is used)
mysql stop && mount -o remount,ro /mnt/x
problem persists: io error.
mount -o remount,rw /mnt/x
still io error
umount /mnt/x
mount /mnt/x
NO io error, md5sum works!
The umount/mount ALWAYS solved the problem for me, mount -o remount,ro
was tried for the first time, but it was not enought. Reboot was not needed.
(kernel 4.2.4)
László Szalma
2015-10-27 07:23 keltezéssel, Duncan írta:
> Marc Joliet posted on Mon, 26 Oct 2015 15:23:39 +0100 as excerpted:
>
>> Occasionally they go away by themselves, but usually I have to reboot to
>> make them go away. This happens when getmail attempts to fetch mail,
>> which fails due to the above error. After the reboot getmail succeeds
>> again.
> Just out of curiosity, does a remount,ro, followed by a remount,rw, solve
> the problem?
>
> The ro/rw cycle should force anything in memory to device, so if that
> eliminates the problem, it could well be some sort of sync issue. If it
> doesn't, then it's more likely an in-memory filesystem state issue,
> that's cleared by the reboot.
>
> And if the ro/rw cycle doesn't clear the problem, what about a full
> unmount/mount cycle, at least of that subvolume?
>
> If you're running multiple subvolumes with root being one of them, you
> can't of course unmount the entire filesystem, but you could go down to
> emergency mode (systemctl emergency), try unmounting everything but /,
> mounting / ro, and then switching back to normal mode (from emergency
> mode, simply exiting should return you to normal multi-user or gui
> target, remounting filesystems as necessary, etc).
>
> IOW, does it take a full reboot to clear the problem, or is a simple ro/rw
> mount cycle enough, or an unmount/remount?
>
> Finally, assuming root itself isn't btrfs, if you have btrfs configured
> as a module, you could try unmounting all btrfs and then unloading the
> module, then reloading and remounting. That should entirely clear all in-
> memory btrfs state, so if that doesn't solve the problem, while rebooting
> does, then the problem's very possibly outside of btrfs scope. Of course
> if root is btrfs, you can't really check that.
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-28 5:21 ` Duncan
@ 2015-10-28 11:23 ` Austin S Hemmelgarn
2015-10-29 21:10 ` Marc Joliet
1 sibling, 0 replies; 16+ messages in thread
From: Austin S Hemmelgarn @ 2015-10-28 11:23 UTC (permalink / raw)
To: Duncan, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 3685 bytes --]
On 2015-10-28 01:21, Duncan wrote:
> Marc Joliet posted on Tue, 27 Oct 2015 21:54:40 +0100 as excerpted:
>
>>> IOW, does it take a full reboot to clear the problem, or is a simple
>>> ro/rw mount cycle enough, or an unmount/remount?
>>
>> Seems that a full reboot is needed, but I would expect that it would
>> have the same effect if I were to pivot back into the initramfs, unmount
>> / from there,
>> then boot back into the system. Because quite frankly, I can't think of
>> any reason why a power cycle to the SSD should make a difference here.
>> I vaguely remember that systemd can do that, so I'll see if I can find
>> out how.
>
> Agree with both the systemd returning to the initr* point (which I
> actually had in mind while writing the above but don't remember the
> details either, so chose to omit in the interest of limiting the size of
> the reply and research necessary to generate it), and the ssd power-cycle
> point.
>
>>> Finally, assuming root itself isn't btrfs, if you have btrfs configured
>>> as a module, you could try unmounting all btrfs and then unloading the
>>> module, then reloading and remounting. That should entirely clear all
>>> in-memory btrfs state, so if that doesn't solve the problem, while
>>> rebooting does, then the problem's very possibly outside of btrfs scope.
>>> Of course if root is btrfs, you can't really check that.
>>
>> Nope, btrfs is built-in (though it doesn't have to be, what with me
>> using an initramfs).
>
> Same here, also gentoo as I guess you know from previous exchanges. But
> unfortunately, if your initr* is anything like mine, and your kernel
> monolithic as mine, making btrfs a module with a btrfs root isn't the
> easy thing it might seem to those who run ordinary distro supplied binary
> kernels with pretty much everything modularized, as doing so involves a
> whole new set of research on how to get that module properly included in
> the initr* and loaded there, as well as installing and building the whole
> module-handling infrastructure (modprobe and friends) again, as it's not
> actually installed on the system at all at this point, because with the
> kernel entirely monolithic, module-handling tools are unnecessary and
> thus just another unnecessary package to have to keep building updates
> for, if they remain installed.
FWIW, I'm pretty sure that genkernel-next (and possibly genkernel too at
this point, but that's never had working userland support for BTRFS
AFAICT) includes btrfs.ko and it's dependencies in any initr* it
generates when it sees those modules on the system, although I'm not
100% certain because I always build the driver for my root filesystem
into the kernel directly.
>
[ snip ]
> What's left in the lib_user list after a dip to emergency mode, whether
> you've returned to multi-user mode or not, is often only systemd and
> perhaps its journald service, itself. Journald can be restarted manually
> in the usual way (systemctl restart journald.service or whatever, I
> usually use tab completion so don't pay all that much attention to the
> specific name), if necessary, while systemd itself can be "reexecuted"
> using the systemctl daemon-reexec (tab-completion again) command.
Udev might also be listed (for some reason, a lot of distros don't stop
it when going to emergency mode, I don't know about what Gentoo does in
this case though because I don't use systemd), and at least the last
time I checked, dropping to emergency mode on Debian and Fedora testing
versions also keeps any dhcp clients or other stuff needed for
maintaining the network connection running as well.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-28 8:44 ` Szalma László
@ 2015-10-28 12:46 ` Duncan
2015-11-02 18:26 ` Szalma László
1 sibling, 0 replies; 16+ messages in thread
From: Duncan @ 2015-10-28 12:46 UTC (permalink / raw)
To: linux-btrfs
Szalma László posted on Wed, 28 Oct 2015 09:44:12 +0100 as excerpted:
> The umount/mount ALWAYS solved the problem for me, mount -o remount,ro
> was tried for the first time, but it was not enought. Reboot was not
> needed.
> (kernel 4.2.4)
So that means it's filesystem state that's tracked thru a remount, not
something like deleted/orphan files that a remount should square away.
But a full unmount clears the state, without having to unload the btrfs
kernel module or reboot.
Which at least limits the active zone of the problem. Unfortunately, I'm
not a dev and don't have a clue where to go from there... unless it's
related to the recent delayed-refs bug, in which case I believe a late
4.3 rc, or 4.3.0 when released, should fix it. That fix should be CCed
to stable and thus appear there eventually, but I normally track pre-
releases, not stable, so don't know how long it might be... And again, I
don't know it's even related, but it does appear to be the active bug of
the moment, so in the absence of knowing, one can hope.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-28 5:21 ` Duncan
2015-10-28 11:23 ` Austin S Hemmelgarn
@ 2015-10-29 21:10 ` Marc Joliet
2015-10-30 9:32 ` Duncan
1 sibling, 1 reply; 16+ messages in thread
From: Marc Joliet @ 2015-10-29 21:10 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 5659 bytes --]
On Wednesday 28 October 2015 05:21:13 Duncan wrote:
>Marc Joliet posted on Tue, 27 Oct 2015 21:54:40 +0100 as excerpted:
>>>IOW, does it take a full reboot to clear the problem, or is a simple
>>>ro/rw mount cycle enough, or an unmount/remount?
>>>
>> Seems that a full reboot is needed, but I would expect that it would
>> have the same effect if I were to pivot back into the initramfs, unmount
>> / from there,
>> then boot back into the system. Because quite frankly, I can't think of
>> any reason why a power cycle to the SSD should make a difference here.
>> I vaguely remember that systemd can do that, so I'll see if I can find
>> out how.
>
>Agree with both the systemd returning to the initr* point (which I
>actually had in mind while writing the above but don't remember the
>details either, so chose to omit in the interest of limiting the size of
>the reply and research necessary to generate it), and the ssd power-cycle
>point.
I haven't found any single command that lets you do that, but I can try one of
the special targets as detailed in bootup(7) (e.g., initrd.target) when I have
a chance.
>>>Finally, assuming root itself isn't btrfs, if you have btrfs configured
>>>as a module, you could try unmounting all btrfs and then unloading the
>>>module, then reloading and remounting. That should entirely clear all
>>>in-memory btrfs state, so if that doesn't solve the problem, while
>>>rebooting does, then the problem's very possibly outside of btrfs scope.
>>>
>>> Of course if root is btrfs, you can't really check that.
>>
>> Nope, btrfs is built-in (though it doesn't have to be, what with me
>> using an initramfs).
>
>Same here, also gentoo as I guess you know from previous exchanges. But
>unfortunately, if your initr* is anything like mine, and your kernel
>monolithic as mine, making btrfs a module with a btrfs root isn't the
>easy thing it might seem to those who run ordinary distro supplied binary
>kernels with pretty much everything modularized, as doing so involves a
>whole new set of research on how to get that module properly included in
>the initr* and loaded there, as well as installing and building the whole
>module-handling infrastructure (modprobe and friends) again, as it's not
>actually installed on the system at all at this point, because with the
>kernel entirely monolithic, module-handling tools are unnecessary and
>thus just another unnecessary package to have to keep building updates
>for, if they remain installed.
My kernel is fairly modular, and I use dracut to make my initramfs, so I
wouldn't be surprised if it works. For me, personally, I just don't see any
point in making btrfs a module.
(And yes, of course I know you run Gentoo ;-) .)
>So I definitely sympathize with the feeling that such a stone is better
>left unturned, if overturning it is at all a possibility that can be
>avoided, as it is here, this whole exercise being simply one of better
>pinning the bug down, not yet actually trying to solve it. And given
>that unturned stone, there are certainly easier ways.
>
>And one of those easier ways is investigating that whole systemd return
>to initr* idea, since we both remember reading something about it, but
>aren't familiar with the details. In addition to addressing the problem
>headon if anyone offers a way to do so, that's the path I'd be looking at
>right now.
Like I said above, I'll try it out when I have a moment where I have a more
"steady hand" so-to-speak.
[snip deleted files stuff]
>app-admin/lib_users
[snip the rest of the deleted files stuff]
I use that to find processes that need restarting after upgrades, though I'll
sometimes check to see if it's really a library that's causing it to show up,
since often a process is listed because of stuff like the font cache, or, in
the case of the FISH shell, it's own history file.
But yeah, didn't think of running that, but in rescue mode there were at most
a dozen processes running, so there's not much to choose from, anyway. I did
have to kill two remaining user processes first (pulseaudio and... I forgot
the other one). I didn't try the same with / and /var because I was eager to
get back to a normally running system ;-) .
>Of course, if lib_users reports nothing further still holding references
>to deleted files, and a remount read-only STILL fails, that's a major
>note of trouble and an important finding in itself.
I don't expect that, but I'll make note of it if I encounter it.
>Meanwhile, as explained in the systemd docs (specifically the systemd for
>administrators series, IIRC), systemd dropping back to the initr* is
>actually its way of automatically doing effectively the same thing we
>were using lib_users and all those restarts to do, getting rid of all
>possible still running on root executables, including systemd itself, by
>reexecing systemd itself back in the initr*, as a way to help eliminate
>*everything* running on root, so it can not only be remounted read-only,
>but actually unmounted the same as any other filesystem, as userspace is
>now actually running from the initr* once again. That's a far *FAR*
>safer reboot after upgrade than traditional sysvinit solutions were able
>to do. =:^)
Yeah, the ability to do that is a nice plus of using an initramfs. Although
I've never been clear on why it's *safer*. Is it because the remount might
fail? Or are there other reasons, too?
[...]
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-29 21:10 ` Marc Joliet
@ 2015-10-30 9:32 ` Duncan
0 siblings, 0 replies; 16+ messages in thread
From: Duncan @ 2015-10-30 9:32 UTC (permalink / raw)
To: linux-btrfs
Marc Joliet posted on Thu, 29 Oct 2015 22:10:24 +0100 as excerpted:
>>Meanwhile, as explained in the systemd docs (specifically the systemd
>>for administrators series, IIRC), systemd dropping back to the initr* is
>>actually its way of automatically doing effectively the same thing we
>>were using lib_users and all those restarts to do, getting rid of all
>>possible still running on root executables, including systemd itself, by
>>reexecing systemd itself back in the initr*, as a way to help eliminate
>>*everything* running on root, so it can not only be remounted read-only,
>>but actually unmounted the same as any other filesystem, as userspace is
>>now actually running from the initr* once again. That's a far *FAR*
>>safer reboot after upgrade than traditional sysvinit solutions were able
>>to do. =:^)
>
> Yeah, the ability to do that is a nice plus of using an initramfs.
> Although I've never been clear on why it's *safer*. Is it because the
> remount might fail? Or are there other reasons, too?
While I don't claim anything but informed admin level authority on the
problem...
It's first worth noting that the problem a return to initramfs helps
solve is in practice reasonably rare and obscure, since if it weren't,
people would have been experiencing it in serious numbers on sysvinit-
based systems all along, and something would have been done to solve it
long before systemd came along. So it's a relatively narrow issue that
in practice can only affect a few users, a relatively small portion of
the time.
>From my read of the systemd docs, it's more pointing out a theoretical
issue than a practical one, pointing out that systemd is in fact a more
theoretically correct solution to the (implicitly mostly theoretical)
problem.
In that context, I believe the (mostly theoretical) point is as much that
we were treating / (and perhaps another mount or two) special, remounting
it read-only instead of unmounting it because in practice there wasn't
any other choice, and that now that systemd offers the choice, it can in
fact be treated just like any other filesystem, fully unmounting it
before shutdown.
Since exceptions to rules are nice places for bugs to hide, in theory at
least (the remount-ro root being such a universal exception that in
practice it's a rule of its own, and bugs couldn't long hide in that
exception /because/ of its universalness), being able to treat / like any
other filesystem and unmount it is a "purer and more correct" solution.
IOW, it's a nice counter to the "systemd isn't unixy enough" point, as
here, it's more "unixy" than sysvinit ever was.
That said, I expect that over the years there have been plenty of
otherwise nice implementations of various useful things that ran into a
shutdown/reboot-time problem due to root's remount-ro exception, that
either limited them to non-root-filesystem deployment or sent them back
for a workaround, if not causing them to rejected outright as unworkable,
that in this new return-to-initr*-and-unmount-root environment will see
faster deployment without the workarounds that heretofore were required.
Of course that'll end up being a limitation on deployment on non-initr*
direct-to-root boot sequences, but in this primarily prebuild binary
distro with prebuild by-necessity-modular-kernel-and-initr* environment,
that's unlikely to slow down wide deployment by much, and anyone wanting
to do direct-to-root boots and/or non-systemd-based deployments will just
have to find their own workarounds, which may ultimately be incorporated
into upstream, or not, depending on upstream's whims.
Which, bringing it all back to the btrfs list title topic, is already
where multi-device btrfs as / filesystem is in terms of initr*, since
that's basically broken without an initr* to assemble it. And of course
the same thing goes for / on LVM, since it too requires userspace to
activate, which means initr* if / is on it.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-28 8:44 ` Szalma László
2015-10-28 12:46 ` Duncan
@ 2015-11-02 18:26 ` Szalma László
2016-02-21 12:01 ` Philipp Serr
1 sibling, 1 reply; 16+ messages in thread
From: Szalma László @ 2015-11-02 18:26 UTC (permalink / raw)
To: linux-btrfs
2015-10-28 09:44 keltezéssel, Szalma László írta:
> Ok, I had a chance to try some things.
>
> 1.: the error
>
> md5sum xyz
> md5sum: xyz: Input/output error
>
> (no any errors in dmesg)
>
> 2.: mount -o remount,ro /mnt/x
>
> (could not do, it is used)
> mysql stop && mount -o remount,ro /mnt/x
> problem persists: io error.
> mount -o remount,rw /mnt/x
> still io error
> umount /mnt/x
> mount /mnt/x
> NO io error, md5sum works!
>
> The umount/mount ALWAYS solved the problem for me, mount -o remount,ro
> was tried for the first time, but it was not enought. Reboot was not
> needed.
> (kernel 4.2.4)
>
> László Szalma
>
Unfortunately the problem with kernel 4.3.0 still exists.
László Szalma
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-11-02 18:26 ` Szalma László
@ 2016-02-21 12:01 ` Philipp Serr
0 siblings, 0 replies; 16+ messages in thread
From: Philipp Serr @ 2016-02-21 12:01 UTC (permalink / raw)
To: linux-btrfs
On Mon, 2 Nov 2015 19:26:01 +0100, Szalma László wrote:
> Unfortunately the problem with kernel 4.3.0 still exists.
Any news on that issue? I'm currently @ 4.3.3 and the issue persists for
me, too.
There's one more information I might be able to contribute: In my case
always the same single file is affected. The issue is triggered mostly
when backing up a read-only snapshot of the volume using rsync. Once the
issue occures, the file within the snapshot volume as well as the file
in the main volume can't be read anymore until reboot (didn't try
remounting the affected root partition).
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2015-10-26 14:23 ` Marc Joliet
2015-10-27 6:23 ` Duncan
@ 2016-04-22 13:17 ` Marc Joliet
2016-05-07 15:22 ` Marc Joliet
1 sibling, 1 reply; 16+ messages in thread
From: Marc Joliet @ 2016-04-22 13:17 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 5005 bytes --]
Am Friday 22 April 2016
schrieb Marc Joliet <marcec@gmx.de>
>Hi
>
>FWIW, this sounds like what I've been seeing with dovecot. In case it's
>relevant, I'll try to explain.
>
>After some uptime, I'll see log messages like this:
>
>Okt 26 12:05:46 thetick dovecot[467]: imap(marcec): Error: pread() failed
>with file /home/marcec/.mdbox/mailboxes/BTRFS/dbox-Mails/dovecot.index.log:
>Input/output error
>
>Occasionally they go away by themselves, but usually I have to reboot to make
>them go away. This happens when getmail attempts to fetch mail, which fails
>due to the above error. After the reboot getmail succeeds again.
>
>As in Szalma's case, btrfs-scrub never reports anything wrong.
FWIW, this still happens. I tried various things, in order:
- set mmap_disable=yes (as mentioned in [0] and [1]),
- stop mounting with compress=lzo, and
- upgrade dovecot 2.2.19 to 2.2.23,
none of which helped.
I found very little on the web: [0] (in German) is fairly recent, but the
error apparently just went away. [1] is pretty old (2007) and the reporter
was using unionfs. I also found a russian forum entry [2], which relates to
btrfs again, but if Google Translate is accurate enough, the user "solved" the
problem by using XFS instead.
My current attempt is to use the maildir mailbox format (instead of mdbox) in
the hope that it doesn't trigger this bug, while reverting the changes above
(except for the dovecot upgrade). It's "just" a home server, so I don't
expect any major differences in performance (also since KMail probably uses
akonadi for searching).
[...]
>Anyway, the current state of the system is:
>
># uname -r
>4.1.9-gentoo-r1
># btrfs filesystem show /
>Label: 'MARCEC_ROOT' uuid: 0267d8b3-a074-460a-832d-5d5fd36bae64
> Total devices 1 FS bytes used 74.40GiB
> devid 1 size 107.79GiB used 105.97GiB path /dev/sda1
>
>btrfs-progs v4.2.2
># btrfs filesystem df /
>Data, single: total=98.94GiB, used=72.30GiB
>System, single: total=32.00MiB, used=20.00KiB
>Metadata, single: total=7.00GiB, used=2.10GiB
>GlobalReserve, single: total=512.00MiB, used=0.00B
>
>The filesystem is mounted as (leaving out subvolume mounts which use the same
>mount options):
>/dev/sda1 on / type btrfs (rw,noatime,compress=lzo,ssd,discard,space_cache)
I have since obtained an old Mac Mini and have been using it as a home server
since early April, onto which I moved dovecot (using doveadm-sync(1)). Its
current state is:
diefledermaus etc # uname -r
4.4.8-gentoo
diefledermaus etc # btrfs --version
btrfs-progs v4.5.1
diefledermaus etc # btrfs fi show
Label: 'MACROOT' uuid: 8b2b0242-5816-4529-9e88-6c82ffff2eaf
Total devices 1 FS bytes used 12.29GiB
devid 1 size 251.20GiB used 15.09GiB path /dev/sda3
Label: 'MARCEC_BACKUP' uuid: f97b3cda-15e8-418b-bb9b-235391ef2a38
Total devices 1 FS bytes used 797.14GiB
devid 1 size 976.56GiB used 819.12GiB path /dev/sdb2
diefledermaus etc # btrfs fi df /
Data, single: total=12.01GiB, used=11.38GiB
System, DUP: total=40.00MiB, used=16.00KiB
Metadata, DUP: total=1.50GiB, used=926.77MiB
GlobalReserve, single: total=288.00MiB, used=0.00B
diefledermaus etc # mount | grep root
/dev/sda3 on / type btrfs
(rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=257,subvol=/rootfs)
/dev/sda3 on /mnt/rootfs type btrfs
(rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=5,subvol=/)
Note that the file system was *freshly* created using btrfs-progs 4.3.1 and
gentoo-sources 4.1.15-r1 (the current stable versions in Gentoo; -r1 is a
Gentoo-specific revision number).
References:
[0] https://debianforum.de/forum/viewtopic.php?f=27&t=157374
[1] http://www.dovecot.org/list/dovecot/2007-November/026551.html
[2] https://www.linux.org.ru/forum/general/11598803
-----------------------------------------------------------
I'd like to add that dovecot is the *only* software I use that has this sort
of problem with btrfs -- and it's *only* the transaction *.log files; the few
times the index files had problems, they were automatically fixed. However,
once the problem shows up, no other software can read the files, either (e.g.,
cat fails). Furthermore, restarting dovecot doesn't help, only rebooting,
i.e., unmounting the entire file system, not just the subvolume the mailbox
resides on (as discussed previously in this sub-thread). So I'm not sure
what, exactly, is at fault here: dovecot, btrfs, or both?
So given all the above, does anybody have any further suggestions on how to
proceed? Because regardless of whether the switch to maildir ends up
resolving the issue for me, something is going wrong. I'm thinking of filing
a bug report with dovecot; perhaps none of their devs test with btrfs?
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: random i/o error without error in dmesg
2016-04-22 13:17 ` Marc Joliet
@ 2016-05-07 15:22 ` Marc Joliet
0 siblings, 0 replies; 16+ messages in thread
From: Marc Joliet @ 2016-05-07 15:22 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 822 bytes --]
Am Saturday 07 May 2016
schrieb Marc Joliet <marcec@gmx.de>
>I'm thinking of filing
>a bug report with dovecot; perhaps none of their devs test with btrfs?
So I did this, and got a little bit of feedback. Quoting from the reply I
got:
"*.index.log files are always appended to using O_APPEND flag. Maybe this is
relevant.
Also when a new .log file is created it's opened without the O_APPEND flag and
the O_APPEND is added later. This was causing a bug recently in unionfs, which
ignored the flag change and caused log file corruption."
The other bit of advise was to stress test dovecot using imaptest, but I'll
have to do that when I have more time.
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2016-05-07 15:22 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-10-26 11:23 random i/o error without error in dmesg Szalma László
2015-10-26 14:23 ` Marc Joliet
2015-10-27 6:23 ` Duncan
2015-10-27 9:19 ` Marc Joliet
2015-10-27 14:57 ` Szalma László
2015-10-27 20:54 ` Marc Joliet
2015-10-28 5:21 ` Duncan
2015-10-28 11:23 ` Austin S Hemmelgarn
2015-10-29 21:10 ` Marc Joliet
2015-10-30 9:32 ` Duncan
2015-10-28 8:44 ` Szalma László
2015-10-28 12:46 ` Duncan
2015-11-02 18:26 ` Szalma László
2016-02-21 12:01 ` Philipp Serr
2016-04-22 13:17 ` Marc Joliet
2016-05-07 15:22 ` Marc Joliet
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.