* Files not touched in weeks got truncated after a crash
@ 2013-11-13 16:36 Guido Winkelmann
2013-11-13 16:51 ` Roger Willcocks
2013-11-13 20:39 ` Ric Wheeler
0 siblings, 2 replies; 6+ messages in thread
From: Guido Winkelmann @ 2013-11-13 16:36 UTC (permalink / raw)
To: xfs
Hi,
We are having some trouble with one of our fileservers using XFS (on linux).
Yesterday, one of the external RAIDs on the server failed. Of course, it is
unavoidable that some data would get lost from the fileserver in such an
event, however, we lost a lot more files than would seem reasonable. In
particular, we lost a number of files that had not been written to (but had
been been read from, in some cases) in several weeks.
The data loss manifested itself through files being truncated to length 0 or
to some other size short of what they should be. (We happen to have an
external database that keeps track of that.)
The fileserver is based on CentOS 6.3 with kernel version
2.6.32-279.9.1.el6.x86_64. It has got several external RAIDs in the 100 TB
range, connected via FibreChannel.
In case it matters: The server's primary role is as a samba server servicing a
large number of Windows XP and Windows 7 machines.
We had already been trying to reduce the possible impact of a hardware failure
by setting a few tunables in /etc/sysctl.conf to try and make the kernel not
keep dirty buffers around too long:
vm.dirty_background_bytes = 536870912
vm.dirty_bytes = 134217728
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 3000
and by issuing a sync from cron every 15 minutes:
0,15,30,45 * * * * /bin/sync
Unfortunately, I seem to be unable so far to reproduce the issue on a smaller
system - and I cannot exactly just walk up to the in-production fileserver and
rip out yet another array just to see what happens...
This leaves me with a few questions:
Why did we lose so much data through the crash?
Why did not even a sync every 15 minutes prevent further damage?
What can we do to prevent this from happening again in the future?
Regards,
Guido
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Files not touched in weeks got truncated after a crash
2013-11-13 16:36 Files not touched in weeks got truncated after a crash Guido Winkelmann
@ 2013-11-13 16:51 ` Roger Willcocks
2013-11-14 10:25 ` Guido Winkelmann
2013-11-13 20:39 ` Ric Wheeler
1 sibling, 1 reply; 6+ messages in thread
From: Roger Willcocks @ 2013-11-13 16:51 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: xfs
On Wed, 2013-11-13 at 17:36 +0100, Guido Winkelmann wrote:
> Hi,
>
> We are having some trouble with one of our fileservers using XFS (on linux).
> Yesterday, one of the external RAIDs on the server failed. Of course, it is
> unavoidable that some data would get lost from the fileserver in such an
> event, however, we lost a lot more files than would seem reasonable. In
> particular, we lost a number of files that had not been written to (but had
> been been read from, in some cases) in several weeks.
>
> The data loss manifested itself through files being truncated to length 0 or
> to some other size short of what they should be. (We happen to have an
> external database that keeps track of that.)
>
> The fileserver is based on CentOS 6.3 with kernel version
> 2.6.32-279.9.1.el6.x86_64. It has got several external RAIDs in the 100 TB
> range, connected via FibreChannel.
>
> In case it matters: The server's primary role is as a samba server servicing a
> large number of Windows XP and Windows 7 machines.
>
> We had already been trying to reduce the possible impact of a hardware failure
> by setting a few tunables in /etc/sysctl.conf to try and make the kernel not
> keep dirty buffers around too long:
>
> vm.dirty_background_bytes = 536870912
> vm.dirty_bytes = 134217728
> vm.dirty_writeback_centisecs = 500
> vm.dirty_expire_centisecs = 3000
>
> and by issuing a sync from cron every 15 minutes:
>
> 0,15,30,45 * * * * /bin/sync
>
> Unfortunately, I seem to be unable so far to reproduce the issue on a smaller
> system - and I cannot exactly just walk up to the in-production fileserver and
> rip out yet another array just to see what happens...
>
> This leaves me with a few questions:
>
> Why did we lose so much data through the crash?
>
> Why did not even a sync every 15 minutes prevent further damage?
>
> What can we do to prevent this from happening again in the future?
>
> Regards,
>
> Guido
Syncing won't protect you from hardware failures.
Without more details about what happened to the external RAID and what
you did to recover it's impossible to answer your question.
--
Roger
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Files not touched in weeks got truncated after a crash
2013-11-13 16:36 Files not touched in weeks got truncated after a crash Guido Winkelmann
2013-11-13 16:51 ` Roger Willcocks
@ 2013-11-13 20:39 ` Ric Wheeler
2013-11-13 21:43 ` Stefan Ring
1 sibling, 1 reply; 6+ messages in thread
From: Ric Wheeler @ 2013-11-13 20:39 UTC (permalink / raw)
To: Guido Winkelmann, xfs
You should update your kernel - this sounds like an issue that Dave fixed quite
a few months back (and got shipped in RHEL and other distros, I don't know about
when Centos would pick it up)
Ric
On 11/14/2013 01:36 AM, Guido Winkelmann wrote:
> Hi,
>
> We are having some trouble with one of our fileservers using XFS (on linux).
> Yesterday, one of the external RAIDs on the server failed. Of course, it is
> unavoidable that some data would get lost from the fileserver in such an
> event, however, we lost a lot more files than would seem reasonable. In
> particular, we lost a number of files that had not been written to (but had
> been been read from, in some cases) in several weeks.
>
> The data loss manifested itself through files being truncated to length 0 or
> to some other size short of what they should be. (We happen to have an
> external database that keeps track of that.)
>
> The fileserver is based on CentOS 6.3 with kernel version
> 2.6.32-279.9.1.el6.x86_64. It has got several external RAIDs in the 100 TB
> range, connected via FibreChannel.
>
> In case it matters: The server's primary role is as a samba server servicing a
> large number of Windows XP and Windows 7 machines.
>
> We had already been trying to reduce the possible impact of a hardware failure
> by setting a few tunables in /etc/sysctl.conf to try and make the kernel not
> keep dirty buffers around too long:
>
> vm.dirty_background_bytes = 536870912
> vm.dirty_bytes = 134217728
> vm.dirty_writeback_centisecs = 500
> vm.dirty_expire_centisecs = 3000
>
> and by issuing a sync from cron every 15 minutes:
>
> 0,15,30,45 * * * * /bin/sync
>
> Unfortunately, I seem to be unable so far to reproduce the issue on a smaller
> system - and I cannot exactly just walk up to the in-production fileserver and
> rip out yet another array just to see what happens...
>
> This leaves me with a few questions:
>
> Why did we lose so much data through the crash?
>
> Why did not even a sync every 15 minutes prevent further damage?
>
> What can we do to prevent this from happening again in the future?
>
> Regards,
>
> Guido
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Files not touched in weeks got truncated after a crash
2013-11-13 20:39 ` Ric Wheeler
@ 2013-11-13 21:43 ` Stefan Ring
2013-11-14 10:11 ` Guido Winkelmann
0 siblings, 1 reply; 6+ messages in thread
From: Stefan Ring @ 2013-11-13 21:43 UTC (permalink / raw)
To: Linux fs XFS; +Cc: Guido Winkelmann
On Wed, Nov 13, 2013 at 9:39 PM, Ric Wheeler <ricwheeler@gmail.com> wrote:
> You should update your kernel - this sounds like an issue that Dave fixed
> quite a few months back (and got shipped in RHEL and other distros, I don't
> know about when Centos would pick it up)
It's in the CentOS 6.4 kernel, according to the release notes:
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/6.4_Technical_Notes/kernel.html
(BZ#855139)
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Files not touched in weeks got truncated after a crash
2013-11-13 21:43 ` Stefan Ring
@ 2013-11-14 10:11 ` Guido Winkelmann
0 siblings, 0 replies; 6+ messages in thread
From: Guido Winkelmann @ 2013-11-14 10:11 UTC (permalink / raw)
To: xfs
On Wednesday 13 November 2013 22:43:23 Stefan Ring wrote:
>On Wed, Nov 13, 2013 at 9:39 PM, Ric Wheeler <ricwheeler@gmail.com> wrote:
>> You should update your kernel - this sounds like an issue that Dave fixed
>> quite a few months back (and got shipped in RHEL and other distros, I don't
>> know about when Centos would pick it up)
>
>It's in the CentOS 6.4 kernel, according to the release notes:
>https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6
>/html/6.4_Technical_Notes/kernel.html (BZ#855139)
Hm, this could be it, but it still sounds a bit unlikely.
The bug description implies that the filesystem would have to have been idling
from the point where the affected files were written until the point of the
system crash (or unclean unmount), such that the old, and by that time
incorrect, metadata updates would still linger in the journal, but that was
not really the case here. The filesystem was quite busy during those weeks....
Also, the bug report speaks only of zero-length files, while many of our files
had been truncated to lengths other than zero.
Guido
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Files not touched in weeks got truncated after a crash
2013-11-13 16:51 ` Roger Willcocks
@ 2013-11-14 10:25 ` Guido Winkelmann
0 siblings, 0 replies; 6+ messages in thread
From: Guido Winkelmann @ 2013-11-14 10:25 UTC (permalink / raw)
To: xfs
On Wednesday 13 November 2013 16:51:44 Roger Willcocks wrote:
>Syncing won't protect you from hardware failures.
Obviously not, but it might - should, I would argue - lessen the impact of
one.
>Without more details about what happened to the external RAID and what
>you did to recover it's impossible to answer your question.
Well, if I understood the admin correctly, the backplane in the RAID in
question died, probably from a short-circuit on the board. (But I think that
latter part was just speculation.) He removed the harddisks from that
enclosure, put them into another one of the same make and model, mirrored over
the configuration from the broken to the working one and reattached it to the
server. The server was shut down in the mean time.
Guido
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-11-14 10:25 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-11-13 16:36 Files not touched in weeks got truncated after a crash Guido Winkelmann
2013-11-13 16:51 ` Roger Willcocks
2013-11-14 10:25 ` Guido Winkelmann
2013-11-13 20:39 ` Ric Wheeler
2013-11-13 21:43 ` Stefan Ring
2013-11-14 10:11 ` Guido Winkelmann
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.