Re: WARNING in xfs_lwr.c, xfs_write()

[Ilia Mirkin <imirkin@alum.mit.edu>]

On Sun, Jun 13, 2010 at 9:29 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Sun, Jun 13, 2010 at 07:10:30PM -0400, Ilia Mirkin wrote:
>> On Sun, Jun 13, 2010 at 6:47 PM, Dave Chinner <david@fromorbit.com> wrote:
>> > On Sat, Jun 12, 2010 at 01:00:52AM -0400, Ilia Mirkin wrote:
>> >> Sorry to pick up an old-ish thread, but I have a similar situation:
>> >>
>> >> On Sun, May 23, 2010 at 9:19 PM, Dave Chinner <david@fromorbit.com> wrote:
>> >> > On Sun, May 23, 2010 at 09:23:44AM -0500, Roman Kononov wrote:
>> >> >> On 2010-05-23, 20:18:56 +1000, Dave Chinner <david@fromorbit.com> wrote:
>> >> >> > Can you find out what the application is triggering this?
>> >>
>> >> I noticed this happening with mysql and xtrabackup -- the latter opens
>> >> up mysql's files while mysql is still running (and modifying its own
>> >> files) and backs them up in a (hopefully) safe way.
>> >
>> > That's not safe at all - there's no guarantee you'll end up with a
>> > consistent database image doing backups like this. Have you ever
>> > tried to restore and use one of these backups?
>>
>> Yep, works great. [Used it to initialize a slave, did the full
>> checksums, so it's unlikely to have randomly corrupt data.]
>
> You were lucky, I'd say.  xtrabackup is supposed to be tightly
> integrated with mysql, so perhaps it should be using the same IO
> methods that the admin has selected for their database. Maybe you
> need to talk to the xtrabackup folks to get them to add a "backup
> via direct IO" method if the mysql database is using direct IO so
> that other uses don't have the same issues.

Maybe. We've been using this technique, although on a different
physical machine and with ext3, for quite some time (and we verify all
backups). I did notice that there is a minor difference in
configuration, esp wrt direct IO, so I'll check it out in more detail.
[We're now setting innodb_flush_method to O_DIRECT whereas we weren't
before... although based on the documentation and a cursory
understanding of how xtrabackup works, this shouldn't be harmful.]

> And from a "I read it on the interwebs so it must be true"
> perspective, without a loud obnoxious warning we'll never hear about
> problems until someone flames us about silent data corruption on a
> random blog that gets slashdotted and then referenced for the next
> 10 years as the next canonical "XFS eats my data!" reference for the
> clueless....

Instead it will be "mysql works fine on ext3, but with xfs it spams
the logs with warnings, therefore xfs must be broken". I don't think
there's anything realistically that you can do about uninformed users
and FUD. Although I wasn't suggesting to get rid of the warning,
rather to make it more explicit as to what it's warning about. I
interpret a WARN as a BUG that can be recovered but where the
underlying system needs a careful look; my first inclination after
seeing a fs-related WARN would be to take the system down and run an
fsck. What's happening here seems more akin to getting a WARN when
calling an ioctl with invalid parameters.

---
Ilia Mirkin
imirkin@alum.mit.edu