All of lore.kernel.org
 help / color / mirror / Atom feed
From: Linus Torvalds <torvalds@linux-foundation.org>
To: Andreas Gruenbacher <agruenba@redhat.com>
Cc: Christoph Hellwig <hch@infradead.org>,
	"Darrick J. Wong" <djwong@kernel.org>,
	Dave Chinner <dchinner@redhat.com>,
	cluster-devel <cluster-devel@redhat.com>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: [GIT PULL] gfs2 fix
Date: Tue, 3 May 2022 09:19:24 -0700	[thread overview]
Message-ID: <CAHk-=whfP+m6--NtUeOm5XTuhBGHkyoqd00ypW6v3RkzMFLU8g@mail.gmail.com> (raw)
In-Reply-To: <CAHc6FU77KGn76B4ieu9Tn895deK-1yV4y=8ou4gTfUf=7C-4XQ@mail.gmail.com>

On Tue, May 3, 2022 at 1:56 AM Andreas Gruenbacher <agruenba@redhat.com> wrote:
>
> We still get data corruption with the patch applied. The
> WARN_ON_ONCE(!bytes) doesn't trigger.

Oh well. I was so sure that I'd finally found something.. That partial
write case has had bugs before.

> As an additional experiment, I've added code to check the iterator
> position that iomap_file_buffered_write() returns, and it's all
> looking good as well: an iov_iter_advance(orig_from, written) from the
> original position always gets us to the same iterator.

Yeah, I've looked at the iterator parts (and iov_iter_revert() in
particular) multiple times, because that too is an area where we've
had bugs before.

That too may be easy to get wrong, but I couldn't for the life of me
see any issues there.

> This points at gfs2 getting things wrong after a short write, for
> example, marking a page / folio uptodate that isn't. But the uptodate
> handling happens at the iomap layer, so this doesn't leave me with an
> immediate suspect.

Yeah, the uptodate setting looked safe, particularly with that "if we
copied less than we thought we would, and it wasn't uptodate, just
claim we didn't do anything at all".

That said, I now have a *new* suspect: the 'iter->pos' handling in
iomap_write_iter().

In particular, let's look at iomap_file_buffered_write(), which does:

        while ((ret = iomap_iter(&iter, ops)) > 0)
                iter.processed = iomap_write_iter(&iter, i);

and then look at what happens to iter.pos here.

iomap_write_iter() does this:

        loff_t pos = iter->pos;
        ...
                pos += status;

but it never seems to write the updated position back to the iterator.

So what happens next time iomap_write_iter() gets called?

This looks like such a huge bug that I'm probably missing something,
but I wonder if this is normally hidden by the fact that usually
iomap_write_iter() consumes the whole 'iter', so despite the 'while()'
loop, it's actually effectively only called once.

Except if it gets a short write due to an unhandled page fault..

Am I entirely blind, and that 'iter.pos' is updated somewhere and I
just missed it?

Or is this maybe the reason for it all?

                Linus

WARNING: multiple messages have this Message-ID (diff)
From: Linus Torvalds <torvalds@linux-foundation.org>
To: cluster-devel.redhat.com
Subject: [Cluster-devel] [GIT PULL] gfs2 fix
Date: Tue, 3 May 2022 09:19:24 -0700	[thread overview]
Message-ID: <CAHk-=whfP+m6--NtUeOm5XTuhBGHkyoqd00ypW6v3RkzMFLU8g@mail.gmail.com> (raw)
In-Reply-To: <CAHc6FU77KGn76B4ieu9Tn895deK-1yV4y=8ou4gTfUf=7C-4XQ@mail.gmail.com>

On Tue, May 3, 2022 at 1:56 AM Andreas Gruenbacher <agruenba@redhat.com> wrote:
>
> We still get data corruption with the patch applied. The
> WARN_ON_ONCE(!bytes) doesn't trigger.

Oh well. I was so sure that I'd finally found something.. That partial
write case has had bugs before.

> As an additional experiment, I've added code to check the iterator
> position that iomap_file_buffered_write() returns, and it's all
> looking good as well: an iov_iter_advance(orig_from, written) from the
> original position always gets us to the same iterator.

Yeah, I've looked at the iterator parts (and iov_iter_revert() in
particular) multiple times, because that too is an area where we've
had bugs before.

That too may be easy to get wrong, but I couldn't for the life of me
see any issues there.

> This points at gfs2 getting things wrong after a short write, for
> example, marking a page / folio uptodate that isn't. But the uptodate
> handling happens at the iomap layer, so this doesn't leave me with an
> immediate suspect.

Yeah, the uptodate setting looked safe, particularly with that "if we
copied less than we thought we would, and it wasn't uptodate, just
claim we didn't do anything at all".

That said, I now have a *new* suspect: the 'iter->pos' handling in
iomap_write_iter().

In particular, let's look at iomap_file_buffered_write(), which does:

        while ((ret = iomap_iter(&iter, ops)) > 0)
                iter.processed = iomap_write_iter(&iter, i);

and then look at what happens to iter.pos here.

iomap_write_iter() does this:

        loff_t pos = iter->pos;
        ...
                pos += status;

but it never seems to write the updated position back to the iterator.

So what happens next time iomap_write_iter() gets called?

This looks like such a huge bug that I'm probably missing something,
but I wonder if this is normally hidden by the fact that usually
iomap_write_iter() consumes the whole 'iter', so despite the 'while()'
loop, it's actually effectively only called once.

Except if it gets a short write due to an unhandled page fault..

Am I entirely blind, and that 'iter.pos' is updated somewhere and I
just missed it?

Or is this maybe the reason for it all?

                Linus


  parent reply	other threads:[~2022-05-03 16:19 UTC|newest]

Thread overview: 64+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-04-26 14:54 [GIT PULL] gfs2 fix Andreas Gruenbacher
2022-04-26 14:54 ` [Cluster-devel] " Andreas Gruenbacher
2022-04-26 18:31 ` Linus Torvalds
2022-04-26 18:31   ` [Cluster-devel] " Linus Torvalds
2022-04-26 21:27   ` Andreas Gruenbacher
2022-04-26 21:27     ` [Cluster-devel] " Andreas Gruenbacher
2022-04-26 23:33     ` Linus Torvalds
2022-04-26 23:33       ` [Cluster-devel] " Linus Torvalds
2022-04-27 12:29       ` Andreas Gruenbacher
2022-04-27 12:29         ` [Cluster-devel] " Andreas Gruenbacher
2022-04-27 17:13         ` Linus Torvalds
2022-04-27 17:13           ` [Cluster-devel] " Linus Torvalds
2022-04-27 19:41           ` Andreas Gruenbacher
2022-04-27 19:41             ` [Cluster-devel] " Andreas Gruenbacher
2022-04-27 20:25             ` Linus Torvalds
2022-04-27 20:25               ` [Cluster-devel] " Linus Torvalds
2022-04-27 21:26               ` Andreas Gruenbacher
2022-04-27 21:26                 ` [Cluster-devel] " Andreas Gruenbacher
2022-04-27 22:20                 ` Linus Torvalds
2022-04-27 22:20                   ` [Cluster-devel] " Linus Torvalds
2022-04-28  0:00                   ` Linus Torvalds
2022-04-28  0:00                     ` [Cluster-devel] " Linus Torvalds
2022-04-28 13:26                     ` Andreas Gruenbacher
2022-04-28 13:26                       ` [Cluster-devel] " Andreas Gruenbacher
2022-04-28 17:09                       ` Linus Torvalds
2022-04-28 17:09                         ` [Cluster-devel] " Linus Torvalds
2022-04-28 17:17                         ` Linus Torvalds
2022-04-28 17:17                           ` [Cluster-devel] " Linus Torvalds
2022-04-28 17:21                           ` Andreas Gruenbacher
2022-04-28 17:21                             ` [Cluster-devel] " Andreas Gruenbacher
2022-04-28 17:38                         ` Andreas Gruenbacher
2022-04-28 17:38                           ` [Cluster-devel] " Andreas Gruenbacher
2022-05-02 18:31                           ` Linus Torvalds
2022-05-02 18:31                             ` [Cluster-devel] " Linus Torvalds
2022-05-02 18:58                             ` Linus Torvalds
2022-05-02 18:58                               ` [Cluster-devel] " Linus Torvalds
2022-05-02 20:24                               ` Andreas Gruenbacher
2022-05-02 20:24                                 ` [Cluster-devel] " Andreas Gruenbacher
2022-05-03  8:56                             ` Andreas Gruenbacher
2022-05-03  8:56                               ` [Cluster-devel] " Andreas Gruenbacher
2022-05-03 13:30                               ` Andreas Gruenbacher
2022-05-03 13:30                                 ` [Cluster-devel] " Andreas Gruenbacher
2022-05-03 16:19                               ` Linus Torvalds [this message]
2022-05-03 16:19                                 ` Linus Torvalds
2022-05-03 16:41                                 ` Andreas Gruenbacher
2022-05-03 16:41                                   ` [Cluster-devel] " Andreas Gruenbacher
2022-05-03 16:50                                   ` Linus Torvalds
2022-05-03 16:50                                     ` [Cluster-devel] " Linus Torvalds
2022-05-03 21:35                               ` Andreas Gruenbacher
2022-05-03 21:35                                 ` [Cluster-devel] " Andreas Gruenbacher
2022-05-03 22:41                                 ` Linus Torvalds
2022-05-03 22:41                                   ` [Cluster-devel] " Linus Torvalds
2022-05-04 17:52                                   ` Andreas Gruenbacher
2022-05-04 17:52                                     ` [Cluster-devel] " Andreas Gruenbacher
2022-04-28 18:16                       ` pr-tracker-bot
2022-04-28 18:16                         ` [Cluster-devel] " pr-tracker-bot
2022-04-26 19:07 ` pr-tracker-bot
2022-04-26 19:07   ` [Cluster-devel] " pr-tracker-bot
2023-06-06 12:48 Andreas Gruenbacher
2023-06-06 12:55 ` Linus Torvalds
2023-06-06 13:32   ` Andreas Gruenbacher
2023-06-06 13:22 ` pr-tracker-bot
2024-03-25 12:10 Andreas Gruenbacher
2024-03-25 18:18 ` pr-tracker-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAHk-=whfP+m6--NtUeOm5XTuhBGHkyoqd00ypW6v3RkzMFLU8g@mail.gmail.com' \
    --to=torvalds@linux-foundation.org \
    --cc=agruenba@redhat.com \
    --cc=cluster-devel@redhat.com \
    --cc=dchinner@redhat.com \
    --cc=djwong@kernel.org \
    --cc=hch@infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.