Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Eric Biggers]

[+Cc linux-fsdevel]

On Wed, Jul 15, 2020 at 05:29:12PM +0200, 'Marco Elver' via syzkaller-bugs wrote:
> On Wed, Jul 15, 2020 at 08:16AM -0700, syzbot wrote:
> > Hello,
> > 
> > syzbot found the following issue on:
> > 
> > HEAD commit:    e9919e11 Merge branch 'for-linus' of git://git.kernel.org/..
> > git tree:       upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=1217a83b100000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=570eb530a65cd98e
> > dashboard link: https://syzkaller.appspot.com/bug?extid=0f1e470df6a4316e0a11
> > compiler:       clang version 11.0.0 (https://github.com/llvm/llvm-project.git ca2dcbd030eadbf0aa9b660efe864ff08af6e18b)
> > 
> > Unfortunately, I don't have any reproducer for this issue yet.
> > 
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+0f1e470df6a4316e0a11@syzkaller.appspotmail.com
> > 
> > ==================================================================
> > BUG: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read
> 
> Our guess is that this is either misuse of an API from userspace, or a
> bug. Can someone clarify?
> 
> Below are the snippets of code around these accesses.

Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
as that implicitly uses the file position.  But it's allowed with sys_pread(),
and also with sys_sendfile() which is the case syzbot is reporting here.

> 
> > write to 0xffff8880968747b0 of 8 bytes by task 6336 on cpu 0:
> >  generic_file_buffered_read+0x18be/0x19e0 mm/filemap.c:2246
> 
> 	...
> 	would_block:
> 		error = -EAGAIN;
> 	out:
> 		ra->prev_pos = prev_index;
> 		ra->prev_pos <<= PAGE_SHIFT;
> 2246)		ra->prev_pos |= prev_offset;
> 
> 		*ppos = ((loff_t)index << PAGE_SHIFT) + offset;
> 		file_accessed(filp);
> 		return written ? written : error;
> 	}
> 	EXPORT_SYMBOL_GPL(generic_file_buffered_read);
> 	...

Well, it's a data race.  Each open file descriptor has just one readahead state
(struct file_ra_state), and concurrent reads of the same file descriptor
use/change that readahead state without any locking.

Presumably this has traditionally been considered okay, since readahead is
"only" for performance and doesn't affect correctness.  And for performance
reasons, we want to avoid locking during file reads.

So we may just need to annotate all access to file_ra_state with
READ_ONCE() and WRITE_ONCE()...

> 
> >  generic_file_read_iter+0x7d/0x3e0 mm/filemap.c:2326
> >  ext4_file_read_iter+0x2d6/0x420 fs/ext4/file.c:74
> >  call_read_iter include/linux/fs.h:1902 [inline]
> >  generic_file_splice_read+0x22a/0x310 fs/splice.c:312
> >  do_splice_to fs/splice.c:870 [inline]
> >  splice_direct_to_actor+0x2a8/0x660 fs/splice.c:950
> >  do_splice_direct+0xf2/0x170 fs/splice.c:1059
> >  do_sendfile+0x562/0xb10 fs/read_write.c:1540
> >  __do_sys_sendfile64 fs/read_write.c:1601 [inline]
> >  __se_sys_sendfile64 fs/read_write.c:1587 [inline]
> >  __x64_sys_sendfile64+0xf2/0x130 fs/read_write.c:1587
> >  do_syscall_64+0x51/0xb0 arch/x86/entry/common.c:384
> >  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > 
> > read to 0xffff8880968747b0 of 8 bytes by task 6334 on cpu 1:
> >  generic_file_buffered_read+0x11e/0x19e0 mm/filemap.c:2011
> 
> 	...
> 	index = *ppos >> PAGE_SHIFT;
> 	prev_index = ra->prev_pos >> PAGE_SHIFT;
> 2011)	prev_offset = ra->prev_pos & (PAGE_SIZE-1);
> 	last_index = (*ppos + iter->count + PAGE_SIZE-1) >> PAGE_SHIFT;
> 	offset = *ppos & ~PAGE_MASK;
> 	...
> 
> >  generic_file_read_iter+0x7d/0x3e0 mm/filemap.c:2326
> >  ext4_file_read_iter+0x2d6/0x420 fs/ext4/file.c:74
> >  call_read_iter include/linux/fs.h:1902 [inline]
> >  generic_file_splice_read+0x22a/0x310 fs/splice.c:312
> >  do_splice_to fs/splice.c:870 [inline]
> >  splice_direct_to_actor+0x2a8/0x660 fs/splice.c:950
> >  do_splice_direct+0xf2/0x170 fs/splice.c:1059
> >  do_sendfile+0x562/0xb10 fs/read_write.c:1540
> >  __do_sys_sendfile64 fs/read_write.c:1601 [inline]
> >  __se_sys_sendfile64 fs/read_write.c:1587 [inline]
> >  __x64_sys_sendfile64+0xf2/0x130 fs/read_write.c:1587
> >  do_syscall_64+0x51/0xb0 arch/x86/entry/common.c:384
> >  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > 
> > Reported by Kernel Concurrency Sanitizer on:
> > CPU: 1 PID: 6334 Comm: syz-executor.0 Not tainted 5.8.0-rc5-syzkaller #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > ==================================================================
> 
> Thanks,
> -- Marco

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Marco Elver]

On Wed, 15 Jul 2020 at 18:33, Eric Biggers <ebiggers@kernel.org> wrote:
>
> [+Cc linux-fsdevel]
>
> On Wed, Jul 15, 2020 at 05:29:12PM +0200, 'Marco Elver' via syzkaller-bugs wrote:
> > On Wed, Jul 15, 2020 at 08:16AM -0700, syzbot wrote:
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit:    e9919e11 Merge branch 'for-linus' of git://git.kernel.org/..
> > > git tree:       upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=1217a83b100000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=570eb530a65cd98e
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=0f1e470df6a4316e0a11
> > > compiler:       clang version 11.0.0 (https://github.com/llvm/llvm-project.git ca2dcbd030eadbf0aa9b660efe864ff08af6e18b)
> > >
> > > Unfortunately, I don't have any reproducer for this issue yet.
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+0f1e470df6a4316e0a11@syzkaller.appspotmail.com
> > >
> > > ==================================================================
> > > BUG: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read
> >
> > Our guess is that this is either misuse of an API from userspace, or a
> > bug. Can someone clarify?
> >
> > Below are the snippets of code around these accesses.
>
> Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
> as that implicitly uses the file position.  But it's allowed with sys_pread(),
> and also with sys_sendfile() which is the case syzbot is reporting here.
>
> >
> > > write to 0xffff8880968747b0 of 8 bytes by task 6336 on cpu 0:
> > >  generic_file_buffered_read+0x18be/0x19e0 mm/filemap.c:2246
> >
> >       ...
> >       would_block:
> >               error = -EAGAIN;
> >       out:
> >               ra->prev_pos = prev_index;
> >               ra->prev_pos <<= PAGE_SHIFT;
> > 2246)         ra->prev_pos |= prev_offset;
> >
> >               *ppos = ((loff_t)index << PAGE_SHIFT) + offset;
> >               file_accessed(filp);
> >               return written ? written : error;
> >       }
> >       EXPORT_SYMBOL_GPL(generic_file_buffered_read);
> >       ...
>
> Well, it's a data race.  Each open file descriptor has just one readahead state
> (struct file_ra_state), and concurrent reads of the same file descriptor
> use/change that readahead state without any locking.
>
> Presumably this has traditionally been considered okay, since readahead is
> "only" for performance and doesn't affect correctness.  And for performance
> reasons, we want to avoid locking during file reads.
>
> So we may just need to annotate all access to file_ra_state with
> READ_ONCE() and WRITE_ONCE()...

The thing that stood out here are the multiple accesses both on the
reader and writer side. If it was only 1 access, where the race is
expected, a simple READ/WRITE_ONCE might have been OK.

But here, we actually have several writes to the same variable
'prev_pos'. The reader is also doing several reads to the same
variable. Maybe we got lucky because the compiler just turns it into 1
load, keeps it in a register and does the various modifications, and
then 1 store to write back. Similar on the reader side, we may have
gotten lucky in that the compiler just does 1 actual load. If that
behaviour is safe, it needs to be made explicit to make it impossible
for the compiler to generate anything else.

> > >  generic_file_read_iter+0x7d/0x3e0 mm/filemap.c:2326
> > >  ext4_file_read_iter+0x2d6/0x420 fs/ext4/file.c:74
> > >  call_read_iter include/linux/fs.h:1902 [inline]
> > >  generic_file_splice_read+0x22a/0x310 fs/splice.c:312
> > >  do_splice_to fs/splice.c:870 [inline]
> > >  splice_direct_to_actor+0x2a8/0x660 fs/splice.c:950
> > >  do_splice_direct+0xf2/0x170 fs/splice.c:1059
> > >  do_sendfile+0x562/0xb10 fs/read_write.c:1540
> > >  __do_sys_sendfile64 fs/read_write.c:1601 [inline]
> > >  __se_sys_sendfile64 fs/read_write.c:1587 [inline]
> > >  __x64_sys_sendfile64+0xf2/0x130 fs/read_write.c:1587
> > >  do_syscall_64+0x51/0xb0 arch/x86/entry/common.c:384
> > >  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > > read to 0xffff8880968747b0 of 8 bytes by task 6334 on cpu 1:
> > >  generic_file_buffered_read+0x11e/0x19e0 mm/filemap.c:2011
> >
> >       ...
> >       index = *ppos >> PAGE_SHIFT;
> >       prev_index = ra->prev_pos >> PAGE_SHIFT;
> > 2011) prev_offset = ra->prev_pos & (PAGE_SIZE-1);
> >       last_index = (*ppos + iter->count + PAGE_SIZE-1) >> PAGE_SHIFT;
> >       offset = *ppos & ~PAGE_MASK;
> >       ...
> >
> > >  generic_file_read_iter+0x7d/0x3e0 mm/filemap.c:2326
> > >  ext4_file_read_iter+0x2d6/0x420 fs/ext4/file.c:74
> > >  call_read_iter include/linux/fs.h:1902 [inline]
> > >  generic_file_splice_read+0x22a/0x310 fs/splice.c:312
> > >  do_splice_to fs/splice.c:870 [inline]
> > >  splice_direct_to_actor+0x2a8/0x660 fs/splice.c:950
> > >  do_splice_direct+0xf2/0x170 fs/splice.c:1059
> > >  do_sendfile+0x562/0xb10 fs/read_write.c:1540
> > >  __do_sys_sendfile64 fs/read_write.c:1601 [inline]
> > >  __se_sys_sendfile64 fs/read_write.c:1587 [inline]
> > >  __x64_sys_sendfile64+0xf2/0x130 fs/read_write.c:1587
> > >  do_syscall_64+0x51/0xb0 arch/x86/entry/common.c:384
> > >  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > > Reported by Kernel Concurrency Sanitizer on:
> > > CPU: 1 PID: 6334 Comm: syz-executor.0 Not tainted 5.8.0-rc5-syzkaller #0
> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > > ==================================================================

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Dave Chinner]

On Wed, Jul 15, 2020 at 09:32:56AM -0700, Eric Biggers wrote:
> [+Cc linux-fsdevel]
> 
> On Wed, Jul 15, 2020 at 05:29:12PM +0200, 'Marco Elver' via syzkaller-bugs wrote:
> > On Wed, Jul 15, 2020 at 08:16AM -0700, syzbot wrote:
> > > Hello,
> > > 
> > > syzbot found the following issue on:
> > > 
> > > HEAD commit:    e9919e11 Merge branch 'for-linus' of git://git.kernel.org/..
> > > git tree:       upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=1217a83b100000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=570eb530a65cd98e
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=0f1e470df6a4316e0a11
> > > compiler:       clang version 11.0.0 (https://github.com/llvm/llvm-project.git ca2dcbd030eadbf0aa9b660efe864ff08af6e18b)
> > > 
> > > Unfortunately, I don't have any reproducer for this issue yet.
> > > 
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+0f1e470df6a4316e0a11@syzkaller.appspotmail.com
> > > 
> > > ==================================================================
> > > BUG: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read
> > 
> > Our guess is that this is either misuse of an API from userspace, or a
> > bug. Can someone clarify?
> > 
> > Below are the snippets of code around these accesses.
> 
> Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
> as that implicitly uses the file position.  But it's allowed with sys_pread(),
> and also with sys_sendfile() which is the case syzbot is reporting here.

Concurrent read()s are fine, they'll just read from the same offset.


> 
> > 
> > > write to 0xffff8880968747b0 of 8 bytes by task 6336 on cpu 0:
> > >  generic_file_buffered_read+0x18be/0x19e0 mm/filemap.c:2246
> > 
> > 	...
> > 	would_block:
> > 		error = -EAGAIN;
> > 	out:
> > 		ra->prev_pos = prev_index;
> > 		ra->prev_pos <<= PAGE_SHIFT;
> > 2246)		ra->prev_pos |= prev_offset;
> > 
> > 		*ppos = ((loff_t)index << PAGE_SHIFT) + offset;
> > 		file_accessed(filp);
> > 		return written ? written : error;
> > 	}
> > 	EXPORT_SYMBOL_GPL(generic_file_buffered_read);
> > 	...
> 
> Well, it's a data race.  Each open file descriptor has just one readahead state
> (struct file_ra_state), and concurrent reads of the same file descriptor
> use/change that readahead state without any locking.
> 
> Presumably this has traditionally been considered okay, since readahead is
> "only" for performance and doesn't affect correctness.  And for performance
> reasons, we want to avoid locking during file reads.
> 
> So we may just need to annotate all access to file_ra_state with
> READ_ONCE() and WRITE_ONCE()...

Please, no. Can we stop making the code hard to read, more difficult
to maintain and preventing the compiler from optimising it by doing
stupid "turn off naive static checker warnings" stuff like this?

If the code is fine with races, then -leave it alone-. If it's not
fine with a data race, then please go and work out the correct
ordering and place well documented barriers and/or release/acquire
ordering semantics in the code so that we do not need to hide data
races behind a compiler optimisation defeating macro....

Yes, I know data_race() exists to tell the tooling that it should
ignore data races in the expression, but that makes just as much
mess of the code as READ_ONCE/WRITE_ONCE being spewed everywhere
indiscriminately because <some tool said we need to do that>.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Eric Biggers]

On Thu, Jul 16, 2020 at 09:42:03AM +1000, Dave Chinner wrote:
> On Wed, Jul 15, 2020 at 09:32:56AM -0700, Eric Biggers wrote:
> > [+Cc linux-fsdevel]
> > 
> > On Wed, Jul 15, 2020 at 05:29:12PM +0200, 'Marco Elver' via syzkaller-bugs wrote:
> > > On Wed, Jul 15, 2020 at 08:16AM -0700, syzbot wrote:
> > > > Hello,
> > > > 
> > > > syzbot found the following issue on:
> > > > 
> > > > HEAD commit:    e9919e11 Merge branch 'for-linus' of git://git.kernel.org/..
> > > > git tree:       upstream
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=1217a83b100000
> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=570eb530a65cd98e
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=0f1e470df6a4316e0a11
> > > > compiler:       clang version 11.0.0 (https://github.com/llvm/llvm-project.git ca2dcbd030eadbf0aa9b660efe864ff08af6e18b)
> > > > 
> > > > Unfortunately, I don't have any reproducer for this issue yet.
> > > > 
> > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > > Reported-by: syzbot+0f1e470df6a4316e0a11@syzkaller.appspotmail.com
> > > > 
> > > > ==================================================================
> > > > BUG: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read
> > > 
> > > Our guess is that this is either misuse of an API from userspace, or a
> > > bug. Can someone clarify?
> > > 
> > > Below are the snippets of code around these accesses.
> > 
> > Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
> > as that implicitly uses the file position.  But it's allowed with sys_pread(),
> > and also with sys_sendfile() which is the case syzbot is reporting here.
> 
> Concurrent read()s are fine, they'll just read from the same offset.
> 

Actually the VFS serializes concurrent read()'s on the same fd, at least for
regular files.  Anyway, doesn't matter since we can consider pread() instead.

> 
> > 
> > > 
> > > > write to 0xffff8880968747b0 of 8 bytes by task 6336 on cpu 0:
> > > >  generic_file_buffered_read+0x18be/0x19e0 mm/filemap.c:2246
> > > 
> > > 	...
> > > 	would_block:
> > > 		error = -EAGAIN;
> > > 	out:
> > > 		ra->prev_pos = prev_index;
> > > 		ra->prev_pos <<= PAGE_SHIFT;
> > > 2246)		ra->prev_pos |= prev_offset;
> > > 
> > > 		*ppos = ((loff_t)index << PAGE_SHIFT) + offset;
> > > 		file_accessed(filp);
> > > 		return written ? written : error;
> > > 	}
> > > 	EXPORT_SYMBOL_GPL(generic_file_buffered_read);
> > > 	...
> > 
> > Well, it's a data race.  Each open file descriptor has just one readahead state
> > (struct file_ra_state), and concurrent reads of the same file descriptor
> > use/change that readahead state without any locking.
> > 
> > Presumably this has traditionally been considered okay, since readahead is
> > "only" for performance and doesn't affect correctness.  And for performance
> > reasons, we want to avoid locking during file reads.
> > 
> > So we may just need to annotate all access to file_ra_state with
> > READ_ONCE() and WRITE_ONCE()...
> 
> Please, no. Can we stop making the code hard to read, more difficult
> to maintain and preventing the compiler from optimising it by doing
> stupid "turn off naive static checker warnings" stuff like this?
> 
> If the code is fine with races, then -leave it alone-. If it's not
> fine with a data race, then please go and work out the correct
> ordering and place well documented barriers and/or release/acquire
> ordering semantics in the code so that we do not need to hide data
> races behind a compiler optimisation defeating macro....
> 
> Yes, I know data_race() exists to tell the tooling that it should
> ignore data races in the expression, but that makes just as much
> mess of the code as READ_ONCE/WRITE_ONCE being spewed everywhere
> indiscriminately because <some tool said we need to do that>.
> 

Data races are undefined behavior, so it's never guaranteed "fine".  We can only
attempt to conclude that it's fine "in practice" and is too difficult to fix,
and therefore doesn't meet the bar to be fixed (for now).

Of course, in most cases the preferred solution for data races is to introduce
proper synchronization.  As I said, I'm not sure that's feasible here.  Memory
barriers aren't the issue here; we'd need *locking*, which would mean concurrent
readers would start contending for the lock.  Other suggestions appreciated...

- Eric

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Nicholas Piggin]

Excerpts from Eric Biggers's message of July 16, 2020 1:03 pm:
> On Thu, Jul 16, 2020 at 09:42:03AM +1000, Dave Chinner wrote:
>> On Wed, Jul 15, 2020 at 09:32:56AM -0700, Eric Biggers wrote:
>> > [+Cc linux-fsdevel]
>> > 
>> > On Wed, Jul 15, 2020 at 05:29:12PM +0200, 'Marco Elver' via syzkaller-bugs wrote:
>> > > On Wed, Jul 15, 2020 at 08:16AM -0700, syzbot wrote:
>> > > > Hello,
>> > > > 
>> > > > syzbot found the following issue on:
>> > > > 
>> > > > HEAD commit:    e9919e11 Merge branch 'for-linus' of git://git.kernel.org/..
>> > > > git tree:       upstream
>> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=1217a83b100000
>> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=570eb530a65cd98e
>> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=0f1e470df6a4316e0a11
>> > > > compiler:       clang version 11.0.0 (https://github.com/llvm/llvm-project.git ca2dcbd030eadbf0aa9b660efe864ff08af6e18b)
>> > > > 
>> > > > Unfortunately, I don't have any reproducer for this issue yet.
>> > > > 
>> > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> > > > Reported-by: syzbot+0f1e470df6a4316e0a11@syzkaller.appspotmail.com
>> > > > 
>> > > > ==================================================================
>> > > > BUG: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read
>> > > 
>> > > Our guess is that this is either misuse of an API from userspace, or a
>> > > bug. Can someone clarify?
>> > > 
>> > > Below are the snippets of code around these accesses.
>> > 
>> > Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
>> > as that implicitly uses the file position.  But it's allowed with sys_pread(),
>> > and also with sys_sendfile() which is the case syzbot is reporting here.
>> 
>> Concurrent read()s are fine, they'll just read from the same offset.
>> 
> 
> Actually the VFS serializes concurrent read()'s on the same fd, at least for
> regular files.

Hmm, where?

> Anyway, doesn't matter since we can consider pread() instead.
> 
>> 
>> > 
>> > > 
>> > > > write to 0xffff8880968747b0 of 8 bytes by task 6336 on cpu 0:
>> > > >  generic_file_buffered_read+0x18be/0x19e0 mm/filemap.c:2246
>> > > 
>> > > 	...
>> > > 	would_block:
>> > > 		error = -EAGAIN;
>> > > 	out:
>> > > 		ra->prev_pos = prev_index;
>> > > 		ra->prev_pos <<= PAGE_SHIFT;
>> > > 2246)		ra->prev_pos |= prev_offset;
>> > > 
>> > > 		*ppos = ((loff_t)index << PAGE_SHIFT) + offset;
>> > > 		file_accessed(filp);
>> > > 		return written ? written : error;
>> > > 	}
>> > > 	EXPORT_SYMBOL_GPL(generic_file_buffered_read);
>> > > 	...
>> > 
>> > Well, it's a data race.  Each open file descriptor has just one readahead state
>> > (struct file_ra_state), and concurrent reads of the same file descriptor
>> > use/change that readahead state without any locking.
>> > 
>> > Presumably this has traditionally been considered okay, since readahead is
>> > "only" for performance and doesn't affect correctness.  And for performance
>> > reasons, we want to avoid locking during file reads.
>> > 
>> > So we may just need to annotate all access to file_ra_state with
>> > READ_ONCE() and WRITE_ONCE()...
>> 
>> Please, no. Can we stop making the code hard to read, more difficult
>> to maintain and preventing the compiler from optimising it by doing
>> stupid "turn off naive static checker warnings" stuff like this?
>> 
>> If the code is fine with races, then -leave it alone-. If it's not
>> fine with a data race, then please go and work out the correct
>> ordering and place well documented barriers and/or release/acquire
>> ordering semantics in the code so that we do not need to hide data
>> races behind a compiler optimisation defeating macro....
>> 
>> Yes, I know data_race() exists to tell the tooling that it should
>> ignore data races in the expression, but that makes just as much
>> mess of the code as READ_ONCE/WRITE_ONCE being spewed everywhere
>> indiscriminately because <some tool said we need to do that>.
>> 
> 
> Data races are undefined behavior, so it's never guaranteed "fine".

Is this a new requirement for the kernel? Even code which is purely an 
optimisation (e.g. a readahead heuristic) can never be guaranteed to
be fine for a data race? As in, the compiler might be free to start
scribbling on memory because of undefined behaviour?

What we used to be able to do is assume that the variable might take on 
one or other value at any time its used (or even see split between the
two if the thing wasn't naturally aligned for example), but that was 
quite well "defined". So we could in fact guarantee that it would be 
fine.

> We can only
> attempt to conclude that it's fine "in practice" and is too difficult to fix,
> and therefore doesn't meet the bar to be fixed (for now).
> 
> Of course, in most cases the preferred solution for data races is to introduce
> proper synchronization.  As I said, I'm not sure that's feasible here.  Memory
> barriers aren't the issue here; we'd need *locking*, which would mean concurrent
> readers would start contending for the lock.  Other suggestions appreciated...


 		ra->prev_pos = prev_index;
 		ra->prev_pos <<= PAGE_SHIFT;
 2246)		ra->prev_pos |= prev_offset;


In this case we can do better I guess, in case some compiler decides to 
store a half-done calculation there because it ran out of registers.

WRITE_ONCE(ra->prev_pos, ((loff_t)prev_index << PAGE_SHIFT) | prev_offset);

As Dave said, adding WRITE_ONCE to the individual accesses would be 
stupid because it does nothing to solve the actual race and makes it 
harder to read in more than one way.

Thanks,
Nick

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Eric Biggers]

On Thu, Jul 16, 2020 at 04:24:01PM +1000, Nicholas Piggin wrote:
> Excerpts from Eric Biggers's message of July 16, 2020 1:03 pm:
> > On Thu, Jul 16, 2020 at 09:42:03AM +1000, Dave Chinner wrote:
> >> On Wed, Jul 15, 2020 at 09:32:56AM -0700, Eric Biggers wrote:
> >> > [+Cc linux-fsdevel]
> >> > 
> >> > On Wed, Jul 15, 2020 at 05:29:12PM +0200, 'Marco Elver' via syzkaller-bugs wrote:
> >> > > On Wed, Jul 15, 2020 at 08:16AM -0700, syzbot wrote:
> >> > > > Hello,
> >> > > > 
> >> > > > syzbot found the following issue on:
> >> > > > 
> >> > > > HEAD commit:    e9919e11 Merge branch 'for-linus' of git://git.kernel.org/..
> >> > > > git tree:       upstream
> >> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=1217a83b100000
> >> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=570eb530a65cd98e
> >> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=0f1e470df6a4316e0a11
> >> > > > compiler:       clang version 11.0.0 (https://github.com/llvm/llvm-project.git ca2dcbd030eadbf0aa9b660efe864ff08af6e18b)
> >> > > > 
> >> > > > Unfortunately, I don't have any reproducer for this issue yet.
> >> > > > 
> >> > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> >> > > > Reported-by: syzbot+0f1e470df6a4316e0a11@syzkaller.appspotmail.com
> >> > > > 
> >> > > > ==================================================================
> >> > > > BUG: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read
> >> > > 
> >> > > Our guess is that this is either misuse of an API from userspace, or a
> >> > > bug. Can someone clarify?
> >> > > 
> >> > > Below are the snippets of code around these accesses.
> >> > 
> >> > Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
> >> > as that implicitly uses the file position.  But it's allowed with sys_pread(),
> >> > and also with sys_sendfile() which is the case syzbot is reporting here.
> >> 
> >> Concurrent read()s are fine, they'll just read from the same offset.
> >> 
> > 
> > Actually the VFS serializes concurrent read()'s on the same fd, at least for
> > regular files.
> 
> Hmm, where?

It's serialized by file->f_pos_lock.  See fdget_pos().

> >> > > > write to 0xffff8880968747b0 of 8 bytes by task 6336 on cpu 0:
> >> > > >  generic_file_buffered_read+0x18be/0x19e0 mm/filemap.c:2246
> >> > > 
> >> > > 	...
> >> > > 	would_block:
> >> > > 		error = -EAGAIN;
> >> > > 	out:
> >> > > 		ra->prev_pos = prev_index;
> >> > > 		ra->prev_pos <<= PAGE_SHIFT;
> >> > > 2246)		ra->prev_pos |= prev_offset;
> >> > > 
> >> > > 		*ppos = ((loff_t)index << PAGE_SHIFT) + offset;
> >> > > 		file_accessed(filp);
> >> > > 		return written ? written : error;
> >> > > 	}
> >> > > 	EXPORT_SYMBOL_GPL(generic_file_buffered_read);
> >> > > 	...
> >> > 
> >> > Well, it's a data race.  Each open file descriptor has just one readahead state
> >> > (struct file_ra_state), and concurrent reads of the same file descriptor
> >> > use/change that readahead state without any locking.
> >> > 
> >> > Presumably this has traditionally been considered okay, since readahead is
> >> > "only" for performance and doesn't affect correctness.  And for performance
> >> > reasons, we want to avoid locking during file reads.
> >> > 
> >> > So we may just need to annotate all access to file_ra_state with
> >> > READ_ONCE() and WRITE_ONCE()...
> >> 
> >> Please, no. Can we stop making the code hard to read, more difficult
> >> to maintain and preventing the compiler from optimising it by doing
> >> stupid "turn off naive static checker warnings" stuff like this?
> >> 
> >> If the code is fine with races, then -leave it alone-. If it's not
> >> fine with a data race, then please go and work out the correct
> >> ordering and place well documented barriers and/or release/acquire
> >> ordering semantics in the code so that we do not need to hide data
> >> races behind a compiler optimisation defeating macro....
> >> 
> >> Yes, I know data_race() exists to tell the tooling that it should
> >> ignore data races in the expression, but that makes just as much
> >> mess of the code as READ_ONCE/WRITE_ONCE being spewed everywhere
> >> indiscriminately because <some tool said we need to do that>.
> >> 
> > 
> > Data races are undefined behavior, so it's never guaranteed "fine".
> 
> Is this a new requirement for the kernel? Even code which is purely an 
> optimisation (e.g. a readahead heuristic) can never be guaranteed to
> be fine for a data race? As in, the compiler might be free to start
> scribbling on memory because of undefined behaviour?
> 
> What we used to be able to do is assume that the variable might take on 
> one or other value at any time its used (or even see split between the
> two if the thing wasn't naturally aligned for example), but that was 
> quite well "defined". So we could in fact guarantee that it would be 
> fine.

Not really, it's always been undefined behavior.

AFAICT, there's tribal knowledge among some kernel developers about what types
of undefined behavior are "okay" because they're thought to be unlikely to cause
problems in practice.  However except in certain cases (e.g., the kernel uses
-fwrapv to make signed integer overflow well-defined, and -fno-strict-aliasing
to make type aliasing well-defined) these cases have never been formally
defined, and people disagree about them.  If they have actually been formally
defined, please point me to the documentation or compiler options.

Data races in particular are tricky because there are a lot of ways for things
to go wrong that people fail to think of; for some examples see:
https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf
https://software.intel.com/content/www/us/en/develop/blogs/benign-data-races-what-could-possibly-go-wrong.html

> > We can only
> > attempt to conclude that it's fine "in practice" and is too difficult to fix,
> > and therefore doesn't meet the bar to be fixed (for now).
> > 
> > Of course, in most cases the preferred solution for data races is to introduce
> > proper synchronization.  As I said, I'm not sure that's feasible here.  Memory
> > barriers aren't the issue here; we'd need *locking*, which would mean concurrent
> > readers would start contending for the lock.  Other suggestions appreciated...
> 
> 
>  		ra->prev_pos = prev_index;
>  		ra->prev_pos <<= PAGE_SHIFT;
>  2246)		ra->prev_pos |= prev_offset;
> 
> 
> In this case we can do better I guess, in case some compiler decides to 
> store a half-done calculation there because it ran out of registers.
> 
> WRITE_ONCE(ra->prev_pos, ((loff_t)prev_index << PAGE_SHIFT) | prev_offset);
> 
> As Dave said, adding WRITE_ONCE to the individual accesses would be 
> stupid because it does nothing to solve the actual race and makes it 
> harder to read in more than one way.

Yes, obviously if we were to add READ/WRITE_ONCE we'd want to avoid storing
intermediate results like that, in order to avoid some obvious race conditions.
However, the overall use of file_ra_state is still racy.  And it's passed to the
functions in mm/readahead.c like page_cache_async_readahead() too, so all the
accesses to it in those functions are data races too.

I'm not really suggesting any specific solution; locking isn't really feasible
here, and there would be an annoyingly large number of places that would need
READ/WRITE_ONCE.

I just wish we had a better plan than "let's write some code with
undefined behavior and hope it's okay".

- Eric

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Nicholas Piggin]

Excerpts from Eric Biggers's message of July 16, 2020 4:54 pm:
> On Thu, Jul 16, 2020 at 04:24:01PM +1000, Nicholas Piggin wrote:
>> Excerpts from Eric Biggers's message of July 16, 2020 1:03 pm:
>> > On Thu, Jul 16, 2020 at 09:42:03AM +1000, Dave Chinner wrote:
>> >> On Wed, Jul 15, 2020 at 09:32:56AM -0700, Eric Biggers wrote:
>> >> > [+Cc linux-fsdevel]
>> >> > 
>> >> > On Wed, Jul 15, 2020 at 05:29:12PM +0200, 'Marco Elver' via syzkaller-bugs wrote:
>> >> > > On Wed, Jul 15, 2020 at 08:16AM -0700, syzbot wrote:
>> >> > > > Hello,
>> >> > > > 
>> >> > > > syzbot found the following issue on:
>> >> > > > 
>> >> > > > HEAD commit:    e9919e11 Merge branch 'for-linus' of git://git.kernel.org/..
>> >> > > > git tree:       upstream
>> >> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=1217a83b100000
>> >> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=570eb530a65cd98e
>> >> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=0f1e470df6a4316e0a11
>> >> > > > compiler:       clang version 11.0.0 (https://github.com/llvm/llvm-project.git ca2dcbd030eadbf0aa9b660efe864ff08af6e18b)
>> >> > > > 
>> >> > > > Unfortunately, I don't have any reproducer for this issue yet.
>> >> > > > 
>> >> > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> >> > > > Reported-by: syzbot+0f1e470df6a4316e0a11@syzkaller.appspotmail.com
>> >> > > > 
>> >> > > > ==================================================================
>> >> > > > BUG: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read
>> >> > > 
>> >> > > Our guess is that this is either misuse of an API from userspace, or a
>> >> > > bug. Can someone clarify?
>> >> > > 
>> >> > > Below are the snippets of code around these accesses.
>> >> > 
>> >> > Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
>> >> > as that implicitly uses the file position.  But it's allowed with sys_pread(),
>> >> > and also with sys_sendfile() which is the case syzbot is reporting here.
>> >> 
>> >> Concurrent read()s are fine, they'll just read from the same offset.
>> >> 
>> > 
>> > Actually the VFS serializes concurrent read()'s on the same fd, at least for
>> > regular files.
>> 
>> Hmm, where?
> 
> It's serialized by file->f_pos_lock.  See fdget_pos().

Ah thanks! Missed that.

>> >> > > > write to 0xffff8880968747b0 of 8 bytes by task 6336 on cpu 0:
>> >> > > >  generic_file_buffered_read+0x18be/0x19e0 mm/filemap.c:2246
>> >> > > 
>> >> > > 	...
>> >> > > 	would_block:
>> >> > > 		error = -EAGAIN;
>> >> > > 	out:
>> >> > > 		ra->prev_pos = prev_index;
>> >> > > 		ra->prev_pos <<= PAGE_SHIFT;
>> >> > > 2246)		ra->prev_pos |= prev_offset;
>> >> > > 
>> >> > > 		*ppos = ((loff_t)index << PAGE_SHIFT) + offset;
>> >> > > 		file_accessed(filp);
>> >> > > 		return written ? written : error;
>> >> > > 	}
>> >> > > 	EXPORT_SYMBOL_GPL(generic_file_buffered_read);
>> >> > > 	...
>> >> > 
>> >> > Well, it's a data race.  Each open file descriptor has just one readahead state
>> >> > (struct file_ra_state), and concurrent reads of the same file descriptor
>> >> > use/change that readahead state without any locking.
>> >> > 
>> >> > Presumably this has traditionally been considered okay, since readahead is
>> >> > "only" for performance and doesn't affect correctness.  And for performance
>> >> > reasons, we want to avoid locking during file reads.
>> >> > 
>> >> > So we may just need to annotate all access to file_ra_state with
>> >> > READ_ONCE() and WRITE_ONCE()...
>> >> 
>> >> Please, no. Can we stop making the code hard to read, more difficult
>> >> to maintain and preventing the compiler from optimising it by doing
>> >> stupid "turn off naive static checker warnings" stuff like this?
>> >> 
>> >> If the code is fine with races, then -leave it alone-. If it's not
>> >> fine with a data race, then please go and work out the correct
>> >> ordering and place well documented barriers and/or release/acquire
>> >> ordering semantics in the code so that we do not need to hide data
>> >> races behind a compiler optimisation defeating macro....
>> >> 
>> >> Yes, I know data_race() exists to tell the tooling that it should
>> >> ignore data races in the expression, but that makes just as much
>> >> mess of the code as READ_ONCE/WRITE_ONCE being spewed everywhere
>> >> indiscriminately because <some tool said we need to do that>.
>> >> 
>> > 
>> > Data races are undefined behavior, so it's never guaranteed "fine".
>> 
>> Is this a new requirement for the kernel? Even code which is purely an 
>> optimisation (e.g. a readahead heuristic) can never be guaranteed to
>> be fine for a data race? As in, the compiler might be free to start
>> scribbling on memory because of undefined behaviour?
>> 
>> What we used to be able to do is assume that the variable might take on 
>> one or other value at any time its used (or even see split between the
>> two if the thing wasn't naturally aligned for example), but that was 
>> quite well "defined". So we could in fact guarantee that it would be 
>> fine.
> 
> Not really, it's always been undefined behavior.
> 
> AFAICT, there's tribal knowledge among some kernel developers about what types
> of undefined behavior are "okay" because they're thought to be unlikely to cause
> problems in practice.  However except in certain cases (e.g., the kernel uses
> -fwrapv to make signed integer overflow well-defined, and -fno-strict-aliasing
> to make type aliasing well-defined) these cases have never been formally
> defined, and people disagree about them.  If they have actually been formally
> defined, please point me to the documentation or compiler options.

Well we did traditionally say stores to natural aligned word types and
smaller were atomic (although being loff_t may not be true for 32-bit).
Kernel behaviour, rather than C (which as you say is not kernel 
semantics).

> 
> Data races in particular are tricky because there are a lot of ways for things
> to go wrong that people fail to think of; for some examples see:
> https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf
> https://software.intel.com/content/www/us/en/develop/blogs/benign-data-races-what-could-possibly-go-wrong.html

If we abandon that and go with always explicit accessors okay. But none 
of those are things that surprise the kernel model except this one

"So if a program stores to a variable X, the compiler can legally reuse 
X’s storage for any temporal data inside of some region of code before 
the store (e.g. to reduce stack frame size)."

Which is wrong and we'd never tolerate it in the kernel. We don't just
race with other threads but also our interrupts. preempt_enable()
called somewhere can't allow the compiler to enable preemption by
spilling zero to preempt_count in code before the call, for example.

So that would be disabled exactly the same as other insanity.

The only argument really is for race checkers.

> 
>> > We can only
>> > attempt to conclude that it's fine "in practice" and is too difficult to fix,
>> > and therefore doesn't meet the bar to be fixed (for now).
>> > 
>> > Of course, in most cases the preferred solution for data races is to introduce
>> > proper synchronization.  As I said, I'm not sure that's feasible here.  Memory
>> > barriers aren't the issue here; we'd need *locking*, which would mean concurrent
>> > readers would start contending for the lock.  Other suggestions appreciated...
>> 
>> 
>>  		ra->prev_pos = prev_index;
>>  		ra->prev_pos <<= PAGE_SHIFT;
>>  2246)		ra->prev_pos |= prev_offset;
>> 
>> 
>> In this case we can do better I guess, in case some compiler decides to 
>> store a half-done calculation there because it ran out of registers.
>> 
>> WRITE_ONCE(ra->prev_pos, ((loff_t)prev_index << PAGE_SHIFT) | prev_offset);
>> 
>> As Dave said, adding WRITE_ONCE to the individual accesses would be 
>> stupid because it does nothing to solve the actual race and makes it 
>> harder to read in more than one way.
> 
> Yes, obviously if we were to add READ/WRITE_ONCE we'd want to avoid storing
> intermediate results like that, in order to avoid some obvious race conditions.

Well the suggestion was to just simply add READ/WRITE once to all 
accesses, not to fix them up. That would actually add more race
conditions.

> However, the overall use of file_ra_state is still racy.  And it's passed to the
> functions in mm/readahead.c like page_cache_async_readahead() too, so all the
> accesses to it in those functions are data races too.
> 
> I'm not really suggesting any specific solution; locking isn't really feasible
> here, and there would be an annoyingly large number of places that would need
> READ/WRITE_ONCE.

If you put behind some accessor functions it might become easier,
but...

> I just wish we had a better plan than "let's write some code with
> undefined behavior and hope it's okay".

It really isn't so undefined as you think. Again, we enforce against
insane compilers de facto if not written anywhere with our interrupt
races. So we really can guarantee it'll be okay.

Thanks,
Nick

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Marco Elver]

On Thu, 16 Jul 2020 at 09:52, Nicholas Piggin <npiggin@gmail.com> wrote:
>
> Excerpts from Eric Biggers's message of July 16, 2020 4:54 pm:
> > On Thu, Jul 16, 2020 at 04:24:01PM +1000, Nicholas Piggin wrote:
> >> Excerpts from Eric Biggers's message of July 16, 2020 1:03 pm:
> >> > On Thu, Jul 16, 2020 at 09:42:03AM +1000, Dave Chinner wrote:
> >> >> On Wed, Jul 15, 2020 at 09:32:56AM -0700, Eric Biggers wrote:
[...]
> >> >> > > > ==================================================================
> >> >> > > > BUG: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read
[...]
> >> >> > Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
> >> >> > as that implicitly uses the file position.  But it's allowed with sys_pread(),
> >> >> > and also with sys_sendfile() which is the case syzbot is reporting here.
> >> >>
> >> >> Concurrent read()s are fine, they'll just read from the same offset.
> >> >
> >> > Actually the VFS serializes concurrent read()'s on the same fd, at least for
> >> > regular files.
> >>
> >> Hmm, where?
> >
> > It's serialized by file->f_pos_lock.  See fdget_pos().
>
> Ah thanks! Missed that.
>
> >> >> > > > write to 0xffff8880968747b0 of 8 bytes by task 6336 on cpu 0:
> >> >> > > >  generic_file_buffered_read+0x18be/0x19e0 mm/filemap.c:2246
> >> >> > >
> >> >> > >       ...
> >> >> > >       would_block:
> >> >> > >               error = -EAGAIN;
> >> >> > >       out:
> >> >> > >               ra->prev_pos = prev_index;
> >> >> > >               ra->prev_pos <<= PAGE_SHIFT;
> >> >> > > 2246)         ra->prev_pos |= prev_offset;
> >> >> > >
> >> >> > >               *ppos = ((loff_t)index << PAGE_SHIFT) + offset;
> >> >> > >               file_accessed(filp);
> >> >> > >               return written ? written : error;
> >> >> > >       }
> >> >> > >       EXPORT_SYMBOL_GPL(generic_file_buffered_read);
> >> >> > >       ...
> >> >> >
> >> >> > Well, it's a data race.  Each open file descriptor has just one readahead state
> >> >> > (struct file_ra_state), and concurrent reads of the same file descriptor
> >> >> > use/change that readahead state without any locking.
> >> >> >
> >> >> > Presumably this has traditionally been considered okay, since readahead is
> >> >> > "only" for performance and doesn't affect correctness.  And for performance
> >> >> > reasons, we want to avoid locking during file reads.
> >> >> >
> >> >> > So we may just need to annotate all access to file_ra_state with
> >> >> > READ_ONCE() and WRITE_ONCE()...
> >> >>
> >> >> Please, no. Can we stop making the code hard to read, more difficult
> >> >> to maintain and preventing the compiler from optimising it by doing
> >> >> stupid "turn off naive static checker warnings" stuff like this?
> >> >>
> >> >> If the code is fine with races, then -leave it alone-. If it's not
> >> >> fine with a data race, then please go and work out the correct
> >> >> ordering and place well documented barriers and/or release/acquire
> >> >> ordering semantics in the code so that we do not need to hide data
> >> >> races behind a compiler optimisation defeating macro....
> >> >>
> >> >> Yes, I know data_race() exists to tell the tooling that it should
> >> >> ignore data races in the expression, but that makes just as much
> >> >> mess of the code as READ_ONCE/WRITE_ONCE being spewed everywhere
> >> >> indiscriminately because <some tool said we need to do that>.
> >> >>
> >> >
> >> > Data races are undefined behavior, so it's never guaranteed "fine".
> >>
> >> Is this a new requirement for the kernel? Even code which is purely an
> >> optimisation (e.g. a readahead heuristic) can never be guaranteed to
> >> be fine for a data race? As in, the compiler might be free to start
> >> scribbling on memory because of undefined behaviour?
> >>
> >> What we used to be able to do is assume that the variable might take on
> >> one or other value at any time its used (or even see split between the
> >> two if the thing wasn't naturally aligned for example), but that was
> >> quite well "defined". So we could in fact guarantee that it would be
> >> fine.
> >
> > Not really, it's always been undefined behavior.
> >
> > AFAICT, there's tribal knowledge among some kernel developers about what types
> > of undefined behavior are "okay" because they're thought to be unlikely to cause
> > problems in practice.  However except in certain cases (e.g., the kernel uses
> > -fwrapv to make signed integer overflow well-defined, and -fno-strict-aliasing
> > to make type aliasing well-defined) these cases have never been formally
> > defined, and people disagree about them.  If they have actually been formally
> > defined, please point me to the documentation or compiler options.
>
> Well we did traditionally say stores to natural aligned word types and
> smaller were atomic (although being loff_t may not be true for 32-bit).
> Kernel behaviour, rather than C (which as you say is not kernel
> semantics).

This is one assumption that KCSAN is definitely aware of and is
included in its default config. I'd still prefer a WRITE_ONCE(), as
I'm a little more paranoid of things breaking on some arch with some
compiler, and as it documents the fact there is concurrency. In the
end nobody is forcing anything yet.

> > Data races in particular are tricky because there are a lot of ways for things
> > to go wrong that people fail to think of; for some examples see:
> > https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf
> > https://software.intel.com/content/www/us/en/develop/blogs/benign-data-races-what-could-possibly-go-wrong.html

I'll add to this, our recent kernel-specific summary,
  https://lwn.net/Articles/816850/#Why%20should%20we%20care%20about%20data%20races?
and
  https://lwn.net/Articles/793253/

In short, the kernel makes its own rules and are meant to be captured
in the LKMM.

Data races in the kernel are tolerable in certain codes, but due to
the subtleness in which things can break, we ought to try and document
intentional data races with 'data_race(..)'. Because otherwise, there
is no telling if it was intentional or not.

> If we abandon that and go with always explicit accessors okay. But none
> of those are things that surprise the kernel model except this one
>
> "So if a program stores to a variable X, the compiler can legally reuse
> X’s storage for any temporal data inside of some region of code before
> the store (e.g. to reduce stack frame size)."
>
> Which is wrong and we'd never tolerate it in the kernel. We don't just
> race with other threads but also our interrupts. preempt_enable()
> called somewhere can't allow the compiler to enable preemption by
> spilling zero to preempt_count in code before the call, for example.
>
> So that would be disabled exactly the same as other insanity.

Indeed, the LKMM tries to capture some of the kernel's rules, in
effect we do roll our own standard. However, because of the disconnect
between what the compiler is aware of and what we merely assume, we
need to be extra careful. Some things can be enforced with more
compiler flags (but not everything).

> The only argument really is for race checkers.

The race checker, in this case KCSAN, does follow as closely as
possible to the kernel's written rules (i.e. LKMM), but also some
unwritten rules ("assume aligned writes up to word size are atomic")
by default. What other unwritten rules are we missing?

> >> > We can only
> >> > attempt to conclude that it's fine "in practice" and is too difficult to fix,
> >> > and therefore doesn't meet the bar to be fixed (for now).
> >> >
> >> > Of course, in most cases the preferred solution for data races is to introduce
> >> > proper synchronization.  As I said, I'm not sure that's feasible here.  Memory
> >> > barriers aren't the issue here; we'd need *locking*, which would mean concurrent
> >> > readers would start contending for the lock.  Other suggestions appreciated...
> >>
> >>
> >>              ra->prev_pos = prev_index;
> >>              ra->prev_pos <<= PAGE_SHIFT;
> >>  2246)               ra->prev_pos |= prev_offset;
> >>
> >>
> >> In this case we can do better I guess, in case some compiler decides to
> >> store a half-done calculation there because it ran out of registers.
> >>
> >> WRITE_ONCE(ra->prev_pos, ((loff_t)prev_index << PAGE_SHIFT) | prev_offset);
> >>
> >> As Dave said, adding WRITE_ONCE to the individual accesses would be
> >> stupid because it does nothing to solve the actual race and makes it
> >> harder to read in more than one way.
> >
> > Yes, obviously if we were to add READ/WRITE_ONCE we'd want to avoid storing
> > intermediate results like that, in order to avoid some obvious race conditions.
>
> Well the suggestion was to just simply add READ/WRITE once to all
> accesses, not to fix them up. That would actually add more race
> conditions.
>
> > However, the overall use of file_ra_state is still racy.  And it's passed to the
> > functions in mm/readahead.c like page_cache_async_readahead() too, so all the
> > accesses to it in those functions are data races too.
> >
> > I'm not really suggesting any specific solution; locking isn't really feasible
> > here, and there would be an annoyingly large number of places that would need
> > READ/WRITE_ONCE.
>
> If you put behind some accessor functions it might become easier,
> but...
>
> > I just wish we had a better plan than "let's write some code with
> > undefined behavior and hope it's okay".
>
> It really isn't so undefined as you think. Again, we enforce against
> insane compilers de facto if not written anywhere with our interrupt
> races. So we really can guarantee it'll be okay.

Agreed that we make our own rules, for better or worse. But I think
even within these rules and assumptions, the case around the
'prev_pos' accesses just isn't safe. In particular, these plain
compound ops / read-modify-writes just aren't guaranteed to be atomic,
even with tons of compiler flags.

Thanks,
-- Marco

Re: KCSAN: data-race in generic_file_buffered_read / generic_file_buffered_read

[Matthew Wilcox]

On Wed, Jul 15, 2020 at 11:54:54PM -0700, Eric Biggers wrote:
> > >> > Concurrent reads on the same file descriptor are allowed.  Not with sys_read(),
> > >> > as that implicitly uses the file position.  But it's allowed with sys_pread(),
> > >> > and also with sys_sendfile() which is the case syzbot is reporting here.
> > >> 
> > >> Concurrent read()s are fine, they'll just read from the same offset.
> > >> 
> > > 
> > > Actually the VFS serializes concurrent read()'s on the same fd, at least for
> > > regular files.
> > 
> > Hmm, where?
> 
> It's serialized by file->f_pos_lock.  See fdget_pos().

What if we trylock either f_lock or f_pos_lock for readahead and just
skip all the readahead code if it's already taken?  I'd suggest that if
there are two readers using the same struct file, this is probably not
a workload that benefits greatly from readahead.