Cache coherency bug: stale reads on /dev/sda1

Cache coherency bug: stale reads on /dev/sda1

[Joshua Hudson]

uname -a
Linux nova 6.4.12 #1 SMP PREEMPT_DYNAMIC Sat Aug 26 09:11:27 PDT 2023
x86_64 GNU/Linux

Kernel source is
https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.4.12.tar.xz

Can reliably reproduce as follows:

# hexedit /dev/sda1 (this is my EFI partition)
PgDn
# hexedit /dev/sda
G 100000  (your value may vary--we want to seek to the start of the
EFI partition)
PgDn
PgDn
^C
write a marker to the padding between the BPB and the first FAT sector.
(If your system doesn't have one, edit an error message in the boot sector)
^X
# hexedit /dev/sda1
PgDn
Look for marker written above, find it's not there !!!
^C

I discovered this one trying to defragment my EFI partition after a
grub upgrade left it very fragmented due to replacing logos.

Re: Cache coherency bug: stale reads on /dev/sda1

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

On Sat, Aug 26, 2023 at 02:59:45PM -0700, Joshua Hudson wrote:
> uname -a
> Linux nova 6.4.12 #1 SMP PREEMPT_DYNAMIC Sat Aug 26 09:11:27 PDT 2023
> x86_64 GNU/Linux
> 
> Kernel source is
> https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.4.12.tar.xz
> 
> Can reliably reproduce as follows:
> 
> # hexedit /dev/sda1 (this is my EFI partition)
> PgDn
> # hexedit /dev/sda
> G 100000  (your value may vary--we want to seek to the start of the
> EFI partition)
> PgDn
> PgDn
> ^C
> write a marker to the padding between the BPB and the first FAT sector.
> (If your system doesn't have one, edit an error message in the boot sector)
> ^X
> # hexedit /dev/sda1
> PgDn
> Look for marker written above, find it's not there !!!
> ^C
> 
> I discovered this one trying to defragment my EFI partition after a
> grub upgrade left it very fragmented due to replacing logos.

Are you sure this is hardware issue?

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Cache coherency bug: stale reads on /dev/sda1

[Joshua Hudson]

I am presuming you mean "Are you sure this not a hardware issue?"

I am sure it is not, for two reasons.

1) If it were a hardware issue I would still expect the two device
nodes (whole disk and partition) to report the *same* data.

2) I have since developed a workaround involving BLKFLSDEV. The
workaround is really ugly.

On Sat, Aug 26, 2023 at 6:03 PM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>
> On Sat, Aug 26, 2023 at 02:59:45PM -0700, Joshua Hudson wrote:
> > uname -a
> > Linux nova 6.4.12 #1 SMP PREEMPT_DYNAMIC Sat Aug 26 09:11:27 PDT 2023
> > x86_64 GNU/Linux
> >
> > Kernel source is
> > https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.4.12.tar.xz
> >
> > Can reliably reproduce as follows:
> >
> > # hexedit /dev/sda1 (this is my EFI partition)
> > PgDn
> > # hexedit /dev/sda
> > G 100000  (your value may vary--we want to seek to the start of the
> > EFI partition)
> > PgDn
> > PgDn
> > ^C
> > write a marker to the padding between the BPB and the first FAT sector.
> > (If your system doesn't have one, edit an error message in the boot sector)
> > ^X
> > # hexedit /dev/sda1
> > PgDn
> > Look for marker written above, find it's not there !!!
> > ^C
> >
> > I discovered this one trying to defragment my EFI partition after a
> > grub upgrade left it very fragmented due to replacing logos.
>
> Are you sure this is hardware issue?
>
> --
> An old man doll... just what I always wanted! - Clara

Re: Cache coherency bug: stale reads on /dev/sda1

[Al Viro]

On Sat, Aug 26, 2023 at 06:25:58PM -0700, Joshua Hudson wrote:
> I am presuming you mean "Are you sure this not a hardware issue?"
> 
> I am sure it is not, for two reasons.
> 
> 1) If it were a hardware issue I would still expect the two device
> nodes (whole disk and partition) to report the *same* data.
> 
> 2) I have since developed a workaround involving BLKFLSDEV. The
> workaround is really ugly.

Whole disk and all partitions have page caches of their own.
There is no coherency between those; write through one, then
read through another and you are not guaranteed that read won't
see the stale cached data.

Doctor, it hurts when I do it...

Incidentally, read from device/write to the place you've read via
file/read from device again also has no coherency warranties, for
exact same reason.

IOW, not a bug.

Re: Cache coherency bug: stale reads on /dev/sda1

[Joshua Hudson]

"Whole disk and all partitions have page caches of their own."

That's so bad.

I can think of numerous cases where this will cause problems; including
some I encountered last year and did not understand at the time. Manipulating
EFI partitions through the whole disk device makes sense because FAT filesystems
*know their offset on the disk*, and some of the existing tools really
don't like being
given a partition device.

There's also the astounding: write stuff to disk, umount everything,
copy one disk to
another using the whole disk device doesn't work because reading the whole disk
yields a stale cache (sometimes).

On the other hand, I can think of very few cases where the file vs
disk buffer pool
matters, because the loop device is unaffected (writing to a loop
block device is
coherent with the file).

On Sat, Aug 26, 2023 at 7:06 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Sat, Aug 26, 2023 at 06:25:58PM -0700, Joshua Hudson wrote:
> > I am presuming you mean "Are you sure this not a hardware issue?"
> >
> > I am sure it is not, for two reasons.
> >
> > 1) If it were a hardware issue I would still expect the two device
> > nodes (whole disk and partition) to report the *same* data.
> >
> > 2) I have since developed a workaround involving BLKFLSDEV. The
> > workaround is really ugly.
>
> Whole disk and all partitions have page caches of their own.
> There is no coherency between those; write through one, then
> read through another and you are not guaranteed that read won't
> see the stale cached data.
>
> Doctor, it hurts when I do it...
>
> Incidentally, read from device/write to the place you've read via
> file/read from device again also has no coherency warranties, for
> exact same reason.
>
> IOW, not a bug.

Re: Cache coherency bug: stale reads on /dev/sda1

[Bagas Sanjaya]

On 27/08/2023 09:38, Joshua Hudson wrote:
> "Whole disk and all partitions have page caches of their own."
> 
> That's so bad.
> 
> I can think of numerous cases where this will cause problems; including
> some I encountered last year and did not understand at the time. Manipulating
> EFI partitions through the whole disk device makes sense because FAT filesystems
> *know their offset on the disk*, and some of the existing tools really
> don't like being
> given a partition device.
> 
> There's also the astounding: write stuff to disk, umount everything,
> copy one disk to
> another using the whole disk device doesn't work because reading the whole disk
> yields a stale cache (sometimes).
> 
> On the other hand, I can think of very few cases where the file vs
> disk buffer pool
> matters, because the loop device is unaffected (writing to a loop
> block device is
> coherent with the file).
> 

tl;dr:

> A: http://en.wikipedia.org/wiki/Top_post
> Q: Were do I find info about this thing called top-posting?
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
> 
> A: No.
> Q: Should I include quotations after my reply?
> 
> http://daringfireball.net/2007/07/on_top

What cases on the loop devices?

-- 
An old man doll... just what I always wanted! - Clara

Re: Cache coherency bug: stale reads on /dev/sda1

[Al Viro]

On Sat, Aug 26, 2023 at 07:38:57PM -0700, Joshua Hudson wrote:
> "Whole disk and all partitions have page caches of their own."
> 
> That's so bad.
> 
> I can think of numerous cases where this will cause problems; including
> some I encountered last year and did not understand at the time. Manipulating
> EFI partitions through the whole disk device makes sense because FAT filesystems
> *know their offset on the disk*, and some of the existing tools really
> don't like being
> given a partition device.

Explain, please.  How does FAT filesystem know its offset on disk?
Since when?  It had always been possible to copy a FAT image into
a partition verbatim and it works no matter where on disk that
partition happens to be...

Has that changed at some point?  Do you have any references?  Ideally
with some kind of rationale for that weirdness...

Or am I misparsing what you wrote?

Re: Cache coherency bug: stale reads on /dev/sda1

[Joshua Hudson]

On Sat, Aug 26, 2023 at 9:17 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Sat, Aug 26, 2023 at 07:38:57PM -0700, Joshua Hudson wrote:
> > "Whole disk and all partitions have page caches of their own."
> >
> > That's so bad.
> >
> > I can think of numerous cases where this will cause problems; including
> > some I encountered last year and did not understand at the time. Manipulating
> > EFI partitions through the whole disk device makes sense because FAT filesystems
> > *know their offset on the disk*, and some of the existing tools really
> > don't like being
> > given a partition device.
>
> Explain, please.  How does FAT filesystem know its offset on disk?
> Since when?  It had always been possible to copy a FAT image into
> a partition verbatim and it works no matter where on disk that
> partition happens to be...
>
> Has that changed at some point?  Do you have any references?  Ideally
> with some kind of rationale for that weirdness...
>
> Or am I misparsing what you wrote?

(Good news: finally found the invisible button to edit reply quote)

Offset 0x1C into the FAT filesystem is defined as "Count of hidden
sectors preceding
the partition that contains this FAT volume." It's been there since
DOS 3.0. The Linux
Kernel does not care what's in this field, but I have tools that have
a hard time of it
not being there. One example of a tool is BootDuet.

The rationale is to reduce the number of tools that have to walk the
partition table, the
most obvious one being the boot sector itself which doesn't have room.

With random BIOS bugs in UEFI still; I would *not* want to find out
what happens with
a wrong value here on an EFI partition.

Source: https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system

(The field changed size; you actually want to read the entry under DOS 3.31)

Re: Cache coherency bug: stale reads on /dev/sda1

[Al Viro]

On Sun, Aug 27, 2023 at 07:39:03AM -0700, Joshua Hudson wrote:
> (Good news: finally found the invisible button to edit reply quote)
> 
> Offset 0x1C into the FAT filesystem is defined as "Count of hidden
> sectors preceding
> the partition that contains this FAT volume." It's been there since
> DOS 3.0. The Linux
> Kernel does not care what's in this field, but I have tools that have
> a hard time of it
> not being there. One example of a tool is BootDuet.
> 
> The rationale is to reduce the number of tools that have to walk the
> partition table, the
> most obvious one being the boot sector itself which doesn't have room.

???

That makes no sense whatsoever; "boot sector" here is the first sector
of _partition_, not that of the entire disk (that would be MBR).

To quote the same wikipedia,
----
A volume boot record (VBR) (also known as a volume boot sector,
a partition boot record or a partition boot sector) is a type of boot
sector introduced by the IBM Personal Computer. It may be found on a
partitioned data storage device, such as a hard disk, or an unpartitioned
device, such as a floppy disk, and contains machine code for bootstrapping
programs (usually, but not necessarily, operating systems) stored in other
parts of the device. On non-partitioned storage devices, it is the first
sector of the device. On partitioned devices, it is the first sector of
an individual partition on the device, with the first sector of the entire
device being a Master Boot Record (MBR) containing the partition table.
----

So your rationale doesn't work - you need to know where the partition is
just to read the sector that contains that field.  Or have access to
something that can be asked to read from partition, as opposed to the
entire disk (kernel, for example), but that something can usually be
asked where the hell does partition start.

On anything with UEFI the first sector of the entire disk is likely to be
the "protective MBR" in the beginning of GPT.  No BPB in that on in real
MBR, and in any case - which of the FAT filesystems would it refer to?

Not familiar with BootDuet and currently there's a lot on my platter,
so I'm not about to go and RTFS through it.  Your description really
doesn't seem to make any sense, though...

Re: Cache coherency bug: stale reads on /dev/sda1

[Joshua Hudson]

On Sun, Aug 27, 2023 at 9:30 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Sun, Aug 27, 2023 at 07:39:03AM -0700, Joshua Hudson wrote:
> > Offset 0x1C into the FAT filesystem is defined as "Count of hidden
> > sectors preceding
> > the partition that contains this FAT volume." It's been there since
> > DOS 3.0. The Linux
> > Kernel does not care what's in this field, but I have tools that have
> > a hard time of it
> > not being there. One example of a tool is BootDuet.
> >
> > The rationale is to reduce the number of tools that have to walk the
> > partition table, the
> > most obvious one being the boot sector itself which doesn't have room.
>
> ???
>
> That makes no sense whatsoever; "boot sector" here is the first sector
> of _partition_, not that of the entire disk (that would be MBR).
>
> To quote the same wikipedia,
> ----
> A volume boot record (VBR) (also known as a volume boot sector,
> a partition boot record or a partition boot sector) is a type of boot
> sector introduced by the IBM Personal Computer. It may be found on a
> partitioned data storage device, such as a hard disk, or an unpartitioned
> device, such as a floppy disk, and contains machine code for bootstrapping
> programs (usually, but not necessarily, operating systems) stored in other
> parts of the device. On non-partitioned storage devices, it is the first
> sector of the device. On partitioned devices, it is the first sector of
> an individual partition on the device, with the first sector of the entire
> device being a Master Boot Record (MBR) containing the partition table.
> ----
>
> So your rationale doesn't work - you need to know where the partition is
> just to read the sector that contains that field.  Or have access to
> something that can be asked to read from partition, as opposed to the
> entire disk (kernel, for example), but that something can usually be
> asked where the hell does partition start.

The MBR doesn't look at the BPB. It looks at the first four bytes of the
partition table entry, loads that sector (which is the VBR), and jumps to
it. In the late 90s it became a thing to pass DL all the way through to
enable booting from ZIP disks, but nothing else is passed from the MBR
to the VBR. Thus, the BPB is in the VBR and describes the filesystem
and knows its own offset (otherwise the disk would not boot).

https://github.com/FDOS/kernel/blob/132a0a9f94d23f13c90319bb0e0232a5f33164d8/boot/boot.asm#L203

Almost everybody depended on this exact behavior for dual booting. Each
OS was defined in the boot menu as a path to a 512 byte file, which was
that OS's boot sector. They don't access disk relative to their own position,
but rather what their position should be.

So what I did: unmounted /boot/efi, started a virtual machine with /dev/sda
as the disk, booted a different OS than the host OS, shut it down, and got
very confused when changes to /boot/efi via the guest OS were not reflected
in the host OS.

>
> On anything with UEFI the first sector of the entire disk is likely to be
> the "protective MBR" in the beginning of GPT.
>

# dd if=/dev/sda bs=c skip=446 count=66 | hexdump -C
00000000  00 00 02 00 ee ff ff ff  01 00 00 00 af 88 e0 e8  |....îÿÿÿ....¯.àè|
00000010  80 20 21 00 06 df 13 0c  00 08 00 00 00 20 03 00  |. !..ß....... ..|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  55 aa                                             |Uª|
00000042

I wouldn't be too sure of that. And yes, the second line corresponds
exactly to the EFI partition's entry in GPT.

Re: Cache coherency bug: stale reads on /dev/sda1

[Hannes Reinecke]

On 8/27/23 20:33, Joshua Hudson wrote:
> On Sun, Aug 27, 2023 at 9:30 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
>>
>> On Sun, Aug 27, 2023 at 07:39:03AM -0700, Joshua Hudson wrote:
>>> Offset 0x1C into the FAT filesystem is defined as "Count of hidden
>>> sectors preceding
>>> the partition that contains this FAT volume." It's been there since
>>> DOS 3.0. The Linux
>>> Kernel does not care what's in this field, but I have tools that have
>>> a hard time of it
>>> not being there. One example of a tool is BootDuet.
>>>
>>> The rationale is to reduce the number of tools that have to walk the
>>> partition table, the
>>> most obvious one being the boot sector itself which doesn't have room.
>>
>> ???
>>
>> That makes no sense whatsoever; "boot sector" here is the first sector
>> of _partition_, not that of the entire disk (that would be MBR).
>>
>> To quote the same wikipedia,
>> ----
>> A volume boot record (VBR) (also known as a volume boot sector,
>> a partition boot record or a partition boot sector) is a type of boot
>> sector introduced by the IBM Personal Computer. It may be found on a
>> partitioned data storage device, such as a hard disk, or an unpartitioned
>> device, such as a floppy disk, and contains machine code for bootstrapping
>> programs (usually, but not necessarily, operating systems) stored in other
>> parts of the device. On non-partitioned storage devices, it is the first
>> sector of the device. On partitioned devices, it is the first sector of
>> an individual partition on the device, with the first sector of the entire
>> device being a Master Boot Record (MBR) containing the partition table.
>> ----
>>
>> So your rationale doesn't work - you need to know where the partition is
>> just to read the sector that contains that field.  Or have access to
>> something that can be asked to read from partition, as opposed to the
>> entire disk (kernel, for example), but that something can usually be
>> asked where the hell does partition start.
> 
> The MBR doesn't look at the BPB. It looks at the first four bytes of the
> partition table entry, loads that sector (which is the VBR), and jumps to
> it. In the late 90s it became a thing to pass DL all the way through to
> enable booting from ZIP disks, but nothing else is passed from the MBR
> to the VBR. Thus, the BPB is in the VBR and describes the filesystem
> and knows its own offset (otherwise the disk would not boot).
> 
> https://github.com/FDOS/kernel/blob/132a0a9f94d23f13c90319bb0e0232a5f33164d8/boot/boot.asm#L203
> 
> Almost everybody depended on this exact behavior for dual booting. Each
> OS was defined in the boot menu as a path to a 512 byte file, which was
> that OS's boot sector. They don't access disk relative to their own position,
> but rather what their position should be.
> 
> So what I did: unmounted /boot/efi, started a virtual machine with /dev/sda
> as the disk, booted a different OS than the host OS, shut it down, and got
> very confused when changes to /boot/efi via the guest OS were not reflected
> in the host OS.
> 
>>
>> On anything with UEFI the first sector of the entire disk is likely to be
>> the "protective MBR" in the beginning of GPT.
>>
> 
> # dd if=/dev/sda bs=c skip=446 count=66 | hexdump -C
> 00000000  00 00 02 00 ee ff ff ff  01 00 00 00 af 88 e0 e8  |....îÿÿÿ....¯.àè|
> 00000010  80 20 21 00 06 df 13 0c  00 08 00 00 00 20 03 00  |. !..ß....... ..|
> 00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
> 00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
> 00000040  55 aa                                             |Uª|
> 00000042
> 
> I wouldn't be too sure of that. And yes, the second line corresponds
> exactly to the EFI partition's entry in GPT.

Weelll ... if you already know that there's something in the FAT 
documentation which isn't implemented in Linux, why are you surprised 
that things don't work if you modify it?
And if you know that tools are relying on that specific field, why not 
implement support for that feature in linux?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman