blkid identifying whole disk ntfs as both ntfs and msdos partition table

blkid identifying whole disk ntfs as both ntfs and msdos partition table

[Mike Fleetwood]

Hi,

Tested with mkntfs from ntfsprogs and blkid and wipefs from util-linux git HEAD.
Test case:

# cat /etc/fedora-release
Fedora release 30 (Thirty)
# dd if=/dev/zero bs=1M of=/dev/sdb
dd: error writing '/dev/sdb': No space left on device
8193+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 39.6128 s, 217 MB/s
#  mkntfs --version
mkntfs v2017.3.23 (libntfs-3g)
...
# mkntfs -Q -F /dev/sdb
/dev/sdb is entire device, not just one partition.
mkntfs forced anyway.
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
mkntfs completed successfully. Have a nice day.
# ./blkid --version
lt-blkid from util-linux 2.34.24-e3bb9  (libblkid 2.34.24, 14-Jun-2019)
# ./blkid /dev/sdb
/dev/sdb: UUID="3B64D28A5CF89146" TYPE="ntfs" PTTYPE="dos"
# ./wipefs /dev/sdb
DEVICE OFFSET TYPE UUID             LABEL
sdb    0x3    ntfs 3B64D28A5CF89146
sdb    0x1fe  dos

Note that the test ntfs image in util-linux doesn't cause this.  Just file
systems created by mkntfs from ntfsprogs.

# dd if=/dev/zero bs=1M of=/dev/sdb
# xzcat tests/ts/blkid/images-fs/ntfs.img.xz > /dev/sdb
# ./blkid /dev/sdb
/dev/sdb: LABEL="M-PM-^]M-PM->M-PM-2M-QM-^KM-PM-9 M-QM-^BM-PM->M-PM-<"
UUID="09CBB6DE30C87310" TYPE="ntfs"
# ./wipefs /dev/sdb
DEVICE OFFSET TYPE UUID             LABEL
sdb    0x3    ntfs 09CBB6DE30C87310 Новый том

Thanks,
Mike

Re: blkid identifying whole disk ntfs as both ntfs and msdos partition table

[Theodore Ts'o]

On Tue, Jul 02, 2019 at 11:43:34AM +0100, Mike Fleetwood wrote:
> Hi,
> 
> Tested with mkntfs from ntfsprogs and blkid and wipefs from util-linux git HEAD.
> Test case:
> 
> # mkntfs -Q -F /dev/sdb
> /dev/sdb is entire device, not just one partition.
> mkntfs forced anyway.
> Cluster size has been automatically set to 4096 bytes.
> Creating NTFS volume structures.
> mkntfs completed successfully. Have a nice day.
> # ./blkid --version
> lt-blkid from util-linux 2.34.24-e3bb9  (libblkid 2.34.24, 14-Jun-2019)
> # ./blkid /dev/sdb
> /dev/sdb: UUID="3B64D28A5CF89146" TYPE="ntfs" PTTYPE="dos"
> # ./wipefs /dev/sdb
> DEVICE OFFSET TYPE UUID             LABEL
> sdb    0x3    ntfs 3B64D28A5CF89146
> sdb    0x1fe  dos

So what we call the dos partition table, Microsoft calls the "boot
sector", and it's not necessarily a partition table at all.  (Note
also that normally, Microsoft won't let you put an ntfs file system on
the whole disk --- and you had to force this with mkntfs -F.  It's
actually a bad idea, because some versions of Windows will see what it
thinks is an empty disk, and the contents of that whole disk file
system can be very easily overwritten by a confused Windows OS.)

You can see this in include/layout.h in the ntfs-3g sources.

This is a pretty broad and general confusion; fdisk will get confused
and think it is an empty dos partition table, and blkid will as well.
I suspect the right answer here is if the oem_id is "NTFS ", we should
not consider it a dos partition table at all, even if there is an "end
of boot sector" magic value of 0xaa55 at the end of the sector.

   		      	       	      	 - Ted

Re: blkid identifying whole disk ntfs as both ntfs and msdos partition table

[Karel Zak]

On Tue, Jul 02, 2019 at 02:14:39PM -0400, Theodore Ts'o wrote:
> On Tue, Jul 02, 2019 at 11:43:34AM +0100, Mike Fleetwood wrote:
> > Hi,
> > 
> > Tested with mkntfs from ntfsprogs and blkid and wipefs from util-linux git HEAD.
> > Test case:
> > 
> > # mkntfs -Q -F /dev/sdb
> > /dev/sdb is entire device, not just one partition.
> > mkntfs forced anyway.
> > Cluster size has been automatically set to 4096 bytes.
> > Creating NTFS volume structures.
> > mkntfs completed successfully. Have a nice day.
> > # ./blkid --version
> > lt-blkid from util-linux 2.34.24-e3bb9  (libblkid 2.34.24, 14-Jun-2019)
> > # ./blkid /dev/sdb
> > /dev/sdb: UUID="3B64D28A5CF89146" TYPE="ntfs" PTTYPE="dos"
> > # ./wipefs /dev/sdb
> > DEVICE OFFSET TYPE UUID             LABEL
> > sdb    0x3    ntfs 3B64D28A5CF89146
> > sdb    0x1fe  dos

 Thanks for your report Mike.

> So what we call the dos partition table, Microsoft calls the "boot
> sector", and it's not necessarily a partition table at all.  (Note
> also that normally, Microsoft won't let you put an ntfs file system on
> the whole disk --- and you had to force this with mkntfs -F.  It's
> actually a bad idea, because some versions of Windows will see what it
> thinks is an empty disk, and the contents of that whole disk file
> system can be very easily overwritten by a confused Windows OS.)
> 
> You can see this in include/layout.h in the ntfs-3g sources.
> 
> This is a pretty broad and general confusion; fdisk will get confused
> and think it is an empty dos partition table, and blkid will as well.
> I suspect the right answer here is if the oem_id is "NTFS ", we should
> not consider it a dos partition table at all, even if there is an "end
> of boot sector" magic value of 0xaa55 at the end of the sector.

 Well, our MBR detection is more sophisticated check for 0xaa55 magic
 ;-) but NTFS is not in the game now. I'll fix it. We need extra check
 for NTFS (as we already use for some FATs or GPT+PMBR).

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: blkid identifying whole disk ntfs as both ntfs and msdos partition table

[Karel Zak]

On Tue, Jul 02, 2019 at 11:43:34AM +0100, Mike Fleetwood wrote:
> # mkntfs -Q -F /dev/sdb
> /dev/sdb is entire device, not just one partition.
> mkntfs forced anyway.
> Cluster size has been automatically set to 4096 bytes.
> Creating NTFS volume structures.
> mkntfs completed successfully. Have a nice day.
> # ./blkid --version
> lt-blkid from util-linux 2.34.24-e3bb9  (libblkid 2.34.24, 14-Jun-2019)
> # ./blkid /dev/sdb
> /dev/sdb: UUID="3B64D28A5CF89146" TYPE="ntfs" PTTYPE="dos"

Fixed version:

 # ./blkid -p /dev/sdc
 /dev/sdc: UUID="18CBAEE61457A850" TYPE="ntfs" USAGE="filesystem"

> # ./wipefs /dev/sdb
> DEVICE OFFSET TYPE UUID             LABEL
> sdb    0x3    ntfs 3B64D28A5CF89146
> sdb    0x1fe  dos

wipefs has a little bit different semantic, it tries to gather as
much as possible magic strings, so after the bug fix it still reports
"dos" on 0x1fe. It's expected behavior.

wipefs internally masks previously detected magic string (e.g. "ntfs")
to avoid any interaction with the next probing. This is because we
want to see all possibilities. Note wipefs wipes only the magic
strings (and nothing else), so if you wipe "NTFS" than it will be
interpreted as MBR ("dos") as there is no more valid NTFS.

> Note that the test ntfs image in util-linux doesn't cause this.  Just file
> systems created by mkntfs from ntfsprogs.
> 
> # dd if=/dev/zero bs=1M of=/dev/sdb
> # xzcat tests/ts/blkid/images-fs/ntfs.img.xz > /dev/sdb
> # ./blkid /dev/sdb
> /dev/sdb: LABEL="M-PM-^]M-PM->M-PM-2M-QM-^KM-PM-9 M-QM-^BM-PM->M-PM-<"
> UUID="09CBB6DE30C87310" TYPE="ntfs"
> # ./wipefs /dev/sdb
> DEVICE OFFSET TYPE UUID             LABEL
> sdb    0x3    ntfs 09CBB6DE30C87310 Новый том

Sounds like we need another test image :-)


Thanks for your reports!

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: blkid identifying whole disk ntfs as both ntfs and msdos partition table

[Mike Fleetwood]

On Wed, 17 Jul 2019 at 12:16, Karel Zak <kzak@redhat.com> wrote:
>
> On Tue, Jul 02, 2019 at 11:43:34AM +0100, Mike Fleetwood wrote:
> > # mkntfs -Q -F /dev/sdb
> > /dev/sdb is entire device, not just one partition.
> > mkntfs forced anyway.
> > Cluster size has been automatically set to 4096 bytes.
> > Creating NTFS volume structures.
> > mkntfs completed successfully. Have a nice day.
> > # ./blkid --version
> > lt-blkid from util-linux 2.34.24-e3bb9  (libblkid 2.34.24, 14-Jun-2019)
> > # ./blkid /dev/sdb
> > /dev/sdb: UUID="3B64D28A5CF89146" TYPE="ntfs" PTTYPE="dos"
>
> Fixed version:
>
>  # ./blkid -p /dev/sdc
>  /dev/sdc: UUID="18CBAEE61457A850" TYPE="ntfs" USAGE="filesystem"
>
> > # ./wipefs /dev/sdb
> > DEVICE OFFSET TYPE UUID             LABEL
> > sdb    0x3    ntfs 3B64D28A5CF89146
> > sdb    0x1fe  dos
>
> wipefs has a little bit different semantic, it tries to gather as
> much as possible magic strings, so after the bug fix it still reports
> "dos" on 0x1fe. It's expected behavior.
>
> wipefs internally masks previously detected magic string (e.g. "ntfs")
> to avoid any interaction with the next probing. This is because we
> want to see all possibilities. Note wipefs wipes only the magic
> strings (and nothing else), so if you wipe "NTFS" than it will be
> interpreted as MBR ("dos") as there is no more valid NTFS.
>
> > Note that the test ntfs image in util-linux doesn't cause this.  Just file
> > systems created by mkntfs from ntfsprogs.
> >
> > # dd if=/dev/zero bs=1M of=/dev/sdb
> > # xzcat tests/ts/blkid/images-fs/ntfs.img.xz > /dev/sdb
> > # ./blkid /dev/sdb
> > /dev/sdb: LABEL="M-PM-^]M-PM->M-PM-2M-QM-^KM-PM-9 M-QM-^BM-PM->M-PM-<"
> > UUID="09CBB6DE30C87310" TYPE="ntfs"
> > # ./wipefs /dev/sdb
> > DEVICE OFFSET TYPE UUID             LABEL
> > sdb    0x3    ntfs 09CBB6DE30C87310 Новый том
>
> Sounds like we need another test image :-)
>
>
> Thanks for your reports!
>
>     Karel

Thank you for fixing these bugs so quickly.

Mike