[RFC] udf: 2.01 interoperability issues with Windows 10

[RFC] udf: 2.01 interoperability issues with Windows 10

[Steve Magnani]

Hi,

Recently I have been exploring Advanced Format (4K sector size)
and high capacity aspects of UDF 2.01 support in Linux and
Windows 10. I thought it might be helpful to summarize my findings.

The good news is that I did not see any bugs in the Linux
ecosystem (kernel driver + mkudffs).

The not-so-good news is that Windows has some issues that affect
interoperability. One of my goals in posting this is to open a
discussion on whether changes should be made in the Linux UDF
ecosystem to accommodate these quirks.

My test setup includes the following software components:

* mkudffs 1.3 and 2.0
* kernel 4.15.0-43 and 4.15.0-52
* Windows 10 1803 17134.829
* chkdsk 10.0.17134.1
* udfs.sys 10.0.17134.648


ISSUE 1: Inability of the Linux UDF driver to mount 4K-sector
          media formatted by Windows.

This is because the Windows ecosystem mishandles the ECMA-167
corner case that requires Volume Recognition Sequence components
to be placed at 4K intervals on 4K-sector media, instead of the
2K intervals required with smaller sectors. The Linux UDF driver
emits the following when presented with Windows-formatted media:

   UDF-fs: warning (device sdc1): udf_load_vrs: No VRS found
   UDF-fs: Scanning with blocksize 4096 failed

A hex dump of the VRS written by the Windows 10 'format' utility
yields this:

   0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
   0800: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
   1000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........

We may want to consider tweaking the kernel UDF driver to
tolerate this quirk; if so a question is whether that should be
done automatically, only in response to a mount option or
module parameter, or only with some subsequent confirmation
that the medium was formatted by Windows.


ISSUE 2: Inability of Windows chkdsk to analyze 4K-sector media
          formatted by mkudffs.

This is another aspect of Windows' VRS corner case bug.
Formatting by mkudffs places the VRS components at the proper 4K
intervals. But the chkdsk utility looks for components at 2K
intervals. Not finding a component at byte offset 2048, chkdsk
decides that the media is RAW and cannot be checked. Note that
this bug affects chkdsk only; udfs.sys *does* recognize mkudffs-
formatted 4K-sector media and is able to mount it.

It would be possible to work around this by tweaking mkudffs to
insert dummy BOOT2 components in between the BEA/NSR/TEA:

   0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
   0800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
   1000: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
   1800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
   2000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........

That would introduce a slight ECMA-167 nonconformity, but Linux
does not object and Windows actually performs better. I would
have to tweak udffsck though since I believe this could confuse
its automatic detection of medium block size.


ISSUE 3: Inability of the Windows UDF driver to mount media
          read-write when a maximally-sized space bitmap
          descriptor is present

I suspect this is an off-by-one error in udfs.sys relating to
the maximum number of blocks a space bitmap descriptor can
occupy. The bug causes UDF partitions that are close to 2 TiB
(512-sector media) or 16 TiB (4K-sector media) to be mounted
read-only, with no user-visible indication as to why.

It would be possible for mkudffs to print a warning when
formatting results in a space bitmap that occupies the maximum
number of blocks.


ISSUE 4: chkdsk reports spurious errors when space bitmap
          descriptor exceeds 32 MiB

Some permutations of this:

   * "Correcting errors in Space Bitmap Descriptor at block 0"
     (with no prior mention of any errors)
   * "Space Bitmap Descriptor at block 32 is corrupt or unreadable"

This is actually one of the more crippling issues if one values
Windows' ability to check and repair UDF errors. A limit of
32 MiB on the space bitmap implies a UDF partition size of at
most 137 GB (not GiB) with 512-sector media or at most 1099 GB
with 4K-sector media.

Again, the most I think we could do is code mkudffs to warn of
this possibility. But a message that clearly conveys the issue
and what should be done to avoid it could be a little tricky to
construct.


Obviously the best solution would be for the Windows bugs to
get fixed. If anyone reading this can convey these details into
the Microsoft silo, that would be great.

Regards,
------------------------------------------------------------------------
  Steven J. Magnani               "I claim this network for MARS!
  www.digidescorp.com              Earthling, return my space modulator!"

  #include <standard.disclaimer>

Re: [RFC] udf: 2.01 interoperability issues with Windows 10

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 6955 bytes --]

Hi! I'm adding Vojtěch Vladyka to conversation.

Steve, can you describe what you mean by Advanced Format 4K sector size?

It is hard disk with 512 bytes logical sector size and 4096 bytes
physical sector size? Or do you mean hard disk which has both logical
and physical sector size equal to 4096 bytes?

On Tuesday 09 July 2019 13:27:58 Steve Magnani wrote:
> Hi,
> 
> Recently I have been exploring Advanced Format (4K sector size)
> and high capacity aspects of UDF 2.01 support in Linux and
> Windows 10. I thought it might be helpful to summarize my findings.
> 
> The good news is that I did not see any bugs in the Linux
> ecosystem (kernel driver + mkudffs).
> 
> The not-so-good news is that Windows has some issues that affect
> interoperability. One of my goals in posting this is to open a
> discussion on whether changes should be made in the Linux UDF
> ecosystem to accommodate these quirks.
> 
> My test setup includes the following software components:
> 
> * mkudffs 1.3 and 2.0

Can you do tests also with last version of mkudffs 2.1?

> * kernel 4.15.0-43 and 4.15.0-52
> * Windows 10 1803 17134.829
> * chkdsk 10.0.17134.1
> * udfs.sys 10.0.17134.648
> 
> 
> ISSUE 1: Inability of the Linux UDF driver to mount 4K-sector
>          media formatted by Windows.

Can you check if udfinfo (from udftools) can recognize such disk?

And can blkid (from util-linux) recognize such disk as UDF with reading
all properties?

Can grub2 recognize such disks?

Also can you check if libparted from git master branch can recognize
such disk? In following commit I added support for recognizing UDF
filesystem in libparted, it is only in git master branch, not released:

http://git.savannah.gnu.org/cgit/parted.git/commit/?id=8740cfcff3ea839dd6dc8650dec0a466e9870625

> This is because the Windows ecosystem mishandles the ECMA-167
> corner case that requires Volume Recognition Sequence components
> to be placed at 4K intervals on 4K-sector media, instead of the
> 2K intervals required with smaller sectors. The Linux UDF driver
> emits the following when presented with Windows-formatted media:
> 
>   UDF-fs: warning (device sdc1): udf_load_vrs: No VRS found
>   UDF-fs: Scanning with blocksize 4096 failed
> 
> A hex dump of the VRS written by the Windows 10 'format' utility
> yields this:
> 
>   0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
>   0800: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
>   1000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........
> 
> We may want to consider tweaking the kernel UDF driver to
> tolerate this quirk; if so a question is whether that should be
> done automatically, only in response to a mount option or
> module parameter, or only with some subsequent confirmation
> that the medium was formatted by Windows.
> 
> 
> ISSUE 2: Inability of Windows chkdsk to analyze 4K-sector media
>          formatted by mkudffs.

This is really bad :-(

> This is another aspect of Windows' VRS corner case bug.
> Formatting by mkudffs places the VRS components at the proper 4K
> intervals. But the chkdsk utility looks for components at 2K
> intervals. Not finding a component at byte offset 2048, chkdsk
> decides that the media is RAW and cannot be checked. Note that
> this bug affects chkdsk only; udfs.sys *does* recognize mkudffs-
> formatted 4K-sector media and is able to mount it.
> 
> It would be possible to work around this by tweaking mkudffs to
> insert dummy BOOT2 components in between the BEA/NSR/TEA:
> 
>   0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
>   0800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
>   1000: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
>   1800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
>   2000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........
> 
> That would introduce a slight ECMA-167 nonconformity, but Linux
> does not object and Windows actually performs better. I would
> have to tweak udffsck though since I believe this could confuse
> its automatic detection of medium block size.

I would like to avoid this hack. If chkdsk is unable to detect such
filesystem, it is really a good idea to let it do try doing filesystem
checks and recovery? You are saying that udfs.sys can recognize such
disk and mount it. I think this should be enough.

Currently mkudffs contains workarounds for older util-linux versions or
MBR partition layout. But everything should be fully complaint with
ECMA-67 and OSTA-UDF.

> 
> ISSUE 3: Inability of the Windows UDF driver to mount media
>          read-write when a maximally-sized space bitmap
>          descriptor is present
> 
> I suspect this is an off-by-one error in udfs.sys relating to
> the maximum number of blocks a space bitmap descriptor can
> occupy. The bug causes UDF partitions that are close to 2 TiB
> (512-sector media) or 16 TiB (4K-sector media) to be mounted
> read-only, with no user-visible indication as to why.
> 
> It would be possible for mkudffs to print a warning when
> formatting results in a space bitmap that occupies the maximum
> number of blocks.
> 
> 
> ISSUE 4: chkdsk reports spurious errors when space bitmap
>          descriptor exceeds 32 MiB
> 
> Some permutations of this:
> 
>   * "Correcting errors in Space Bitmap Descriptor at block 0"
>     (with no prior mention of any errors)
>   * "Space Bitmap Descriptor at block 32 is corrupt or unreadable"
> 
> This is actually one of the more crippling issues if one values
> Windows' ability to check and repair UDF errors. A limit of
> 32 MiB on the space bitmap implies a UDF partition size of at
> most 137 GB (not GiB) with 512-sector media or at most 1099 GB
> with 4K-sector media.
> 
> Again, the most I think we could do is code mkudffs to warn of
> this possibility. But a message that clearly conveys the issue
> and what should be done to avoid it could be a little tricky to
> construct.

I have no problems with solution that udftools prints warning message
that current selected configuration does not work on Windows systems.
There are already some such warning messages in mkudffs.

> 
> Obviously the best solution would be for the Windows bugs to
> get fixed. If anyone reading this can convey these details into
> the Microsoft silo, that would be great.

For Windows 10 users, the only option is to report bug via
Use Feedback app on the Start Menu.

> Regards,
> ------------------------------------------------------------------------
>  Steven J. Magnani               "I claim this network for MARS!
>  www.digidescorp.com              Earthling, return my space modulator!"
> 
>  #include <standard.disclaimer>
> 

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC] udf: 2.01 interoperability issues with Windows 10

[Jan Kara]

Hi!

On Tue 09-07-19 13:27:58, Steve Magnani wrote:
> Recently I have been exploring Advanced Format (4K sector size)
> and high capacity aspects of UDF 2.01 support in Linux and
> Windows 10. I thought it might be helpful to summarize my findings.
> 
> The good news is that I did not see any bugs in the Linux
> ecosystem (kernel driver + mkudffs).
> 
> The not-so-good news is that Windows has some issues that affect
> interoperability. One of my goals in posting this is to open a
> discussion on whether changes should be made in the Linux UDF
> ecosystem to accommodate these quirks.
> 
> My test setup includes the following software components:
> 
> * mkudffs 1.3 and 2.0
> * kernel 4.15.0-43 and 4.15.0-52
> * Windows 10 1803 17134.829
> * chkdsk 10.0.17134.1
> * udfs.sys 10.0.17134.648
> 
> 
> ISSUE 1: Inability of the Linux UDF driver to mount 4K-sector
>          media formatted by Windows.
> 
> This is because the Windows ecosystem mishandles the ECMA-167
> corner case that requires Volume Recognition Sequence components
> to be placed at 4K intervals on 4K-sector media, instead of the
> 2K intervals required with smaller sectors. The Linux UDF driver
> emits the following when presented with Windows-formatted media:
> 
>   UDF-fs: warning (device sdc1): udf_load_vrs: No VRS found
>   UDF-fs: Scanning with blocksize 4096 failed
> 
> A hex dump of the VRS written by the Windows 10 'format' utility
> yields this:
> 
>   0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
>   0800: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
>   1000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........
> 
> We may want to consider tweaking the kernel UDF driver to
> tolerate this quirk; if so a question is whether that should be
> done automatically, only in response to a mount option or
> module parameter, or only with some subsequent confirmation
> that the medium was formatted by Windows.

Yeah, I think we could do that although it will be a bit painful. I would
do it by default if we found BEA descriptor but not NSR descriptor. We do
already have handling for various quirks of other udf creators so this is
nothing new... I think Palo replied about the rest of the issues you've
found.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [RFC] udf: 2.01 interoperability issues with Windows 10

[Steve Magnani]

On 7/9/19 1:56 PM, Pali Rohár wrote:
> Steve, can you describe what you mean by Advanced Format 4K sector size?
>
> It is hard disk with 512 bytes logical sector size and 4096 bytes
> physical sector size? Or do you mean hard disk which has both logical
> and physical sector size equal to 4096 bytes?

Sorry, forgot that the term Advanced Format introduces ambiguity.
As far as the OSes are concerned the drive is 4Kn.

On Tuesday 09 July 2019 13:27:58 Steve Magnani wrote:

>> Hi,
>>
>> Recently I have been exploring Advanced Format (4K sector size)
>> and high capacity aspects of UDF 2.01 support in Linux and
>> Windows 10. I thought it might be helpful to summarize my findings.
>>
>> The good news is that I did not see any bugs in the Linux
>> ecosystem (kernel driver + mkudffs).
>>
>> The not-so-good news is that Windows has some issues that affect
>> interoperability. One of my goals in posting this is to open a
>> discussion on whether changes should be made in the Linux UDF
>> ecosystem to accommodate these quirks.
>>
>> My test setup includes the following software components:
>>
>> * mkudffs 1.3 and 2.0
> Can you do tests also with last version of mkudffs 2.1?

A very quick smoke test of a 16-ish TiB 4Kn partition seemed OK.

>> * kernel 4.15.0-43 and 4.15.0-52
>> * Windows 10 1803 17134.829
>> * chkdsk 10.0.17134.1
>> * udfs.sys 10.0.17134.648
>>
>>
>> ISSUE 1: Inability of the Linux UDF driver to mount 4K-sector
>>           media formatted by Windows.
> Can you check if udfinfo (from udftools) can recognize such disk?

It cannot:

   $ ./udfinfo /dev/sdb1
   udfinfo: Error: UDF Volume Recognition Sequence found but not Anchor Volume Descriptor Pointer, maybe wrong --blocksize?
   udfinfo: Error: Cannot process device '/dev/sdb1' as UDF disk

   $ ./udfinfo --blocksize=4096 /dev/sdb1
   udfinfo: Error: UDF Volume Recognition Sequence not found
   udfinfo: Error: Cannot process device '/dev/sdb1' as UDF disk

   $ ./udfinfo
   udfinfo from udftools 2.1

> And can blkid (from util-linux) recognize such disk as UDF with reading
> all properties?

Seemingly:

   $ blkid --info /dev/sdb1
   /dev/sdb1: MINIMUM_IO_SIZE="4096"
              PHYSICAL_SECTOR_SIZE="4096"
              LOGICAL_SECTOR_SIZE="4096"

   $ blkid --probe /dev/sdb1
   /dev/sdb1: VOLUME_ID="UDF Volume"
              UUID="0e131b3b20554446"
              VOLUME_SET_ID="0E131B3B UDF Volume Set"
              LABEL="WIN10_FORMATTED"
              LOGICAL_VOLUME_ID="WIN10_FORMATTED"
              VERSION="2.01"
              TYPE="udf"
              USAGE="filesystem"

   $ blkid --version
   blkid from util-linux 2.31.1  (libblkid 2.31.1, 19-Dec-2017)

> Can grub2 recognize such disks?

I'm not sure what you're asking here. The physical interface to this drive is USB,
and it's not designed for general-purpose storage (or booting). That said, if you
have some grub2 commands you want me to run against this drive/partition let me know.

> Also can you check if libparted from git master branch can recognize
> such disk? In following commit I added support for recognizing UDF
> filesystem in libparted, it is only in git master branch, not released:
>
> http://git.savannah.gnu.org/cgit/parted.git/commit/?id=8740cfcff3ea839dd6dc8650dec0a466e9870625

Build failed:
   In file included from pt-tools.c:114:0:
   pt-tools.c: In function 'pt_limit_lookup':
   pt-limit.gperf:78:1: error: function might be candidate for attribute 'pure' [-Werror=suggest-attribute=pure]

If you send me some other SHA to try I can attempt a rebuild.

> ISSUE 2: Inability of Windows chkdsk to analyze 4K-sector media
>           formatted by mkudffs.
> This is really bad :-(
>
>> It would be possible to work around this by tweaking mkudffs to
>> insert dummy BOOT2 components in between the BEA/NSR/TEA:
>>
>>    0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
>>    0800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
>>    1000: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
>>    1800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
>>    2000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........
>>
>> That would introduce a slight ECMA-167 nonconformity, but Linux
>> does not object and Windows actually performs better. I would
>> have to tweak udffsck though since I believe this could confuse
>> its automatic detection of medium block size.
> I would like to avoid this hack. If chkdsk is unable to detect such
> filesystem, it is really a good idea to let it do try doing filesystem
> checks and recovery? You are saying that udfs.sys can recognize such
> disk and mount it. I think this should be enough.

Fair enough, but it's also reasonable to assume the bugginess is
limited to the VRS corner case. AFAIK that's the only place in ECMA-167
where there is a difference in layout specific to 4K sectors.
With the BOOT2 band-aid chkdsk is able to analyze filesystems on 4Kn media.

I use chkdsk frequently to double-check UDF generation firmware
I am writing, and also udffsck work-in-progress.

------------------------------------------------------------------------
  Steven J. Magnani               "I claim this network for MARS!
  www.digidescorp.com              Earthling, return my space modulator!"

  #include <standard.disclaimer>

Re: [RFC] udf: 2.01 interoperability issues with Windows 10

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 7640 bytes --]

On Tuesday 09 July 2019 15:14:38 Steve Magnani wrote:
> 
> On 7/9/19 1:56 PM, Pali Rohár wrote:
> > Steve, can you describe what you mean by Advanced Format 4K sector size?
> > 
> > It is hard disk with 512 bytes logical sector size and 4096 bytes
> > physical sector size? Or do you mean hard disk which has both logical
> > and physical sector size equal to 4096 bytes?
> 
> Sorry, forgot that the term Advanced Format introduces ambiguity.
> As far as the OSes are concerned the drive is 4Kn.

Ok, so from your blkid output it can be seen that hard disk has both
logical and physical sector size equal to 4096 bytes.

Problem with those keywords like Advanced or Native is that some vendors
use them also for 512/4096 disks, so better to to say logical sector
size explicitly and not using those marketing keywords.

> On Tuesday 09 July 2019 13:27:58 Steve Magnani wrote:
> 
> > > Hi,
> > > 
> > > Recently I have been exploring Advanced Format (4K sector size)
> > > and high capacity aspects of UDF 2.01 support in Linux and
> > > Windows 10. I thought it might be helpful to summarize my findings.
> > > 
> > > The good news is that I did not see any bugs in the Linux
> > > ecosystem (kernel driver + mkudffs).
> > > 
> > > The not-so-good news is that Windows has some issues that affect
> > > interoperability. One of my goals in posting this is to open a
> > > discussion on whether changes should be made in the Linux UDF
> > > ecosystem to accommodate these quirks.
> > > 
> > > My test setup includes the following software components:
> > > 
> > > * mkudffs 1.3 and 2.0
> > Can you do tests also with last version of mkudffs 2.1?
> 
> A very quick smoke test of a 16-ish TiB 4Kn partition seemed OK.
> 
> > > * kernel 4.15.0-43 and 4.15.0-52
> > > * Windows 10 1803 17134.829
> > > * chkdsk 10.0.17134.1
> > > * udfs.sys 10.0.17134.648
> > > 
> > > 
> > > ISSUE 1: Inability of the Linux UDF driver to mount 4K-sector
> > >           media formatted by Windows.
> > Can you check if udfinfo (from udftools) can recognize such disk?
> 
> It cannot:
> 
>   $ ./udfinfo /dev/sdb1
>   udfinfo: Error: UDF Volume Recognition Sequence found but not Anchor Volume Descriptor Pointer, maybe wrong --blocksize?
>   udfinfo: Error: Cannot process device '/dev/sdb1' as UDF disk
> 
>   $ ./udfinfo --blocksize=4096 /dev/sdb1
>   udfinfo: Error: UDF Volume Recognition Sequence not found
>   udfinfo: Error: Cannot process device '/dev/sdb1' as UDF disk
> 
>   $ ./udfinfo
>   udfinfo from udftools 2.1

Ok. This means that also udflabel is unable to show or change UDF label
on such disks.

So not only Linux kernel driver needs to be workarounded, but also
udftools.

> > And can blkid (from util-linux) recognize such disk as UDF with reading
> > all properties?
> 
> Seemingly:
> 
>   $ blkid --info /dev/sdb1
>   /dev/sdb1: MINIMUM_IO_SIZE="4096"
>              PHYSICAL_SECTOR_SIZE="4096"
>              LOGICAL_SECTOR_SIZE="4096"
> 
>   $ blkid --probe /dev/sdb1
>   /dev/sdb1: VOLUME_ID="UDF Volume"
>              UUID="0e131b3b20554446"
>              VOLUME_SET_ID="0E131B3B UDF Volume Set"
>              LABEL="WIN10_FORMATTED"
>              LOGICAL_VOLUME_ID="WIN10_FORMATTED"
>              VERSION="2.01"
>              TYPE="udf"
>              USAGE="filesystem"
> 
>   $ blkid --version
>   blkid from util-linux 2.31.1  (libblkid 2.31.1, 19-Dec-2017)

So it is working. Working blkid is required as its library libblkid is
used for parsing/mounting /etc/fstab file and also that GUI partition
tools use it for identifying block devices.

> > Can grub2 recognize such disks?
> 
> I'm not sure what you're asking here. The physical interface to this drive is USB,

It is USB mass storage device? If yes, then grub2 should be able to
normally use. Read its content, etc... You can use "ls" grub command to
list content of disk with supported filesystem.

> and it's not designed for general-purpose storage (or booting). That said, if you
> have some grub2 commands you want me to run against this drive/partition let me know.

There is also some way for using grub's fs implementation to read disk
images. It is primary used by grub's automated tests. I do not know
right now how to use, I need to look grub documentation. But I have
already used it during implementation of UDF UUID in grub.

> > Also can you check if libparted from git master branch can recognize
> > such disk? In following commit I added support for recognizing UDF
> > filesystem in libparted, it is only in git master branch, not released:
> > 
> > http://git.savannah.gnu.org/cgit/parted.git/commit/?id=8740cfcff3ea839dd6dc8650dec0a466e9870625
> 
> Build failed:
>   In file included from pt-tools.c:114:0:
>   pt-tools.c: In function 'pt_limit_lookup':
>   pt-limit.gperf:78:1: error: function might be candidate for attribute 'pure' [-Werror=suggest-attribute=pure]
> 
> If you send me some other SHA to try I can attempt a rebuild.

Try to use top of master branch. That mentioned commit is already in git
master.

And if it still produce that error, compile without -Werror flag (or add
-Wno-error).

> > ISSUE 2: Inability of Windows chkdsk to analyze 4K-sector media
> >           formatted by mkudffs.
> > This is really bad :-(
> > 
> > > It would be possible to work around this by tweaking mkudffs to
> > > insert dummy BOOT2 components in between the BEA/NSR/TEA:
> > > 
> > >    0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
> > >    0800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
> > >    1000: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
> > >    1800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
> > >    2000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........
> > > 
> > > That would introduce a slight ECMA-167 nonconformity, but Linux
> > > does not object and Windows actually performs better. I would
> > > have to tweak udffsck though since I believe this could confuse
> > > its automatic detection of medium block size.
> > I would like to avoid this hack. If chkdsk is unable to detect such
> > filesystem, it is really a good idea to let it do try doing filesystem
> > checks and recovery? You are saying that udfs.sys can recognize such
> > disk and mount it. I think this should be enough.
> 
> Fair enough, but it's also reasonable to assume the bugginess is
> limited to the VRS corner case. AFAIK that's the only place in ECMA-167
> where there is a difference in layout specific to 4K sectors.
> With the BOOT2 band-aid chkdsk is able to analyze filesystems on 4Kn media.

Main problem with this hack is that it breaks detection of valid UDF
filesystems which use VRS for block size detection. I do not know which
implementation may use VRS for block size detection, but I do not see
anything wrong in this approach.

> I use chkdsk frequently to double-check UDF generation firmware

Vojtěch wrote in his thesis that MS's chkdsk sometimes put UDF
filesystem into more broken state as before.

> I am writing, and also udffsck work-in-progress.

Have you used some Vojtěch's parts? Or are you writing it from scratch?

> ------------------------------------------------------------------------
>  Steven J. Magnani               "I claim this network for MARS!
>  www.digidescorp.com              Earthling, return my space modulator!"
> 
>  #include <standard.disclaimer>
> 

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC] udf: 2.01 interoperability issues with Windows 10

[Steve Magnani]

On 7/9/19 4:04 PM, Pali Rohár wrote:
> On Tuesday 09 July 2019 15:14:38 Steve Magnani wrote:
>> On 7/9/19 1:56 PM, Pali Rohár wrote:
>>> Can grub2 recognize such disks?
>> I'm not sure what you're asking here. The physical interface to this drive is USB,
> It is USB mass storage device? If yes, then grub2 should be able to
> normally use. Read its content, etc... You can use "ls" grub command to
> list content of disk with supported filesystem.

Yes, Mass Storage Bulk-Only Transport.

>
>> and it's not designed for general-purpose storage (or booting). That said, if you
>> have some grub2 commands you want me to run against this drive/partition let me know.
> There is also some way for using grub's fs implementation to read disk
> images. It is primary used by grub's automated tests. I do not know
> right now how to use, I need to look grub documentation. But I have
> already used it during implementation of UDF UUID in grub.

Grub is not recognizing my USB drive, i.e. 'ls' does not show usb0 as an option.
I tried 'insmod usb' but that made no difference. Maybe grub does not support my
USB 3.0 host controller, I will retry on a USB2 port when I have a chance.

>>> Also can you check if libparted from git master branch can recognize
>>> such disk? In following commit I added support for recognizing UDF
>>> filesystem in libparted, it is only in git master branch, not released:
>>>
>>> http://git.savannah.gnu.org/cgit/parted.git/commit/?id=8740cfcff3ea839dd6dc8650dec0a466e9870625
>> Build failed:
>>    In file included from pt-tools.c:114:0:
>>    pt-tools.c: In function 'pt_limit_lookup':
>>    pt-limit.gperf:78:1: error: function might be candidate for attribute 'pure' [-Werror=suggest-attribute=pure]
>>
>> If you send me some other SHA to try I can attempt a rebuild.
> Try to use top of master branch. That mentioned commit is already in git
> master.
>
> And if it still produce that error, compile without -Werror flag (or add
> -Wno-error).

I had to configure with CFLAGS=-Wno-error.

It does not recognize Windows-formatted 4K-sector media:
   Disk /dev/sdb: 17.6TB
   Sector size (logical/physical): 4096B/4096B
   Partition Table: msdos
   Disk Flags:

   Number  Start   End     Size    Type     File system  Flags
    1      1049kB  17.6TB  17.6TB  primary


It does recognize mkudffs-formatted media.

>
>>> ISSUE 2: Inability of Windows chkdsk to analyze 4K-sector media
>>>            formatted by mkudffs.
>>> This is really bad :-(
>>>
>>>> It would be possible to work around this by tweaking mkudffs to
>>>> insert dummy BOOT2 components in between the BEA/NSR/TEA:
>>>>
>>>>     0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
>>>>     0800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
>>>>     1000: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
>>>>     1800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
>>>>     2000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........
>>>>
>>>> That would introduce a slight ECMA-167 nonconformity, but Linux
>>>> does not object and Windows actually performs better. I would
>>>> have to tweak udffsck though since I believe this could confuse
>>>> its automatic detection of medium block size.
>>> I would like to avoid this hack. If chkdsk is unable to detect such
>>> filesystem, it is really a good idea to let it do try doing filesystem
>>> checks and recovery? You are saying that udfs.sys can recognize such
>>> disk and mount it. I think this should be enough.
>> Fair enough, but it's also reasonable to assume the bugginess is
>> limited to the VRS corner case. AFAIK that's the only place in ECMA-167
>> where there is a difference in layout specific to 4K sectors.
>> With the BOOT2 band-aid chkdsk is able to analyze filesystems on 4Kn media.
> Main problem with this hack is that it breaks detection of valid UDF
> filesystems which use VRS for block size detection. I do not know which
> implementation may use VRS for block size detection, but I do not see
> anything wrong in this approach.

I went through this with udffsck. The VRS is not very helpful in
determining block size because the only time the block size can be
determined conclusively is when the interval between VRS components
is > 2048 bytes. With an interval of 2048 bytes, the only conclusion
that can be drawn is that blocks are no larger than 2048 bytes.

>> I use chkdsk frequently to double-check UDF generation firmware
> Vojtěch wrote in his thesis that MS's chkdsk sometimes put UDF
> filesystem into more broken state as before.

Yes, I have personally experienced this. I don't have chkdsk do
repairs any more. In my case the problem may be that chkdsk
poorly handles the cascading corruption that resulted from this:

     https://lkml.org/lkml/2019/2/8/740

>> I am writing, and also udffsck work-in-progress.
> Have you used some Vojtěch's parts? Or are you writing it from scratch?
>
A udffsck discussion should probably continue in another thread.
Here let me just say that I have been enhancing Vojtěch's code,
in this fork:

   https://github.com/smagnani/udftools

...as time permits. Since winter ended the time I have available
for this has plummeted, so progress is very slow. But this recent
kernel driver patch grew out of work to make sure that udffsck
handles the UDF "file tail" properly:

   https://lkml.org/lkml/2019/6/4/551
   https://lkml.org/lkml/2019/6/30/181

------------------------------------------------------------------------
  Steven J. Magnani               "I claim this network for MARS!
  www.digidescorp.com               Earthling, return my space modulator!"

  #include <standard.disclaimer>

Re: [RFC] udf: 2.01 interoperability issues with Windows 10

[Pali Rohár]

On Wednesday 10 July 2019 08:24:09 Steve Magnani wrote:
> 
> On 7/9/19 4:04 PM, Pali Rohár wrote:
> > On Tuesday 09 July 2019 15:14:38 Steve Magnani wrote:
> > > On 7/9/19 1:56 PM, Pali Rohár wrote:
> > > > Can grub2 recognize such disks?
> > > I'm not sure what you're asking here. The physical interface to this drive is USB,
> > It is USB mass storage device? If yes, then grub2 should be able to
> > normally use. Read its content, etc... You can use "ls" grub command to
> > list content of disk with supported filesystem.
> 
> Yes, Mass Storage Bulk-Only Transport.
> 
> > 
> > > and it's not designed for general-purpose storage (or booting). That said, if you
> > > have some grub2 commands you want me to run against this drive/partition let me know.
> > There is also some way for using grub's fs implementation to read disk
> > images. It is primary used by grub's automated tests. I do not know
> > right now how to use, I need to look grub documentation. But I have
> > already used it during implementation of UDF UUID in grub.
> 
> Grub is not recognizing my USB drive, i.e. 'ls' does not show usb0 as an option.
> I tried 'insmod usb' but that made no difference. Maybe grub does not support my
> USB 3.0 host controller, I will retry on a USB2 port when I have a chance.

In some cases, BIOS/UEFI firmware supports USB mass storage devices and
then grub see them... So it depends on how grub access to disk. Pre-boot
environment is always fragile...

> > > > Also can you check if libparted from git master branch can recognize
> > > > such disk? In following commit I added support for recognizing UDF
> > > > filesystem in libparted, it is only in git master branch, not released:
> > > > 
> > > > http://git.savannah.gnu.org/cgit/parted.git/commit/?id=8740cfcff3ea839dd6dc8650dec0a466e9870625
> > > Build failed:
> > >    In file included from pt-tools.c:114:0:
> > >    pt-tools.c: In function 'pt_limit_lookup':
> > >    pt-limit.gperf:78:1: error: function might be candidate for attribute 'pure' [-Werror=suggest-attribute=pure]
> > > 
> > > If you send me some other SHA to try I can attempt a rebuild.
> > Try to use top of master branch. That mentioned commit is already in git
> > master.
> > 
> > And if it still produce that error, compile without -Werror flag (or add
> > -Wno-error).
> 
> I had to configure with CFLAGS=-Wno-error.
> 
> It does not recognize Windows-formatted 4K-sector media:
>   Disk /dev/sdb: 17.6TB
>   Sector size (logical/physical): 4096B/4096B
>   Partition Table: msdos
>   Disk Flags:
> 
>   Number  Start   End     Size    Type     File system  Flags
>    1      1049kB  17.6TB  17.6TB  primary
> 

Ok, so it means that GUI/TUI tools based on libparted would have
problems with these disks too.

So ISSUE 1 is big problem for Linux.

> It does recognize mkudffs-formatted media.

That is expected.

> > 
> > > > ISSUE 2: Inability of Windows chkdsk to analyze 4K-sector media
> > > >            formatted by mkudffs.
> > > > This is really bad :-(
> > > > 
> > > > > It would be possible to work around this by tweaking mkudffs to
> > > > > insert dummy BOOT2 components in between the BEA/NSR/TEA:
> > > > > 
> > > > >     0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
> > > > >     0800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
> > > > >     1000: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
> > > > >     1800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
> > > > >     2000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........
> > > > > 
> > > > > That would introduce a slight ECMA-167 nonconformity, but Linux
> > > > > does not object and Windows actually performs better. I would
> > > > > have to tweak udffsck though since I believe this could confuse
> > > > > its automatic detection of medium block size.
> > > > I would like to avoid this hack. If chkdsk is unable to detect such
> > > > filesystem, it is really a good idea to let it do try doing filesystem
> > > > checks and recovery? You are saying that udfs.sys can recognize such
> > > > disk and mount it. I think this should be enough.
> > > Fair enough, but it's also reasonable to assume the bugginess is
> > > limited to the VRS corner case. AFAIK that's the only place in ECMA-167
> > > where there is a difference in layout specific to 4K sectors.
> > > With the BOOT2 band-aid chkdsk is able to analyze filesystems on 4Kn media.
> > Main problem with this hack is that it breaks detection of valid UDF
> > filesystems which use VRS for block size detection. I do not know which
> > implementation may use VRS for block size detection, but I do not see
> > anything wrong in this approach.
> 
> I went through this with udffsck. The VRS is not very helpful in
> determining block size because the only time the block size can be
> determined conclusively is when the interval between VRS components
> is > 2048 bytes. With an interval of 2048 bytes, the only conclusion
> that can be drawn is that blocks are no larger than 2048 bytes.

Yes, I know. But for >2048 block sizes it can be used and is allowed by
specification.

> > > I use chkdsk frequently to double-check UDF generation firmware
> > Vojtěch wrote in his thesis that MS's chkdsk sometimes put UDF
> > filesystem into more broken state as before.
> 
> Yes, I have personally experienced this. I don't have chkdsk do
> repairs any more. In my case the problem may be that chkdsk
> poorly handles the cascading corruption that resulted from this:
> 
>     https://lkml.org/lkml/2019/2/8/740
> 
> > > I am writing, and also udffsck work-in-progress.
> > Have you used some Vojtěch's parts? Or are you writing it from scratch?
> > 
> A udffsck discussion should probably continue in another thread.
> Here let me just say that I have been enhancing Vojtěch's code,
> in this fork:
> 
>   https://github.com/smagnani/udftools
> 
> ...as time permits. Since winter ended the time I have available
> for this has plummeted, so progress is very slow. But this recent
> kernel driver patch grew out of work to make sure that udffsck
> handles the UDF "file tail" properly:
> 
>   https://lkml.org/lkml/2019/6/4/551
>   https://lkml.org/lkml/2019/6/30/181

Great, thank you for update.

-- 
Pali Rohár
pali.rohar@gmail.com