UDF filesystem image with Write-Once UDF Access Type

UDF filesystem image with Write-Once UDF Access Type

[Pali Rohár]

Hello,

I had discussion with Roald and based on his tests, Linux kernel udf.ko
driver mounts UDF filesystem image with Write-Once UDF Access Type as
normal read/write filesystem.

I think this is a bug as Write-Once Access Type is defined that existing
blocks on filesystem cannot be rewritten. Only new blocks can be
appended after end of device. Basically it means special HW support from
underlying media, e.g. for optical medias packet-writing technique (or
ability to burn new session) and CDROM_LAST_WRITTEN ioctl to locate
"current" end of device.

In my opinion without support for additional layer, kernel should treat
UDF Write-Once Access Type as read-only mount for userspace. And not
classic read/write mount.

If you want to play with Write-Once Access Type, use recent version of
mkudffs and choose --media-type=cdr option, which generates UDF
filesystem suitable for CD-R (Write-Once Access Type with VAT and other
UDF options according to UDF specification).

Also in git master of udftools has mkduffs now new option --read-only
which creates UDF image with Read-Only Access Type.

It seems that udf.ko does not support updating VAT table, so probably it
should treat also filesystem with VAT as read-only too.

-- 
Pali Rohár
pali.rohar@gmail.com

Re: UDF filesystem image with Write-Once UDF Access Type

[Steve Magnani]

Hi,

On 7/12/19 5:02 AM, Pali Rohár wrote:
> In my opinion without support for additional layer, kernel should treat
> UDF Write-Once Access Type as read-only mount for userspace. And not
> classic read/write mount.
>
> ...
>
> It seems that udf.ko does not support updating VAT table, so probably it
> should treat also filesystem with VAT as read-only too.
>

I thinkb085fbe2ef7fa7489903c45271ae7b7a52b0f9ab  <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/udf?h=v5.1&id=b085fbe2ef7fa7489903c45271ae7b7a52b0f9ab>, deployed in 4.20,
does both of the things you want.

One case I ran across today that Windows handles, but Linux doesn't,
is write-protection via flags in the DomainIdentifier fields of the
Logical Volume Descriptor and File Set Descriptor. Linux allows
RW mount when those are marked protected, but Windows forces RO mount.


Regards,
------------------------------------------------------------------------
  Steven J. Magnani               "I claim this network for MARS!
  www.digidescorp.com              Earthling, return my space modulator!"

  #include <standard.disclaimer>

Re: UDF filesystem image with Write-Once UDF Access Type

[Jan Kara]

Hello!

On Fri 12-07-19 12:02:24, Pali Rohár  wrote:
> I had discussion with Roald and based on his tests, Linux kernel udf.ko
> driver mounts UDF filesystem image with Write-Once UDF Access Type as
> normal read/write filesystem.
>
> I think this is a bug as Write-Once Access Type is defined that existing
> blocks on filesystem cannot be rewritten. Only new blocks can be
> appended after end of device. Basically it means special HW support from
> underlying media, e.g. for optical medias packet-writing technique (or
> ability to burn new session) and CDROM_LAST_WRITTEN ioctl to locate
> "current" end of device.
> 
> In my opinion without support for additional layer, kernel should treat
> UDF Write-Once Access Type as read-only mount for userspace. And not
> classic read/write mount.
> 
> If you want to play with Write-Once Access Type, use recent version of
> mkudffs and choose --media-type=cdr option, which generates UDF
> filesystem suitable for CD-R (Write-Once Access Type with VAT and other
> UDF options according to UDF specification).

Reasonably recent kernels should have this bug fixed and mount such fs read
only. That being said I've tested current upstream kernel with a media
created with --media-type=cdr and mounting failed with:

UDF-fs: error (device ubdb): udf_read_inode: (ino 524287) failed !bh
UDF-fs: error (device ubdb): udf_read_inode: (ino 524286) failed !bh
UDF-fs: error (device ubdb): udf_read_inode: (ino 524285) failed !bh
UDF-fs: error (device ubdb): udf_read_inode: (ino 524284) failed !bh
UDF-fs: Scanning with blocksize 2048 failed

So there's something fishy either in the created image or the kernel...
Didn't debug this further yet.

> Also in git master of udftools has mkduffs now new option --read-only
> which creates UDF image with Read-Only Access Type.

I've tested this and the kernel properly mounts the image read-only.

> It seems that udf.ko does not support updating VAT table, so probably it
> should treat also filesystem with VAT as read-only too.

This is definitely handled properly and we mount such fs read-only as well.

									Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: UDF filesystem image with Write-Once UDF Access Type

[Jan Kara]

On Fri 26-07-19 12:44:34, Steve Magnani wrote:
> Hi,
> 
> On 7/12/19 5:02 AM, Pali Rohár wrote:
> > In my opinion without support for additional layer, kernel should treat
> > UDF Write-Once Access Type as read-only mount for userspace. And not
> > classic read/write mount.
> > 
> > ...
> > 
> > It seems that udf.ko does not support updating VAT table, so probably it
> > should treat also filesystem with VAT as read-only too.
> > 
> 
> I thinkb085fbe2ef7fa7489903c45271ae7b7a52b0f9ab  <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/udf?h=v5.1&id=b085fbe2ef7fa7489903c45271ae7b7a52b0f9ab>, deployed in 4.20,
> does both of the things you want.
> 
> One case I ran across today that Windows handles, but Linux doesn't,
> is write-protection via flags in the DomainIdentifier fields of the
> Logical Volume Descriptor and File Set Descriptor. Linux allows
> RW mount when those are marked protected, but Windows forces RO mount.

Yeah, you're right. We are currently completely ignoring the
DomainIdentifier field and at least for read-write mounts we should make
sure it is valid. So that's something that needs fixing.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: UDF filesystem image with Write-Once UDF Access Type

[Jan Kara]

On Thu 01-08-19 09:35:30, Jan Kara wrote:
> > If you want to play with Write-Once Access Type, use recent version of
> > mkudffs and choose --media-type=cdr option, which generates UDF
> > filesystem suitable for CD-R (Write-Once Access Type with VAT and other
> > UDF options according to UDF specification).
> 
> Reasonably recent kernels should have this bug fixed and mount such fs read
> only. That being said I've tested current upstream kernel with a media
> created with --media-type=cdr and mounting failed with:
> 
> UDF-fs: error (device ubdb): udf_read_inode: (ino 524287) failed !bh
> UDF-fs: error (device ubdb): udf_read_inode: (ino 524286) failed !bh
> UDF-fs: error (device ubdb): udf_read_inode: (ino 524285) failed !bh
> UDF-fs: error (device ubdb): udf_read_inode: (ino 524284) failed !bh
> UDF-fs: Scanning with blocksize 2048 failed
> 
> So there's something fishy either in the created image or the kernel...
> Didn't debug this further yet.

Hum, looks like a problem with mkudffs. Relevant debug messages look like:

UDF-fs: fs/udf/super.c:671:udf_check_vsd: Starting at sector 16 (2048 byte sectors)
UDF-fs: fs/udf/super.c:824:udf_load_pvoldesc: recording time 2019/08/01 09:47 (1078)
UDF-fs: fs/udf/super.c:836:udf_load_pvoldesc: volIdent[] = 'LinuxUDF'
UDF-fs: fs/udf/super.c:844:udf_load_pvoldesc: volSetIdent[] = '1564645645200563LinuxUDF'
UDF-fs: fs/udf/super.c:1462:udf_load_logicalvol: Partition (0:0) type 1 on volume 1
UDF-fs: fs/udf/super.c:1462:udf_load_logicalvol: Partition (1:0) type 2 on volume 1
UDF-fs: fs/udf/super.c:1471:udf_load_logicalvol: FileSet found in LogicalVolDesc at block=0, partition=1
UDF-fs: fs/udf/super.c:1218:udf_load_partdesc: Searching map: (0 == 0)
UDF-fs: fs/udf/super.c:1060:udf_fill_partdesc_info: Partition (0 type 1511) starts at physical 288, block length 524000
UDF-fs: fs/udf/super.c:1060:udf_fill_partdesc_info: Partition (1 type 2012) starts at physical 288, block length 524000
UDF-fs: fs/udf/misc.c:223:udf_read_tagged: location mismatch block 524287, tag 0 != 523999
UDF-fs: error (device ubdb): udf_read_inode: (ino 524287) failed !bh

So the fact that location tag was 0 in block 524287 (which should contain
VAT inode) suggests there's something fishy with how / where mkudffs
creates the VAT inode. Can you have a look?

BTW, mkudffs messages look like:
filename=/tmp/image
label=LinuxUDF
uuid=1564645645200563
blocksize=2048
blocks=524288
udfrev=2.01
vatblock=319
start=0, blocks=16, type=RESERVED 
start=16, blocks=4, type=VRS 
start=20, blocks=76, type=USPACE 
start=96, blocks=16, type=MVDS 
start=112, blocks=16, type=USPACE 
start=128, blocks=1, type=LVID 
start=129, blocks=95, type=USPACE 
start=224, blocks=16, type=RVDS 
start=240, blocks=16, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=31, type=USPACE 
start=288, blocks=524000, type=PSPACE 

which suggests that VAT was indeed allocated somewhere in the beginning of
the partition.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: UDF filesystem image with Write-Once UDF Access Type

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 466 bytes --]

On Thursday 01 August 2019 09:35:30 Jan Kara wrote:
> On Fri 12-07-19 12:02:24, Pali Rohár  wrote:
> > Also in git master of udftools has mkduffs now new option --read-only
> > which creates UDF image with Read-Only Access Type.
> 
> I've tested this and the kernel properly mounts the image read-only.

Roald, can you test that problem which you described to me with
read-only access type is now correctly fixed?

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: UDF filesystem image with Write-Once UDF Access Type

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 5853 bytes --]

On Thursday 01 August 2019 10:38:00 Jan Kara wrote:
> On Thu 01-08-19 09:35:30, Jan Kara wrote:
> > > If you want to play with Write-Once Access Type, use recent version of
> > > mkudffs and choose --media-type=cdr option, which generates UDF
> > > filesystem suitable for CD-R (Write-Once Access Type with VAT and other
> > > UDF options according to UDF specification).
> > 
> > Reasonably recent kernels should have this bug fixed and mount such fs read
> > only. That being said I've tested current upstream kernel with a media
> > created with --media-type=cdr and mounting failed with:
> > 
> > UDF-fs: error (device ubdb): udf_read_inode: (ino 524287) failed !bh
> > UDF-fs: error (device ubdb): udf_read_inode: (ino 524286) failed !bh
> > UDF-fs: error (device ubdb): udf_read_inode: (ino 524285) failed !bh
> > UDF-fs: error (device ubdb): udf_read_inode: (ino 524284) failed !bh
> > UDF-fs: Scanning with blocksize 2048 failed
> > 
> > So there's something fishy either in the created image or the kernel...
> > Didn't debug this further yet.

Hi!

Now I verified version from git master branch and it seems to work.

$ ./mkudffs/mkudffs --media-type=cdr --new-file /tmp/cdr 307200
filename=/tmp/cdr
label=LinuxUDF
uuid=1564648861319606
blocksize=2048
blocks=307200
udfrev=2.01
vatblock=319
start=0, blocks=16, type=RESERVED 
start=16, blocks=4, type=VRS 
start=20, blocks=76, type=USPACE 
start=96, blocks=16, type=MVDS 
start=112, blocks=16, type=USPACE 
start=128, blocks=1, type=LVID 
start=129, blocks=95, type=USPACE 
start=224, blocks=16, type=RVDS 
start=240, blocks=16, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=31, type=USPACE 
start=288, blocks=306912, type=PSPACE

$ ls -l -a /tmp/cdr
-rw-r----- 1 pali pali 655360 aug  1 10:41 /tmp/cdr

$ mkdir /mnt/cdr

$ sudo mount /tmp/cdr /mnt/cdr -o loop
mount: /dev/loop0 is write-protected, mounting read-only

$ dmesg | tail
[320934.128836] loop: module loaded
[320934.169960] UDF-fs: warning (device loop0): udf_load_vrs: No anchor found
[320934.169965] UDF-fs: Rescanning with blocksize 2048
[320934.177772] UDF-fs: warning (device loop0): udf_load_vrs: No anchor found
[320934.177777] UDF-fs: Rescanning with blocksize 2048
[320934.181206] UDF-fs: INFO Mounting volume 'LinuxUDF', timestamp 2019/08/01 10:41 (1078)

$ uname -rvm
4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u4 (2019-07-19) x86_64


And CDR code was not modified since release 2.1. At time of releasing
version 2.1 I tested that cdr image created by mkudffs and burned to
CD-R disc can be correctly recognized and mounted by linux kernel.

> Hum, looks like a problem with mkudffs. Relevant debug messages look like:
> 
> UDF-fs: fs/udf/super.c:671:udf_check_vsd: Starting at sector 16 (2048 byte sectors)
> UDF-fs: fs/udf/super.c:824:udf_load_pvoldesc: recording time 2019/08/01 09:47 (1078)
> UDF-fs: fs/udf/super.c:836:udf_load_pvoldesc: volIdent[] = 'LinuxUDF'
> UDF-fs: fs/udf/super.c:844:udf_load_pvoldesc: volSetIdent[] = '1564645645200563LinuxUDF'
> UDF-fs: fs/udf/super.c:1462:udf_load_logicalvol: Partition (0:0) type 1 on volume 1
> UDF-fs: fs/udf/super.c:1462:udf_load_logicalvol: Partition (1:0) type 2 on volume 1
> UDF-fs: fs/udf/super.c:1471:udf_load_logicalvol: FileSet found in LogicalVolDesc at block=0, partition=1
> UDF-fs: fs/udf/super.c:1218:udf_load_partdesc: Searching map: (0 == 0)
> UDF-fs: fs/udf/super.c:1060:udf_fill_partdesc_info: Partition (0 type 1511) starts at physical 288, block length 524000
> UDF-fs: fs/udf/super.c:1060:udf_fill_partdesc_info: Partition (1 type 2012) starts at physical 288, block length 524000
> UDF-fs: fs/udf/misc.c:223:udf_read_tagged: location mismatch block 524287, tag 0 != 523999
> UDF-fs: error (device ubdb): udf_read_inode: (ino 524287) failed !bh
> 
> So the fact that location tag was 0 in block 524287 (which should contain
> VAT inode) suggests there's something fishy with how / where mkudffs
> creates the VAT inode. Can you have a look?
> 
> BTW, mkudffs messages look like:
> filename=/tmp/image
> label=LinuxUDF
> uuid=1564645645200563
> blocksize=2048
> blocks=524288
> udfrev=2.01
> vatblock=319
> start=0, blocks=16, type=RESERVED 
> start=16, blocks=4, type=VRS 
> start=20, blocks=76, type=USPACE 
> start=96, blocks=16, type=MVDS 
> start=112, blocks=16, type=USPACE 
> start=128, blocks=1, type=LVID 
> start=129, blocks=95, type=USPACE 
> start=224, blocks=16, type=RVDS 
> start=240, blocks=16, type=USPACE 
> start=256, blocks=1, type=ANCHOR 
> start=257, blocks=31, type=USPACE 
> start=288, blocks=524000, type=PSPACE 
> 
> which suggests that VAT was indeed allocated somewhere in the beginning of
> the partition.

For write-once media you are not able to modify size of UDF partition.
So if you are creating image for CD-R disc, you need to specify size of
UDF filesystem to match size of CD-R disc. VAT is always burned to the
last block of current track on CD-R.

Therefore if you had pre-allocated big image file for CD-R and then you
run mkudffs for cdr on it, you lost information what is the last used
block on that cdr image. Normally for optical drivers kernel use mmc
commands to retrieve last block of current session and based on it find
VAT. But image files loaded via /dev/loop are not optical drivers and
therefore do not have ability "hardware" ability to ask where is the
last used block. IIRC in this case kernel just fallback to the last
block of block device for VAT, which in this case is not correct.

What should help is to truncate image file to "correct" size after
running mkudffs with --media-type=cdr. Maybe mkudffs itself should do it
when was asked to create UDF filesystem for CD-R on existing image file.

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: UDF filesystem image with Write-Once UDF Access Type

[Roald Strauss]

Hey all


I'm a bit pressed for time these days, so I won't be able to do any 
tests any time soon.

Also have to admit that I lost interest in UDF a bit, since we ended up 
looking towards hardware solutions for write-protections instead.

But when/if mkudffs includes an option to create a write-once / 
read-only filesystem while adding files to it at the same time (because 
how else am I gonna put files onto my read-only filesystem?), then it 
might become interesting again.


- Roald




Den 01/08/2019 kl. 10.44 skrev Pali Rohár:
> On Thursday 01 August 2019 09:35:30 Jan Kara wrote:
>> On Fri 12-07-19 12:02:24, Pali Rohár  wrote:
>>> Also in git master of udftools has mkduffs now new option --read-only
>>> which creates UDF image with Read-Only Access Type.
>> I've tested this and the kernel properly mounts the image read-only.
> Roald, can you test that problem which you described to me with
> read-only access type is now correctly fixed?
>

Re: UDF filesystem image with Write-Once UDF Access Type

[Jan Kara]

On Thu 01-08-19 10:57:55, Pali Rohár wrote:
> On Thursday 01 August 2019 10:38:00 Jan Kara wrote:
> > Hum, looks like a problem with mkudffs. Relevant debug messages look like:
> > 
> > UDF-fs: fs/udf/super.c:671:udf_check_vsd: Starting at sector 16 (2048 byte sectors)
> > UDF-fs: fs/udf/super.c:824:udf_load_pvoldesc: recording time 2019/08/01 09:47 (1078)
> > UDF-fs: fs/udf/super.c:836:udf_load_pvoldesc: volIdent[] = 'LinuxUDF'
> > UDF-fs: fs/udf/super.c:844:udf_load_pvoldesc: volSetIdent[] = '1564645645200563LinuxUDF'
> > UDF-fs: fs/udf/super.c:1462:udf_load_logicalvol: Partition (0:0) type 1 on volume 1
> > UDF-fs: fs/udf/super.c:1462:udf_load_logicalvol: Partition (1:0) type 2 on volume 1
> > UDF-fs: fs/udf/super.c:1471:udf_load_logicalvol: FileSet found in LogicalVolDesc at block=0, partition=1
> > UDF-fs: fs/udf/super.c:1218:udf_load_partdesc: Searching map: (0 == 0)
> > UDF-fs: fs/udf/super.c:1060:udf_fill_partdesc_info: Partition (0 type 1511) starts at physical 288, block length 524000
> > UDF-fs: fs/udf/super.c:1060:udf_fill_partdesc_info: Partition (1 type 2012) starts at physical 288, block length 524000
> > UDF-fs: fs/udf/misc.c:223:udf_read_tagged: location mismatch block 524287, tag 0 != 523999
> > UDF-fs: error (device ubdb): udf_read_inode: (ino 524287) failed !bh
> > 
> > So the fact that location tag was 0 in block 524287 (which should contain
> > VAT inode) suggests there's something fishy with how / where mkudffs
> > creates the VAT inode. Can you have a look?
> > 
> > BTW, mkudffs messages look like:
> > filename=/tmp/image
> > label=LinuxUDF
> > uuid=1564645645200563
> > blocksize=2048
> > blocks=524288
> > udfrev=2.01
> > vatblock=319
> > start=0, blocks=16, type=RESERVED 
> > start=16, blocks=4, type=VRS 
> > start=20, blocks=76, type=USPACE 
> > start=96, blocks=16, type=MVDS 
> > start=112, blocks=16, type=USPACE 
> > start=128, blocks=1, type=LVID 
> > start=129, blocks=95, type=USPACE 
> > start=224, blocks=16, type=RVDS 
> > start=240, blocks=16, type=USPACE 
> > start=256, blocks=1, type=ANCHOR 
> > start=257, blocks=31, type=USPACE 
> > start=288, blocks=524000, type=PSPACE 
> > 
> > which suggests that VAT was indeed allocated somewhere in the beginning of
> > the partition.
> 
> For write-once media you are not able to modify size of UDF partition.
> So if you are creating image for CD-R disc, you need to specify size of
> UDF filesystem to match size of CD-R disc. VAT is always burned to the
> last block of current track on CD-R.
> 
> Therefore if you had pre-allocated big image file for CD-R and then you
> run mkudffs for cdr on it, you lost information what is the last used
> block on that cdr image. Normally for optical drivers kernel use mmc
> commands to retrieve last block of current session and based on it find
> VAT. But image files loaded via /dev/loop are not optical drivers and
> therefore do not have ability "hardware" ability to ask where is the
> last used block. IIRC in this case kernel just fallback to the last
> block of block device for VAT, which in this case is not correct.
> 
> What should help is to truncate image file to "correct" size after
> running mkudffs with --media-type=cdr. Maybe mkudffs itself should do it
> when was asked to create UDF filesystem for CD-R on existing image file.

Ah, right. Thanks for explanation. I somehow assumed that mkudffs will be
considering the last block of the "device file" the last block that it has
to record but you're right that on second though that doesn't really make
sense.
								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR