Linux UDF support

Linux UDF support

[Pali Rohár]

[-- Attachment #1: Type: Text/Plain, Size: 1211 bytes --]

Hi,

I would like to know what is state of linux UDF driver. It is 
experimental or is now suitable for storing data?

According to wikipedia [1] UDF has open specification format and 
can be used also for HDDs (not only optical discs).

In OS support table is written that all major and other minor OSs 
support UDF FS (without needs for additional programs).

So it looks like UDF is good candidate for multi OS filesystem. 
Are there any disadvantages for using UDF on e.g USB flash disk? 
(when I want read/write support on Linux, Windows 7 and Mac OS X)

Because lot of manuals say that FAT32 (or NTFS) is only one 
solution for using USB flash disk on more OS.

On wikipedia there is one note about linux: Write support is only 
up to UDF version 2.01. Is this restriction still valid?

What will happen if I try to mount FS with UDF version 2.60 in 
R/W mode on linux? It will fallback to R/O mode? Or newly written 
files will be in previous (2.01) versions?

And last question: Is there some fsck tool for UDF? Or at least 
tool which print if FS is in inconsistent state?

[1] - http://en.wikipedia.org/wiki/Universal_Disk_Format

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Linux UDF support

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2008 bytes --]

On 2014-08-24 08:46, Pali Rohár wrote:
> Hi,
> 
> I would like to know what is state of linux UDF driver. It is 
> experimental or is now suitable for storing data?
> 
I know that read support works for every version I have tested, but I've
only tested it reading data from DVD's and Blu-Ray discs, so I don't
know how well it works for other purposes.
> According to wikipedia [1] UDF has open specification format and 
> can be used also for HDDs (not only optical discs).
> 
> In OS support table is written that all major and other minor OSs 
> support UDF FS (without needs for additional programs).
> 
> So it looks like UDF is good candidate for multi OS filesystem. 
> Are there any disadvantages for using UDF on e.g USB flash disk? 
> (when I want read/write support on Linux, Windows 7 and Mac OS X)
If you are going to go that way, make sure to use the Spared Build, as
otherwise you will run in to the same media wear-out issues that NTFS
and FAT have.  Also, keep in mind that pre-Vista Windows and pre-10.4
OSX don't have very good support for the newer formats.
> Because lot of manuals say that FAT32 (or NTFS) is only one 
> solution for using USB flash disk on more OS.
> 
> On wikipedia there is one note about linux: Write support is only 
> up to UDF version 2.01. Is this restriction still valid?
I do know that we support reading UDF 2.60 (I've used linux to read
Blu-Ray discs), but I have no idea about write support for versions
above 2.01.
> What will happen if I try to mount FS with UDF version 2.60 in 
> R/W mode on linux? It will fallback to R/O mode? Or newly written 
> files will be in previous (2.01) versions?
> 
> And last question: Is there some fsck tool for UDF? Or at least 
> tool which print if FS is in inconsistent state?
Most Linux distributions have a package called udftools, the upstream
URL given by portage is http://sf.net/projects/linux-udf/
> 
> [1] - http://en.wikipedia.org/wiki/Universal_Disk_Format
> 



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]

Re: Linux UDF support

[Pali Rohár]

[-- Attachment #1: Type: Text/Plain, Size: 2710 bytes --]

Hi,

On Monday 25 August 2014 14:45:13 Austin S Hemmelgarn wrote:
> On 2014-08-24 08:46, Pali Rohár wrote:
> > Hi,
> > 
> > I would like to know what is state of linux UDF driver. It
> > is experimental or is now suitable for storing data?
> 
> I know that read support works for every version I have
> tested, but I've only tested it reading data from DVD's and
> Blu-Ray discs, so I don't know how well it works for other
> purposes.
> 

Ok. I'm thinking about using UDF on HDD and usb flash disks (not 
on optical medium). And here I need write support too.

> > According to wikipedia [1] UDF has open specification format
> > and can be used also for HDDs (not only optical discs).
> > 
> > In OS support table is written that all major and other
> > minor OSs support UDF FS (without needs for additional
> > programs).
> > 
> > So it looks like UDF is good candidate for multi OS
> > filesystem. Are there any disadvantages for using UDF on
> > e.g USB flash disk? (when I want read/write support on
> > Linux, Windows 7 and Mac OS X)
> 
> If you are going to go that way, make sure to use the Spared
> Build, as otherwise you will run in to the same media
> wear-out issues that NTFS and FAT have.  Also, keep in mind
> that pre-Vista Windows and pre-10.4 OSX don't have very good
> support for the newer formats.
> 

What is Spared Build? And how to use it?

Problem with NTFS is that linux driver has write support marked 
as experimental. FAT has problems with big files and for exFAT 
there is no driver in linux kernel yet...

> > Because lot of manuals say that FAT32 (or NTFS) is only one
> > solution for using USB flash disk on more OS.
> > 
> > On wikipedia there is one note about linux: Write support is
> > only up to UDF version 2.01. Is this restriction still
> > valid?
> 
> I do know that we support reading UDF 2.60 (I've used linux to
> read Blu-Ray discs), but I have no idea about write support
> for versions above 2.01.
> 
> > What will happen if I try to mount FS with UDF version 2.60
> > in R/W mode on linux? It will fallback to R/O mode? Or
> > newly written files will be in previous (2.01) versions?
> > 
> > And last question: Is there some fsck tool for UDF? Or at
> > least tool which print if FS is in inconsistent state?
> 
> Most Linux distributions have a package called udftools, the
> upstream URL given by portage is
> http://sf.net/projects/linux-udf/
> 

That project does not have udf fsck tool :-(

> > [1] - http://en.wikipedia.org/wiki/Universal_Disk_Format

Ok, I will wait for response from maintainer Jan, he probably 
would know more...

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Linux UDF support

[Austin S Hemmelgarn]

On 2014-08-25 09:24, Pali Rohár wrote:
> Hi,
> 
> On Monday 25 August 2014 14:45:13 Austin S Hemmelgarn wrote:
>> On 2014-08-24 08:46, Pali Rohár wrote:
>>> Hi,
>>>
>>> I would like to know what is state of linux UDF driver. It
>>> is experimental or is now suitable for storing data?
>>
>> I know that read support works for every version I have
>> tested, but I've only tested it reading data from DVD's and
>> Blu-Ray discs, so I don't know how well it works for other
>> purposes.
>>
> 
> Ok. I'm thinking about using UDF on HDD and usb flash disks (not 
> on optical medium). And here I need write support too.
> 
>>> According to wikipedia [1] UDF has open specification format
>>> and can be used also for HDDs (not only optical discs).
>>>
>>> In OS support table is written that all major and other
>>> minor OSs support UDF FS (without needs for additional
>>> programs).
>>>
>>> So it looks like UDF is good candidate for multi OS
>>> filesystem. Are there any disadvantages for using UDF on
>>> e.g USB flash disk? (when I want read/write support on
>>> Linux, Windows 7 and Mac OS X)
>>
>> If you are going to go that way, make sure to use the Spared
>> Build, as otherwise you will run in to the same media
>> wear-out issues that NTFS and FAT have.  Also, keep in mind
>> that pre-Vista Windows and pre-10.4 OSX don't have very good
>> support for the newer formats.
>>
> 
> What is Spared Build? And how to use it?
Sparse Build is one of the extensions that was added in 1.50.  It
provides a table indicating sectors that have worn out and therefore
should be left unused.  The general idea is that rewritable media is
almost always limited in the number of writes it can handle to a given
location, and when you exceed that number, you get data corruption at
that location.  This is usually most noticeable on flash media, but it
happens on {CD,DVD,BD}-RW discs and hard drives as well.  FAT has no
provisions for handling this, and NTFS and ext4 just return the
corrupted data.  UDF however, has sufficient error correction ability
(because it was designed with optical media in mind) that it can usually
detect the corruption, recover the corrupted data, and then if you are
using the Sparse Build format, mark that block as unusable, and write
the new data out elsewhere.
> 
> Problem with NTFS is that linux driver has write support marked 
> as experimental. FAT has problems with big files and for exFAT 
> there is no driver in linux kernel yet...
> 
For NTFS, you should be using NTFS-3G, not the kernel driver, and I
personally wouldn't trust the OS X driver for NTFS for anything beyond
read-only usage (I only barely trust NTFS-3G as it is, and have about as
much trust in the Linux kernel driver for it as I would in somebody
trying to convince me that arsenic is good for you).  And for exFAT
there is FUSE module.  I haven't tested either for much other than
read-only scenarios, but I can confirm that they are great for data
recovery.
>>> Because lot of manuals say that FAT32 (or NTFS) is only one
>>> solution for using USB flash disk on more OS.
>>>
>>> On wikipedia there is one note about linux: Write support is
>>> only up to UDF version 2.01. Is this restriction still
>>> valid?
>>
>> I do know that we support reading UDF 2.60 (I've used linux to
>> read Blu-Ray discs), but I have no idea about write support
>> for versions above 2.01.
>>
>>> What will happen if I try to mount FS with UDF version 2.60
>>> in R/W mode on linux? It will fallback to R/O mode? Or
>>> newly written files will be in previous (2.01) versions?
>>>
>>> And last question: Is there some fsck tool for UDF? Or at
>>> least tool which print if FS is in inconsistent state?
>>
>> Most Linux distributions have a package called udftools, the
>> upstream URL given by portage is
>> http://sf.net/projects/linux-udf/
>>
> 
> That project does not have udf fsck tool :-(
IIRC, there are a lot of issues that UDF will recover from gracefully.
It has ridiculously good error correction abilities, and the newer
versions even have the option of duplicate copies of filesystem metadata
to provide even more resilience.
> 
>>> [1] - http://en.wikipedia.org/wiki/Universal_Disk_Format
> 
> Ok, I will wait for response from maintainer Jan, he probably 
> would know more...
> 

Re: Linux UDF support

[Pali Rohár]

[-- Attachment #1: Type: Text/Plain, Size: 5666 bytes --]

Hi,

On Monday 25 August 2014 16:05:27 Austin S Hemmelgarn wrote:
> On 2014-08-25 09:24, Pali Rohár wrote:
> > Hi,
> > 
> > On Monday 25 August 2014 14:45:13 Austin S Hemmelgarn wrote:
> >> On 2014-08-24 08:46, Pali Rohár wrote:
> >>> Hi,
> >>> 
> >>> I would like to know what is state of linux UDF driver. It
> >>> is experimental or is now suitable for storing data?
> >> 
> >> I know that read support works for every version I have
> >> tested, but I've only tested it reading data from DVD's and
> >> Blu-Ray discs, so I don't know how well it works for other
> >> purposes.
> > 
> > Ok. I'm thinking about using UDF on HDD and usb flash disks
> > (not on optical medium). And here I need write support too.
> > 
> >>> According to wikipedia [1] UDF has open specification
> >>> format and can be used also for HDDs (not only optical
> >>> discs).
> >>> 
> >>> In OS support table is written that all major and other
> >>> minor OSs support UDF FS (without needs for additional
> >>> programs).
> >>> 
> >>> So it looks like UDF is good candidate for multi OS
> >>> filesystem. Are there any disadvantages for using UDF on
> >>> e.g USB flash disk? (when I want read/write support on
> >>> Linux, Windows 7 and Mac OS X)
> >> 
> >> If you are going to go that way, make sure to use the
> >> Spared Build, as otherwise you will run in to the same
> >> media wear-out issues that NTFS and FAT have.  Also, keep
> >> in mind that pre-Vista Windows and pre-10.4 OSX don't have
> >> very good support for the newer formats.
> > 
> > What is Spared Build? And how to use it?
> 
> Sparse Build is one of the extensions that was added in 1.50. 
> It provides a table indicating sectors that have worn out and
> therefore should be left unused.  The general idea is that
> rewritable media is almost always limited in the number of
> writes it can handle to a given location, and when you exceed
> that number, you get data corruption at that location.  This
> is usually most noticeable on flash media, but it happens on
> {CD,DVD,BD}-RW discs and hard drives as well.  FAT has no
> provisions for handling this, and NTFS and ext4 just return
> the corrupted data.  UDF however, has sufficient error
> correction ability (because it was designed with optical
> media in mind) that it can usually detect the corruption,
> recover the corrupted data, and then if you are using the
> Sparse Build format, mark that block as unusable, and write
> the new data out elsewhere.
> 

Looks good. But I think that when I reformat disk I will lost all 
these data about detected corruption... Or will duplicate mkudffs 
calls preserve these information about HW errors?

So if I format block device (hdd) with mkudffs and rev 0x0201 it 
is automatically enabled? Or is something more needed?

> > Problem with NTFS is that linux driver has write support
> > marked as experimental. FAT has problems with big files and
> > for exFAT there is no driver in linux kernel yet...
> 
> For NTFS, you should be using NTFS-3G, not the kernel driver,
> and I personally wouldn't trust the OS X driver for NTFS for
> anything beyond read-only usage (I only barely trust NTFS-3G
> as it is, and have about as much trust in the Linux kernel
> driver for it as I would in somebody trying to convince me
> that arsenic is good for you).  And for exFAT there is FUSE
> module.  I haven't tested either for much other than
> read-only scenarios, but I can confirm that they are great
> for data recovery.
> 

I'm not sure if using fuse for anything other then experiments is 
reasonable. And speed of fuse (ntfs-3g or exfat) is also 
horrible. And now I think that only usage is read-only data 
recovery...

So still would like to hear what is state of udf read/write 
support in kernel (for usb/flash/hdd disks).

> >>> Because lot of manuals say that FAT32 (or NTFS) is only
> >>> one solution for using USB flash disk on more OS.
> >>> 
> >>> On wikipedia there is one note about linux: Write support
> >>> is only up to UDF version 2.01. Is this restriction still
> >>> valid?
> >> 
> >> I do know that we support reading UDF 2.60 (I've used linux
> >> to read Blu-Ray discs), but I have no idea about write
> >> support for versions above 2.01.
> >> 
> >>> What will happen if I try to mount FS with UDF version
> >>> 2.60 in R/W mode on linux? It will fallback to R/O mode?
> >>> Or newly written files will be in previous (2.01)
> >>> versions?
> >>> 
> >>> And last question: Is there some fsck tool for UDF? Or at
> >>> least tool which print if FS is in inconsistent state?
> >> 
> >> Most Linux distributions have a package called udftools,
> >> the upstream URL given by portage is
> >> http://sf.net/projects/linux-udf/
> > 
> > That project does not have udf fsck tool :-(
> 
> IIRC, there are a lot of issues that UDF will recover from
> gracefully. It has ridiculously good error correction
> abilities, and the newer versions even have the option of
> duplicate copies of filesystem metadata to provide even more
> resilience.
> 

But this depends on driver implementation, rigth? Is linux kernel 
driver able to recovery and fix (on the fly) errors at mount time 
(like ubifs or ext4)? Somebody said that ubifs does not need fsck 
because kernel driver doing similar job of recovery at mount 
time... It is same for udf implementation in linux?

> >>> [1] - http://en.wikipedia.org/wiki/Universal_Disk_Format
> > 
> > Ok, I will wait for response from maintainer Jan, he
> > probably would know more...

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Linux UDF support

[Jan Kara]

  Hello,

On Sun 24-08-14 14:46:46, Pali Rohár wrote:
> I would like to know what is state of linux UDF driver. It is 
> experimental or is now suitable for storing data?
  The kernel driver is quite stable. I would happily use it for USB sticks
etc. I would be reluctant to use it for say /home because we don't have any
reasonable repair tool for the filesystem (there are format checkers out
there but they mostly tell you there is a problem, however there's not a
decent tool which would fix the filesystem AFAIK).

> According to wikipedia [1] UDF has open specification format and 
> can be used also for HDDs (not only optical discs).
  Correct.

> In OS support table is written that all major and other minor OSs 
> support UDF FS (without needs for additional programs).
> 
> So it looks like UDF is good candidate for multi OS filesystem. 
> Are there any disadvantages for using UDF on e.g USB flash disk? 
> (when I want read/write support on Linux, Windows 7 and Mac OS X)
  Yes, it is a good candidate for exchange between different OSs. I know
people who use it as such.

> Because lot of manuals say that FAT32 (or NTFS) is only one 
> solution for using USB flash disk on more OS.
> 
> On wikipedia there is one note about linux: Write support is only 
> up to UDF version 2.01. Is this restriction still valid?
  That is correct.

> What will happen if I try to mount FS with UDF version 2.60 in 
> R/W mode on linux? It will fallback to R/O mode? Or newly written 
> files will be in previous (2.01) versions?
  The kernel refuses to mount the filesystem read-write if its revision is
> 2.01. But frankly for interchange between OSs you don't want to use
advanced features anyway (lower chance fs will be supported) and there's not
much benefit since they are mostly aimed at optical media.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR