[PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

This is a new patch for VFAT long filename support, replacing the one
that I posted last month. It retains a lot more functionality then the
previous patch.

A FAQ will be posted immediately after this patch to answer the
questions that were raised from the previous discussion.

Cheers, Tridge


------------

When VFAT_FS_DUALNAMES is disabled we avoid the creation of 8.3 short
filenames for files on VFAT filesystems that require a long name. The
patch uses a pattern of 11 bytes in the directory entry which contains
invalid characters such that it cannot be considered to be a valid short
filename.

Signed-off-by: Andrew Tridgell <tridge@samba.org>
Acked-by: Dave Kleikamp <shaggy@linux.vnet.ibm.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 fs/fat/Kconfig      |   20 +++++++++++++++++
 fs/fat/dir.c        |   15 ++++++-------
 fs/fat/namei_vfat.c |   59 +++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 86 insertions(+), 8 deletions(-)

diff --git a/fs/fat/Kconfig b/fs/fat/Kconfig
index 182f9ff..907a5de 100644
--- a/fs/fat/Kconfig
+++ b/fs/fat/Kconfig
@@ -74,6 +74,26 @@ config VFAT_FS
 	  To compile this as a module, choose M here: the module will be called
 	  vfat.
 
+config VFAT_FS_DUALNAMES
+	bool "VFAT dual names support"
+	depends on VFAT_FS
+	help
+	  This option provides support for dual filenames on VFAT filesystems.
+	  If this option is disabled then file creation will either put
+	  a short (8.3) name or a long name on the file, but never both.
+	  The field where a shortname would normally go is filled with
+	  invalid characters such that it cannot be considered a valid
+	  short filename.
+
+	  That means that long filenames created with this option
+	  disabled will not be accessible at all to operating systems
+	  that do not understand the VFAT extensions.
+
+	  Users considering enabling this option should consider the implications
+	  of any patents that may exist on dual filenames in VFAT.
+
+	  If unsure, say N
+
 config FAT_DEFAULT_CODEPAGE
 	int "Default codepage for FAT"
 	depends on MSDOS_FS || VFAT_FS
diff --git a/fs/fat/dir.c b/fs/fat/dir.c
index 38ff75a..cd5d3ec 100644
--- a/fs/fat/dir.c
+++ b/fs/fat/dir.c
@@ -420,14 +420,13 @@ parse_record:
 			}
 			i += chl;
 		}
-		if (!last_u)
-			continue;
-
-		/* Compare shortname */
-		bufuname[last_u] = 0x0000;
-		len = fat_uni_to_x8(sbi, bufuname, bufname, sizeof(bufname));
-		if (fat_name_match(sbi, name, name_len, bufname, len))
-			goto found;
+		if (last_u) {
+			/* Compare shortname */
+			bufuname[last_u] = 0x0000;
+			len = fat_uni_to_x8(sbi, bufuname, bufname, sizeof(bufname));
+			if (fat_name_match(sbi, name, name_len, bufname, len))
+				goto found;
+		}
 
 		if (nr_slots) {
 			void *longname = unicode + FAT_MAX_UNI_CHARS;
diff --git a/fs/fat/namei_vfat.c b/fs/fat/namei_vfat.c
index 73471b7..894f44d 100644
--- a/fs/fat/namei_vfat.c
+++ b/fs/fat/namei_vfat.c
@@ -22,6 +22,7 @@
 #include <linux/smp_lock.h>
 #include <linux/buffer_head.h>
 #include <linux/namei.h>
+#include <linux/random.h>
 #include "fat.h"
 
 /*
@@ -586,6 +587,59 @@ xlate_to_uni(const unsigned char *name, int len, unsigned char *outname,
 	return 0;
 }
 
+#ifndef CONFIG_VFAT_FS_DUALNAMES
+/*
+ * build a 11 byte 8.3 buffer which is not a short filename. We want 11
+ * bytes which:
+ *    - will be seen as a constant string to all APIs on Linux and Windows
+ *    - cannot be matched with wildcard patterns
+ *    - cannot be used to access the file
+ *    - has a low probability of collision within a directory
+ *    - has an invalid 3 byte extension
+ *    - contains at least one non-space and non-nul byte
+ */
+static void vfat_build_dummy_83_buffer(struct inode *dir, char *msdos_name)
+{
+	u32 rand_num = random32() & 0x3FFFFFFF;
+	int i;
+
+	/* a value of zero would leave us with only nul and spaces,
+	 * which would not work with older linux systems
+	 */
+	if (rand_num == 0)
+		rand_num = 1;
+
+	/* we start with a space followed by nul as spaces at the
+	 * start of an entry are trimmed in FAT, which means that
+	 * starting the 11 bytes with 0x20 0x00 gives us a value which
+	 * cannot be used to access the file. It also means that the
+	 * value as seen from all Windows and Linux APIs is a constant
+	 */
+	msdos_name[0]  = ' ';
+	msdos_name[1]  = 0;
+
+	/* we use / and 2 nul bytes for the extension. These are
+	 * invalid in FAT and mean that utilities that show the
+	 * directory show no extension, but still work via the long
+	 * name for old Linux kernels
+	 */
+	msdos_name[8]  = '/';
+	msdos_name[9]  = 0;
+	msdos_name[10] = 0;
+
+	/*
+	 * fill the remaining 6 bytes with random invalid values
+	 * This gives us a low collision rate, which means a low
+	 * chance of problems with chkdsk.exe and WindowsXP
+	 */
+	for (i = 2; i < 8; i++) {
+		msdos_name[i] = rand_num & 0x1F;
+		rand_num >>= 5;
+	}
+}
+#endif
+
+
 static int vfat_build_slots(struct inode *dir, const unsigned char *name,
 			    int len, int is_dir, int cluster,
 			    struct timespec *ts,
@@ -628,6 +682,11 @@ static int vfat_build_slots(struct inode *dir, const unsigned char *name,
 		goto shortname;
 	}
 
+#ifndef CONFIG_VFAT_FS_DUALNAMES
+	vfat_build_dummy_83_buffer(dir, msdos_name);
+	lcase = 0;
+#endif
+
 	/* build the entry of long file name */
 	cksum = fat_checksum(msdos_name);
 
-- 
1.6.0.4

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[H. Peter Anvin]

tridge@samba.org wrote:
> This is a new patch for VFAT long filename support, replacing the one
> that I posted last month. It retains a lot more functionality then the
> previous patch.
> 
> A FAQ will be posted immediately after this patch to answer the
> questions that were raised from the previous discussion.
> 

>From a purely technical perspective, there is a major advantage to this
patch: it creates a "pure VFAT" filesystem, without a spurious filename
alias which still occupies namespace, hiddenly, and therefore could
cause an ill-defined amount of problems.  UMSDOS at least had the
decency to not expose its shortnames to a longname-aware OS.

However, as such it really should be paired with a "don't even recognize
the shortname if a longname exists" option.

It's also questionable IMO if this shouldn't be another FAT superdriver,
just as we have VFAT, MS-DOS etc. we could have "purevfat".

	-hpa

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

hpa wrote:
 > However, as such it really should be paired with a "don't even recognize
 > the shortname if a longname exists" option.

Right, we could just skip the 8.3 name matching and go straight to the
longname match in fat_search_long(). I wonder if anyone ever relies on
the 8.3 matching when a file has a long name on Linux?

 > It's also questionable IMO if this shouldn't be another FAT superdriver,
 > just as we have VFAT, MS-DOS etc. we could have "purevfat".

It would be only a few lines of code difference between the two
drivers - is it worth the maintainence overhead of splitting it up?

Maybe a mount option to ignore 8.3 names for files with long names
would be better? Perhaps even the default?

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[OGAWA Hirofumi]

tridge@samba.org writes:

> hpa wrote:
>  > However, as such it really should be paired with a "don't even recognize
>  > the shortname if a longname exists" option.
>
> Right, we could just skip the 8.3 name matching and go straight to the
> longname match in fat_search_long(). I wonder if anyone ever relies on
> the 8.3 matching when a file has a long name on Linux?

Wine or such is using it.

>  > It's also questionable IMO if this shouldn't be another FAT superdriver,
>  > just as we have VFAT, MS-DOS etc. we could have "purevfat".
>
> It would be only a few lines of code difference between the two
> drivers - is it worth the maintainence overhead of splitting it up?
>
> Maybe a mount option to ignore 8.3 names for files with long names
> would be better? Perhaps even the default?

It sounds like to be start of yet another UMSDOS with minimal one. I
wouldn't like to repeat UMSDOS history, i.e. I think it needs real users
and developers. If not, I wouldn't like to add.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Jan Engelhardt]

On Sat, 27 Jun 2009 10:48:25 +0900, OGAWA Hirofumi wrote:

>It sounds like to be start of yet another UMSDOS with minimal one. I 
>wouldn't like to repeat UMSDOS history, i.e. I think it needs real 
>users and developers. If not, I wouldn't like to add.

And in that regard, there is http://posixovl.sf.net/ for everybody who 
needs UMSDOS functionality.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[OGAWA Hirofumi]

tridge@samba.org writes:

> This is a new patch for VFAT long filename support, replacing the one
> that I posted last month. It retains a lot more functionality then the
> previous patch.
>
> A FAQ will be posted immediately after this patch to answer the
> questions that were raised from the previous discussion.

Looks good to me. It seems to be trying to be minimal change, and I
guess it would be worth a try for realworld problem.  Like FAQ is
saying, the default might be arguable though.

> When VFAT_FS_DUALNAMES is disabled we avoid the creation of 8.3 short
> filenames for files on VFAT filesystems that require a long name. The
> patch uses a pattern of 11 bytes in the directory entry which contains
> invalid characters such that it cannot be considered to be a valid short
> filename.
>
> Signed-off-by: Andrew Tridgell <tridge@samba.org>
> Acked-by: Dave Kleikamp <shaggy@linux.vnet.ibm.com>
> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  fs/fat/Kconfig      |   20 +++++++++++++++++
>  fs/fat/dir.c        |   15 ++++++-------
>  fs/fat/namei_vfat.c |   59 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 86 insertions(+), 8 deletions(-)
>
> diff --git a/fs/fat/Kconfig b/fs/fat/Kconfig
> index 182f9ff..907a5de 100644
> --- a/fs/fat/Kconfig
> +++ b/fs/fat/Kconfig
> @@ -74,6 +74,26 @@ config VFAT_FS
>  	  To compile this as a module, choose M here: the module will be called
>  	  vfat.
>  
> +config VFAT_FS_DUALNAMES
> +	bool "VFAT dual names support"
> +	depends on VFAT_FS
> +	help
> +	  This option provides support for dual filenames on VFAT filesystems.
> +	  If this option is disabled then file creation will either put
> +	  a short (8.3) name or a long name on the file, but never both.
> +	  The field where a shortname would normally go is filled with
> +	  invalid characters such that it cannot be considered a valid
> +	  short filename.
> +
> +	  That means that long filenames created with this option
> +	  disabled will not be accessible at all to operating systems
> +	  that do not understand the VFAT extensions.
> +
> +	  Users considering enabling this option should consider the implications
> +	  of any patents that may exist on dual filenames in VFAT.
> +
> +	  If unsure, say N
> +
>  config FAT_DEFAULT_CODEPAGE
>  	int "Default codepage for FAT"
>  	depends on MSDOS_FS || VFAT_FS
> diff --git a/fs/fat/dir.c b/fs/fat/dir.c
> index 38ff75a..cd5d3ec 100644
> --- a/fs/fat/dir.c
> +++ b/fs/fat/dir.c
> @@ -420,14 +420,13 @@ parse_record:
>  			}
>  			i += chl;
>  		}
> -		if (!last_u)
> -			continue;
> -
> -		/* Compare shortname */
> -		bufuname[last_u] = 0x0000;
> -		len = fat_uni_to_x8(sbi, bufuname, bufname, sizeof(bufname));
> -		if (fat_name_match(sbi, name, name_len, bufname, len))
> -			goto found;
> +		if (last_u) {
> +			/* Compare shortname */
> +			bufuname[last_u] = 0x0000;
> +			len = fat_uni_to_x8(sbi, bufuname, bufname, sizeof(bufname));
> +			if (fat_name_match(sbi, name, name_len, bufname, len))
> +				goto found;
> +		}
>  
>  		if (nr_slots) {
>  			void *longname = unicode + FAT_MAX_UNI_CHARS;
> diff --git a/fs/fat/namei_vfat.c b/fs/fat/namei_vfat.c
> index 73471b7..894f44d 100644
> --- a/fs/fat/namei_vfat.c
> +++ b/fs/fat/namei_vfat.c
> @@ -22,6 +22,7 @@
>  #include <linux/smp_lock.h>
>  #include <linux/buffer_head.h>
>  #include <linux/namei.h>
> +#include <linux/random.h>
>  #include "fat.h"
>  
>  /*
> @@ -586,6 +587,59 @@ xlate_to_uni(const unsigned char *name, int len, unsigned char *outname,
>  	return 0;
>  }
>  
> +#ifndef CONFIG_VFAT_FS_DUALNAMES
> +/*
> + * build a 11 byte 8.3 buffer which is not a short filename. We want 11
> + * bytes which:
> + *    - will be seen as a constant string to all APIs on Linux and Windows
> + *    - cannot be matched with wildcard patterns
> + *    - cannot be used to access the file
> + *    - has a low probability of collision within a directory
> + *    - has an invalid 3 byte extension
> + *    - contains at least one non-space and non-nul byte
> + */
> +static void vfat_build_dummy_83_buffer(struct inode *dir, char *msdos_name)
> +{
> +	u32 rand_num = random32() & 0x3FFFFFFF;
> +	int i;
> +
> +	/* a value of zero would leave us with only nul and spaces,
> +	 * which would not work with older linux systems
> +	 */
> +	if (rand_num == 0)
> +		rand_num = 1;
> +
> +	/* we start with a space followed by nul as spaces at the
> +	 * start of an entry are trimmed in FAT, which means that
> +	 * starting the 11 bytes with 0x20 0x00 gives us a value which
> +	 * cannot be used to access the file. It also means that the
> +	 * value as seen from all Windows and Linux APIs is a constant
> +	 */
> +	msdos_name[0]  = ' ';
> +	msdos_name[1]  = 0;
> +
> +	/* we use / and 2 nul bytes for the extension. These are
> +	 * invalid in FAT and mean that utilities that show the
> +	 * directory show no extension, but still work via the long
> +	 * name for old Linux kernels
> +	 */
> +	msdos_name[8]  = '/';
> +	msdos_name[9]  = 0;
> +	msdos_name[10] = 0;
> +
> +	/*
> +	 * fill the remaining 6 bytes with random invalid values
> +	 * This gives us a low collision rate, which means a low
> +	 * chance of problems with chkdsk.exe and WindowsXP
> +	 */
> +	for (i = 2; i < 8; i++) {
> +		msdos_name[i] = rand_num & 0x1F;
> +		rand_num >>= 5;
> +	}
> +}
> +#endif
> +
> +
>  static int vfat_build_slots(struct inode *dir, const unsigned char *name,
>  			    int len, int is_dir, int cluster,
>  			    struct timespec *ts,
> @@ -628,6 +682,11 @@ static int vfat_build_slots(struct inode *dir, const unsigned char *name,
>  		goto shortname;
>  	}
>  
> +#ifndef CONFIG_VFAT_FS_DUALNAMES
> +	vfat_build_dummy_83_buffer(dir, msdos_name);
> +	lcase = 0;
> +#endif
> +
>  	/* build the entry of long file name */
>  	cksum = fat_checksum(msdos_name);

-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Jan Engelhardt]

On Sat, 27 Jun 2009 05:19:33 +1000, tridge@samba.org wrote:
>
>When VFAT_FS_DUALNAMES is disabled we avoid the creation of 8.3
>short filenames for files on VFAT filesystems that require a long
>name. The patch uses a pattern of 11 bytes in the directory entry
>which contains invalid characters such that it cannot be considered
>to be a valid short filename.

Can't we make this a runtime option?

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Jan,

 > Can't we make this a runtime option?

John may be able to give you a more detailed answer, but the short
answer is that it is much safer legally to not have the code in the
binary kernel image at all.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Jamie Lokier]

tridge@samba.org wrote:
> Hi Jan,
> 
>  > Can't we make this a runtime option?
> 
> John may be able to give you a more detailed answer, but the short
> answer is that it is much safer legally to not have the code in the
> binary kernel image at all.

Understood.

For those of us who don't want to omit the code, because we like
compatibility and we're not in affected countries, or for research, it
would be useful to have it as a mount option.

So there should be these compile-time options:

    1. CONFIG_VFAT_FS_DUALNAMES disabled:  Don't create shortnames.

    2. CONFIG_VFAT_FS_DUALNAMES enabled: Create shortnames, unless
       mount option "dualnames=no" is given, in which case that mount
       behaves as if CONFIG_VFAT_FS_DUALNAMES is disabled.

-- Jamie

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Jan Engelhardt]

On Sunday 2009-06-28 23:57, Jamie Lokier wrote:
>For those of us who don't want to omit the code, because we like
>compatibility and we're not in affected countries, or for research, it
>would be useful to have it as a mount option.
>
>So there should be these compile-time options:
>    2. CONFIG_VFAT_FS_DUALNAMES enabled: Create shortnames, unless
>       mount option "dualnames=no" is given, in which case that mount
>       behaves as if CONFIG_VFAT_FS_DUALNAMES is disabled.

If you are not in an affected country, why would you even want
to disable dualnames? [ on a per-mount basis...]

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Jamie Lokier]

Jan Engelhardt wrote:
> 
> On Sunday 2009-06-28 23:57, Jamie Lokier wrote:
> >For those of us who don't want to omit the code, because we like
> >compatibility and we're not in affected countries, or for research, it
> >would be useful to have it as a mount option.
> >
> >So there should be these compile-time options:
> >    2. CONFIG_VFAT_FS_DUALNAMES enabled: Create shortnames, unless
> >       mount option "dualnames=no" is given, in which case that mount
> >       behaves as if CONFIG_VFAT_FS_DUALNAMES is disabled.
> 
> If you are not in an affected country, why would you even want
> to disable dualnames? [ on a per-mount basis...]

   1. To test it's behaviour, without changing the whole running system.

   2. To produce disk images which are the same as would be produced
      in an affected country, or would be produced by a Linux distro
      adopting the behaviour.

-- Jamie

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Jamie,

 > For those of us who don't want to omit the code, because we like
 > compatibility and we're not in affected countries, or for research, it
 > would be useful to have it as a mount option.

For research I think having to recompile the module is not a big
burden.

The compatibility argument is more debatable, and if the patch
involved the loss of a large degree of functionality then I would
probably agree. As it is, the patch loses very little functionality -
it is hard to find real scenarios where not storing a 8.3 name along
with the long name will actually cause any problems.

I also don't think it is worth having a different kernel in different
countries for this change, again because the loss of functionality is
so small. The relevant patents exist in several countries (in various
forms), and while the US might be the country of choice for enforcing
patents it is not the only country where patents are a concern.

I would rather see us err on the side of caution and ensure that
Microsoft is not left with an opportunity to continue to spread FUD
about Linux with regards to these patents.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Eric W. Biederman]

tridge@samba.org writes:

> This is a new patch for VFAT long filename support, replacing the one
> that I posted last month. It retains a lot more functionality then the
> previous patch.
>
> A FAQ will be posted immediately after this patch to answer the
> questions that were raised from the previous discussion.
>
> Cheers, Tridge
>
>
> ------------
>
> When VFAT_FS_DUALNAMES is disabled we avoid the creation of 8.3 short
> filenames for files on VFAT filesystems that require a long name. The
> patch uses a pattern of 11 bytes in the directory entry which contains
> invalid characters such that it cannot be considered to be a valid short
> filename.
>
> Signed-off-by: Andrew Tridgell <tridge@samba.org>
> Acked-by: Dave Kleikamp <shaggy@linux.vnet.ibm.com>
> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  fs/fat/Kconfig      |   20 +++++++++++++++++
>  fs/fat/dir.c        |   15 ++++++-------
>  fs/fat/namei_vfat.c |   59 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 86 insertions(+), 8 deletions(-)
>
> diff --git a/fs/fat/Kconfig b/fs/fat/Kconfig
> index 182f9ff..907a5de 100644
> --- a/fs/fat/Kconfig
> +++ b/fs/fat/Kconfig
> @@ -74,6 +74,26 @@ config VFAT_FS
>  	  To compile this as a module, choose M here: the module will be called
>  	  vfat.
>  
> +config VFAT_FS_DUALNAMES
> +	bool "VFAT dual names support"
> +	depends on VFAT_FS
> +	help
> +	  This option provides support for dual filenames on VFAT filesystems.
> +	  If this option is disabled then file creation will either put
> +	  a short (8.3) name or a long name on the file, but never both.
> +	  The field where a shortname would normally go is filled with
> +	  invalid characters such that it cannot be considered a valid
> +	  short filename.
> +
> +	  That means that long filenames created with this option
> +	  disabled will not be accessible at all to operating systems
> +	  that do not understand the VFAT extensions.
> +
> +	  Users considering enabling this option should consider the implications
> +	  of any patents that may exist on dual filenames in VFAT.
> +
> +	  If unsure, say N
> +
>  config FAT_DEFAULT_CODEPAGE
>  	int "Default codepage for FAT"
>  	depends on MSDOS_FS || VFAT_FS
> diff --git a/fs/fat/dir.c b/fs/fat/dir.c
> index 38ff75a..cd5d3ec 100644
> --- a/fs/fat/dir.c
> +++ b/fs/fat/dir.c
> @@ -420,14 +420,13 @@ parse_record:
>  			}
>  			i += chl;
>  		}
> -		if (!last_u)
> -			continue;
> -
> -		/* Compare shortname */
> -		bufuname[last_u] = 0x0000;
> -		len = fat_uni_to_x8(sbi, bufuname, bufname, sizeof(bufname));
> -		if (fat_name_match(sbi, name, name_len, bufname, len))
> -			goto found;
> +		if (last_u) {
> +			/* Compare shortname */
> +			bufuname[last_u] = 0x0000;
> +			len = fat_uni_to_x8(sbi, bufuname, bufname, sizeof(bufname));
> +			if (fat_name_match(sbi, name, name_len, bufname, len))
> +				goto found;
> +		}
>  
>  		if (nr_slots) {
>  			void *longname = unicode + FAT_MAX_UNI_CHARS;

This hunk allowing the examination of the long name if there is not a
short name is something else entirely.

You don't describe it in your summary, and don't make the case that
this is the correct behavior.

This should probably be split out into a separate patch if it is the
right thing to do, or drop it if (as it looks) there is no point in
that change.

Eric

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Eric,

 > This hunk allowing the examination of the long name if there is not a
 > short name is something else entirely.
 > 
 > You don't describe it in your summary, and don't make the case that
 > this is the correct behavior.

Sorry for not explaining that part of the patch in my posting.

The reason I initially included it was that during the development of
this patch we were experimenting with variations of the
vfat_build_dummy_83_buffer() function which put a wide varity of
different values in the 11 bytes where a short filename would normally
go.

The most technically attractive varient was to put 11 spaces in
there. That has the advantage that Windows then returns the long name
to the GetShortPathName() API call, which is also what Windows does on
a NTFS filesystem when the NTFS filesystem is configured not to have
8.3 names. So it would have been very nice to have this same behaviour
on VFAT, instead of getting a single space back from
GetShortPathName() as we get with the patch I posted.

The reason we didn't go with the 11 spaces varient is that it causes
WindowsXP to bluescreen. This is just a bug in WinXP, and is not
present in Vista or Windows7.

The 2nd technical problem with 11 spaces is that current Linux kernels
would not see the long names, as the last_u logic in fat_search_long()
will skip files where the the 11 bytes don't have at least one
non-space byte before the first nul byte in either the 8 byte prefix
or the 3 byte extension.

The last_u logic is at odds with the behaviour of other operating
systems for VFAT. On all windows varients that I have tested (and on
MacOSX), the long name is still used no matter what values are in the
11 bytes. You can stick all nul bytes, all spaces, or anything else
you like in there and Windows will still allow you to operate on the
file by the long name and the filesystem will work perfectly.

So that part of the patch is bringing us inline with the behaviour of
other VFAT implementations.

More importantly, it gives us the opportunity in the future to move to
the techically better 11 spaces solution when either Microsoft fixes
their WindowsXP crash bug, or WindowsXP has such a low market share
that nobody cares any more (say like WfWg and maybe Win95 has now).

If we don't put in this change now then we won't be able to move to
the technically better solution later as we wouldn't have given Linux
distros a chance to include support for the 11 spaces in distros. By
putting that change in now, when (if?) we change to 11 spaces later we
will not be breaking recent Linux kernels.

 > This should probably be split out into a separate patch if it is the
 > right thing to do, or drop it if (as it looks) there is no point in
 > that change.

hopefully the above explanation makes it clear why the last_u change
should go together with the vfat_build_slots() change.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Eric W. Biederman]

tridge@samba.org writes:

> Hi Eric,
>
>  > This hunk allowing the examination of the long name if there is not a
>  > short name is something else entirely.
>  > 
>  > You don't describe it in your summary, and don't make the case that
>  > this is the correct behavior.
>
> Sorry for not explaining that part of the patch in my posting.
>
> The reason I initially included it was that during the development of
> this patch we were experimenting with variations of the
> vfat_build_dummy_83_buffer() function which put a wide varity of
> different values in the 11 bytes where a short filename would normally
> go.
>
> The most technically attractive varient was to put 11 spaces in
> there. That has the advantage that Windows then returns the long name
> to the GetShortPathName() API call, which is also what Windows does on
> a NTFS filesystem when the NTFS filesystem is configured not to have
> 8.3 names. So it would have been very nice to have this same behaviour
> on VFAT, instead of getting a single space back from
> GetShortPathName() as we get with the patch I posted.
>
> The reason we didn't go with the 11 spaces varient is that it causes
> WindowsXP to bluescreen. This is just a bug in WinXP, and is not
> present in Vista or Windows7.
>
> The 2nd technical problem with 11 spaces is that current Linux kernels
> would not see the long names, as the last_u logic in fat_search_long()
> will skip files where the the 11 bytes don't have at least one
> non-space byte before the first nul byte in either the 8 byte prefix
> or the 3 byte extension.
>
> The last_u logic is at odds with the behaviour of other operating
> systems for VFAT. On all windows varients that I have tested (and on
> MacOSX), the long name is still used no matter what values are in the
> 11 bytes. You can stick all nul bytes, all spaces, or anything else
> you like in there and Windows will still allow you to operate on the
> file by the long name and the filesystem will work perfectly.
>
> So that part of the patch is bringing us inline with the behaviour of
> other VFAT implementations.
>
> More importantly, it gives us the opportunity in the future to move to
> the techically better 11 spaces solution when either Microsoft fixes
> their WindowsXP crash bug, or WindowsXP has such a low market share
> that nobody cares any more (say like WfWg and maybe Win95 has now).
>
> If we don't put in this change now then we won't be able to move to
> the technically better solution later as we wouldn't have given Linux
> distros a chance to include support for the 11 spaces in distros. By
> putting that change in now, when (if?) we change to 11 spaces later we
> will not be breaking recent Linux kernels.
>
>  > This should probably be split out into a separate patch if it is the
>  > right thing to do, or drop it if (as it looks) there is no point in
>  > that change.
>
> hopefully the above explanation makes it clear why the last_u change
> should go together with the vfat_build_slots() change.

It is part of the same investigation certainly.

Given what you have said our interpretation of vfat has a bug,
and that small change is a candidate for -stable.  If it could
be it's own patch.

That change seems much less controversial and obviously correct
than the other.

Eric

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Eric,

 > Given what you have said our interpretation of vfat has a bug,
 > and that small change is a candidate for -stable.  If it could
 > be it's own patch.

good point. 

Hirofumi-san, would you support putting the last_u change into stable?

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[OGAWA Hirofumi]

tridge@samba.org writes:

>  > Given what you have said our interpretation of vfat has a bug,
>  > and that small change is a candidate for -stable.  If it could
>  > be it's own patch.
>
> good point. 
>
> Hirofumi-san, would you support putting the last_u change into stable?

If you want, I have no problem to do. However, I'm not thinking that
part is a bug.  And -stable rule is also "a real bug that bothers
people", but there is even a no bug reporter which tell actual problem.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Eric W. Biederman]

OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:

> tridge@samba.org writes:
>
>>  > Given what you have said our interpretation of vfat has a bug,
>>  > and that small change is a candidate for -stable.  If it could
>>  > be it's own patch.
>>
>> good point. 
>>
>> Hirofumi-san, would you support putting the last_u change into stable?
>
> If you want, I have no problem to do. However, I'm not thinking that
> part is a bug.  And -stable rule is also "a real bug that bothers
> people", but there is even a no bug reporter which tell actual problem.


Tridge.  Is there any reason to believe that Microsoft will continue
to treat Longfilenames without short filenames as valid in vfat?

If this turns into a contest of who can do the silliest things in
their vfat code microsoft could easily introduce a stricter directory
parser and cause all kinds of grief.

It wouldn't even surprise me if you haven't seen such shenanigans
while working on samba.

Eric

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[James Bottomley]

On Sun, 2009-06-28 at 12:51 -0700, Eric W. Biederman wrote:
> OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:
> 
> > tridge@samba.org writes:
> >
> >>  > Given what you have said our interpretation of vfat has a bug,
> >>  > and that small change is a candidate for -stable.  If it could
> >>  > be it's own patch.
> >>
> >> good point. 
> >>
> >> Hirofumi-san, would you support putting the last_u change into stable?
> >
> > If you want, I have no problem to do. However, I'm not thinking that
> > part is a bug.  And -stable rule is also "a real bug that bothers
> > people", but there is even a no bug reporter which tell actual problem.
> 
> 
> Tridge.  Is there any reason to believe that Microsoft will continue
> to treat Longfilenames without short filenames as valid in vfat?
> 
> If this turns into a contest of who can do the silliest things in
> their vfat code microsoft could easily introduce a stricter directory
> parser and cause all kinds of grief.
> 
> It wouldn't even surprise me if you haven't seen such shenanigans
> while working on samba.

If you own the platform, like Microsoft does, there are many things you
*could* do to make life difficult for others (like shipping a slightly
incompatible version of java, for instance ...).  However, having had
one or two consumer and regulatory backlashes from shipping updates
primarily designed to hobble what you think of as a competitor, you tend
to be much more wary about doing it so openly ... particularly when
you're trying to convince a wary customer base that you're the champion
of interoperability nowadays.

The bottom line is that we need to consider the current patch on its
merits for evading the vfat patent.  Speculating about what Microsoft
might or might not do to retaliate doesn't really help with this.

James

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Eric W. Biederman]

James Bottomley <James.Bottomley@HansenPartnership.com> writes:

> On Sun, 2009-06-28 at 12:51 -0700, Eric W. Biederman wrote:
>> OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:
>> 
>> > tridge@samba.org writes:
>> >
>> >>  > Given what you have said our interpretation of vfat has a bug,
>> >>  > and that small change is a candidate for -stable.  If it could
>> >>  > be it's own patch.
>> >>
>> >> good point. 
>> >>
>> >> Hirofumi-san, would you support putting the last_u change into stable?
>> >
>> > If you want, I have no problem to do. However, I'm not thinking that
>> > part is a bug.  And -stable rule is also "a real bug that bothers
>> > people", but there is even a no bug reporter which tell actual problem.
>> 
>> 
>> Tridge.  Is there any reason to believe that Microsoft will continue
>> to treat Longfilenames without short filenames as valid in vfat?
>> 
>> If this turns into a contest of who can do the silliest things in
>> their vfat code microsoft could easily introduce a stricter directory
>> parser and cause all kinds of grief.
>> 
>> It wouldn't even surprise me if you haven't seen such shenanigans
>> while working on samba.
>
> If you own the platform, like Microsoft does, there are many things you
> *could* do to make life difficult for others (like shipping a slightly
> incompatible version of java, for instance ...).  However, having had
> one or two consumer and regulatory backlashes from shipping updates
> primarily designed to hobble what you think of as a competitor, you tend
> to be much more wary about doing it so openly ... particularly when
> you're trying to convince a wary customer base that you're the champion
> of interoperability nowadays.
>
> The bottom line is that we need to consider the current patch on its
> merits for evading the vfat patent.  Speculating about what Microsoft
> might or might not do to retaliate doesn't really help with this.

This is a technical question.  Are we really in spec with the patch?

If we are not in spec. If we break things like checkdisk.exe.   Microsoft
can legitimately say we broke things and take technical measures to avoid
broken filesystems.  We have dome similar things to close source binary
modules.

My impression is that this patch puts us on the hairy edge.  If no
one is generating files without a short filename now.  That may cause
problems.

All else being equal this is a technically problematic patch as it
reduces the chances of interoperability.  I am just trying to gage
what those chances are.

Eric

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[James Bottomley]

On Sun, 2009-06-28 at 13:45 -0700, Eric W. Biederman wrote:
> James Bottomley <James.Bottomley@HansenPartnership.com> writes:
> 
> > On Sun, 2009-06-28 at 12:51 -0700, Eric W. Biederman wrote:
> >> OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:
> >> 
> >> > tridge@samba.org writes:
> >> >
> >> >>  > Given what you have said our interpretation of vfat has a bug,
> >> >>  > and that small change is a candidate for -stable.  If it could
> >> >>  > be it's own patch.
> >> >>
> >> >> good point. 
> >> >>
> >> >> Hirofumi-san, would you support putting the last_u change into stable?
> >> >
> >> > If you want, I have no problem to do. However, I'm not thinking that
> >> > part is a bug.  And -stable rule is also "a real bug that bothers
> >> > people", but there is even a no bug reporter which tell actual problem.
> >> 
> >> 
> >> Tridge.  Is there any reason to believe that Microsoft will continue
> >> to treat Longfilenames without short filenames as valid in vfat?
> >> 
> >> If this turns into a contest of who can do the silliest things in
> >> their vfat code microsoft could easily introduce a stricter directory
> >> parser and cause all kinds of grief.
> >> 
> >> It wouldn't even surprise me if you haven't seen such shenanigans
> >> while working on samba.
> >
> > If you own the platform, like Microsoft does, there are many things you
> > *could* do to make life difficult for others (like shipping a slightly
> > incompatible version of java, for instance ...).  However, having had
> > one or two consumer and regulatory backlashes from shipping updates
> > primarily designed to hobble what you think of as a competitor, you tend
> > to be much more wary about doing it so openly ... particularly when
> > you're trying to convince a wary customer base that you're the champion
> > of interoperability nowadays.
> >
> > The bottom line is that we need to consider the current patch on its
> > merits for evading the vfat patent.  Speculating about what Microsoft
> > might or might not do to retaliate doesn't really help with this.
> 
> This is a technical question.  Are we really in spec with the patch?

Yes, the spec is here:

http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx

(sorry, nasty licence agreement required to read it).

It says that long filenames *may* be created without short ones (but
that if you do this you'll have backwards compatibility problems on FAT
rather than FAT32 systems, which has been explicitly noted: after this
patch you can't read created files on 8.3 FAT).

> If we are not in spec. If we break things like checkdisk.exe.   Microsoft
> can legitimately say we broke things and take technical measures to avoid
> broken filesystems.  We have dome similar things to close source binary
> modules.

The question about spec isn't really that relevant with MS ... as you
can see from the Windows XP problems, they don't follow their own
spec ... the real question is has this been tested against existing
Windows versions; the answer is Yes (see FAQ).

> My impression is that this patch puts us on the hairy edge.  If no
> one is generating files without a short filename now.  That may cause
> problems.

Only for backwards compatibility ... and IBM has explicitly checked that
according to the FAQ.  They actually found Windows XP to be out of spec
(crashed with empty short filename) which is why the patch places an
invisible string in there now.

> All else being equal this is a technically problematic patch as it
> reduces the chances of interoperability.  I am just trying to gage
> what those chances are.

So the patch has been tested with Vista, Windows 7 and Windows XP ...
I'm sure IBM would be willing to do further interop testing if you
request.

James

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Eric,

 > Tridge.  Is there any reason to believe that Microsoft will continue
 > to treat Longfilenames without short filenames as valid in vfat?

I'd be quite surprised if they deliberately changed their VFAT code to
break Linux with this patch. I'd say it is more likely that once Linux
kernels with this change are in widespread use that Microsoft will
start to test any changes in their VFAT filesystem to make sure it
works with Linux with this patch.

 > It wouldn't even surprise me if you haven't seen such shenanigans
 > while working on samba.

In recent times Microsoft has been testing new OS releases against
Samba, and has been actively working with the Samba Team to try to
prevent breakages with new releases. That doesn't mean the scenario
you describe is impossible, it just means it isn't as likely as
perhaps it once was.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Hirofumi-san,

 > If you want, I have no problem to do. However, I'm not thinking that
 > part is a bug.  And -stable rule is also "a real bug that bothers
 > people", but there is even a no bug reporter which tell actual problem.

ok, then what about pushing the whole patch into -stable?

The 'bug' in this case is like a security hole, with Microsoft
demonstrating an active exploit. This falls outside the normal range
of bug fixes for -stable, but I think you'd have to agree that the
whole situation is unusual.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Greg KH]

On Mon, Jun 29, 2009 at 11:30:39AM +1000, tridge@samba.org wrote:
> Hi Hirofumi-san,
> 
>  > If you want, I have no problem to do. However, I'm not thinking that
>  > part is a bug.  And -stable rule is also "a real bug that bothers
>  > people", but there is even a no bug reporter which tell actual problem.
> 
> ok, then what about pushing the whole patch into -stable?
> 
> The 'bug' in this case is like a security hole, with Microsoft
> demonstrating an active exploit. This falls outside the normal range
> of bug fixes for -stable, but I think you'd have to agree that the
> whole situation is unusual.

I have no objection to take this patch for the -stable releases when it
goes into Linus's tree.  Just forward the git commit ids to
stable@kernel.org and I can take care of it.

thanks,

greg k-h

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

 > I have no objection to take this patch for the -stable releases when it
 > goes into Linus's tree.  Just forward the git commit ids to
 > stable@kernel.org and I can take care of it.

ok, thanks Greg.

Hirofumi-san tells me we've missed the merge window for 2.6.31, so it
seems that this will go first into the fatfs tree, then go into
Linus's tree via -next for 2.6.32. So hopefully the patch will get
some more testing (perhaps in -mm?) before it needs to be considered
for -stable.

Meanwhile some of the distros and other vendors might decide to apply
it sooner depending on their level of legal concern.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Greg KH]

On Tue, Jun 30, 2009 at 08:42:42AM +1000, tridge@samba.org wrote:
>  > I have no objection to take this patch for the -stable releases when it
>  > goes into Linus's tree.  Just forward the git commit ids to
>  > stable@kernel.org and I can take care of it.
> 
> ok, thanks Greg.
> 
> Hirofumi-san tells me we've missed the merge window for 2.6.31, so it
> seems that this will go first into the fatfs tree, then go into
> Linus's tree via -next for 2.6.32. So hopefully the patch will get
> some more testing (perhaps in -mm?) before it needs to be considered
> for -stable.

That sounds reasonable.  Just add a:
	Cc: stable <stable@kernel.org>
in the "signed-off-by:" area of the patch, and when it goes into Linus's
tree, the -stable team will be automatically notified of it and we will
pick it up.

It needs to be in Linus's tree before we can add it to the -stable
releases.

thanks,

greg k-h

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[OGAWA Hirofumi]

tridge@samba.org writes:

>  > I have no objection to take this patch for the -stable releases when it
>  > goes into Linus's tree.  Just forward the git commit ids to
>  > stable@kernel.org and I can take care of it.
>
> ok, thanks Greg.
>
> Hirofumi-san tells me we've missed the merge window for 2.6.31, so it
> seems that this will go first into the fatfs tree, then go into
> Linus's tree via -next for 2.6.32. So hopefully the patch will get
> some more testing (perhaps in -mm?) before it needs to be considered
> for -stable.

Yes. And I will not drop this patch even if it's not perfect on test. I
can understand it would not be the option for some users. The default
may be arguable though, like I said at first.

My choice is to provide the option of those to users.

> Meanwhile some of the distros and other vendors might decide to apply
> it sooner depending on their level of legal concern.

Yes, it's good.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hirofumi-san writes:
 > Yes. And I will not drop this patch even if it's not perfect on test. I
 > can understand it would not be the option for some users.

ok, good

 > The default may be arguable though, like I said at first.

I suggest we leave the default as "no dual names" for now to let it
get some more testing, then if any problems come up then we can look
at the choice of default again. If no problems come up we can keep the
default as it is in the patch now.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[OGAWA Hirofumi]

tridge@samba.org writes:

>  > The default may be arguable though, like I said at first.
>
> I suggest we leave the default as "no dual names" for now to let it
> get some more testing, then if any problems come up then we can look
> at the choice of default again. If no problems come up we can keep the
> default as it is in the patch now.

Ok, I've applied the patch + Cc: to fatfs-2.6.git.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

> When VFAT_FS_DUALNAMES is disabled we avoid the creation of 8.3 short
> filenames for files on VFAT filesystems that require a long name. The
> patch uses a pattern of 11 bytes in the directory entry which contains
> invalid characters such that it cannot be considered to be a valid short
> filename.
> 
> Signed-off-by: Andrew Tridgell <tridge@samba.org>
> Acked-by: Dave Kleikamp <shaggy@linux.vnet.ibm.com>
> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  fs/fat/Kconfig      |   20 +++++++++++++++++
>  fs/fat/dir.c        |   15 ++++++-------
>  fs/fat/namei_vfat.c |   59 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 86 insertions(+), 8 deletions(-)
> 
> diff --git a/fs/fat/Kconfig b/fs/fat/Kconfig
> index 182f9ff..907a5de 100644
> --- a/fs/fat/Kconfig
> +++ b/fs/fat/Kconfig
> @@ -74,6 +74,26 @@ config VFAT_FS
>  	  To compile this as a module, choose M here: the module will be called
>  	  vfat.
>  
> +config VFAT_FS_DUALNAMES
> +	bool "VFAT dual names support"
> +	depends on VFAT_FS


Defaults should be back-compatible.

> +
> +	  Users considering enabling this option should consider the implications
> +	  of any patents that may exist on dual filenames in VFAT.
> +
> +	  If unsure, say N

Say Y.

Users considering disabling this should understand that filesystem
they write to will not be valid vfat filesystems, and may trigger bugs
in  some devices.

> +#ifndef CONFIG_VFAT_FS_DUALNAMES
> +/*
> + * build a 11 byte 8.3 buffer which is not a short filename. We want 11
> + * bytes which:
> + *    - will be seen as a constant string to all APIs on Linux and Windows
> + *    - cannot be matched with wildcard patterns
> + *    - cannot be used to access the file
> + *    - has a low probability of collision within a directory
> + *    - has an invalid 3 byte extension
> + *    - contains at least one non-space and non-nul byte
> + */

What happens on collision? With 60000 entries in directory, there will
be 50% chance of collision. BAD.

Why not use something like position in directory instead of random
number?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> What happens on collision? With 60000 entries in directory, there will
> be 50% chance of collision. BAD.

Far more surely - its a birthday paradox.

> Why not use something like position in directory instead of random
> number?

Agreed 100%. I'm also not sure it should be called "vfat" when operating
in this mode as it's not vfat any more - it needs a new name.

Alan

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Alan,

 > > What happens on collision? With 60000 entries in directory, there will
 > > be 50% chance of collision. BAD.
 > 
 > Far more surely - its a birthday paradox.

If you want to do it accurately, the maximum number of long filenames
in a VFAT directory is actually 32767. (it isn't 65536, as each long
filename consumes at least two 8.3 entries, plus you lose the . and
.. entries).

With the patch I've posted there are 30 bits of randomness in each
entry. You could do an accurate binomial expansion to get the exact
probability, but a very good approximation using exponentiation comes
out as a 39.3% chance of a single duplicate appearing in a directory
that is fully populated.

As I mentioned to Pavel, this isn't the whole story though. To cause
the bluescreen the duplicate entries need to be accessed by WindowsXP
in quick succession in a particular pattern. This lowers the
probability a lot. Exactly how much is hard to estimate, but
experiments I've done with deliberately higher probabilities (ie. less
bits of randomness) show that the probability of the bluescreen is
_very_ low.

 > Agreed 100%. I'm also not sure it should be called "vfat" when operating
 > in this mode as it's not vfat any more - it needs a new name.

If the code differed significantly between the two implementations I'd
probably agree, but as the two are extremely close I think maintaining
a separate filesystem isn't worth it.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

>  > Agreed 100%. I'm also not sure it should be called "vfat" when operating
>  > in this mode as it's not vfat any more - it needs a new name.
> 
> If the code differed significantly between the two implementations I'd
> probably agree, but as the two are extremely close I think maintaining
> a separate filesystem isn't worth it.

It needs a different name to the user. If the new fs isn't vfat (which it
isn't) and doubly so if it can crash Windows XP at random on very rare
occasions then users need to know its different.

Imagine someone sticks a Linux written disk into a mission critical
windows server - I think they have a right to know and not accidentally
wander into a situation where they bring that box down ?

mount -o vfat should fail for this non-vfat.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Rusty Russell]

On Tue, 30 Jun 2009 04:01:03 pm Pavel Machek wrote:
> > +config VFAT_FS_DUALNAMES
> > +	bool "VFAT dual names support"
> > +	depends on VFAT_FS
>
> Defaults should be back-compatible.

Hi Pavel,

I disagree with this: given there's been testing with no known issues, it's 
not a back compat question.

I'd actually prefer to see the code ripped out and no config option available; 
it would the clearest avoidance case possible.

I really don't want us to have to do this more than once :(
Rusty.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> I'd actually prefer to see the code ripped out and no config option available; 
> it would the clearest avoidance case possible.

What about the free world - why should we suffer because Americans have
a broken patent system ? That just leads to a Farenheit 451 model where
the kernel sources end up containing no code, text or image that can
offend, harm or be found wanting in any possible legal jurisdiction.

If we put in VFAT american fixes presumably we need to put in monitoring
features required by dubious governments ?

If you want to rip stuff out of your copy feel free. I am quite sure many
US based vendors will do that (because they do so already with things
like video codecs rather than just disabling the build option).

Also please stop calling it VFAT. With the changes made it isn't VFAT and
it's misleading to an end user to advertise it as vfat and users
shouldn't accidentally be able to specify -o vfat and get non-vfat. Thats
asking for incompatibility, data loss and unpleasant unwarned of suprises.

Its "linfat" or something which when you've fixed the bugs Pavel has
pointed out might be semi-compatible with other products (most *FAT using
products don't use Microsofts implementation). Testing it versus Windows
and saying it works is not really adequate. Thats what ACPI and BIOS
people do that *we* moan about all the time.

Alan

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Theodore Tso]

On Wed, Jul 01, 2009 at 12:25:58PM +0100, Alan Cox wrote:
> 
> Also please stop calling it VFAT. With the changes made it isn't VFAT and
> it's misleading to an end user to advertise it as vfat and users
> shouldn't accidentally be able to specify -o vfat and get non-vfat. Thats
> asking for incompatibility, data loss and unpleasant unwarned of suprises.

There really was no such thing as "vfat" anyway.  VFAT in the Windows
world names a specific implementation (the "V" stands for virtual)
back in the days when Windows 3.1 was primarily a 16-bit OS, and there
were two instances of the "fat" filesystem code; one for the 16 bit
implementation, and one for 32-bit applications.  As it happened the
32-bit implementation under Windows 3.1 happened to contain support
for the Long File Names extension, and due to this confusion of
implementation vs. specification, in Linux is misnamed something as
"vfat" when most of the rest of the world simply just calls it "FAT".

Arguably what we should have done is kept it as a single filesystem,
with a mount options "lfn" and "nolfn", but that's water under the
bridge now.

> (most *FAT using products don't use Microsofts
> implementation). Testing it versus Windows and saying it works is
> not really adequate. Thats what ACPI and BIOS people do that *we*
> moan about all the time.

This *is* a legitimate concern.  I'll note that many of the modern-day
products actually use Linux (i.e., the Sony eReader, the Kindle,
etc. --- and so as much as you might kvetch about the rest of the
"Free World" not having the same broken patent system as the US, the
economic realities of these products losing access to the US market
would be quite devastating to most of these products' manufacturers
even if some of them are developed and manufactured outside of the
US), and so doing compatibility testing with Linux is relatively easy.

The other big user I can think of are digital cameras, but (a)
normally most users read from them and then delete the pictures, and
rarely write to media meant for a digital camera, and (b) the DCIM
standard for digital cameras explicitly only supports 8.3 filenames
and so digital camera manufacturers explicitly don't need to deal with
Long File Names at all.  (Hmm.... I wonder why....)  

This suggests that some userspace mechanism for detecting media cards
used for cameras and explicitly mounting them with FAT might be useful
--- since the camera isn't going to be able to recognize LFN's anyway.
It also suggests that some testing to make sure there aren't bugs in
various non-Linux FAT using devices would probably be useful, and that
a config option might be a good way to enable people to do such
testing and fall back if it turns out to be problems with such
devices.

Ultimately, though, requiring that every single possible device be
tested is probably not reasonable, so the best way to do this testing
is the way do most of our testing; we do basic due diligence, but then
we merge it into mainline and let our huge user community try it out.
If there are regressions we can work through those issues if and when
they arise.

						- Ted

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> > shouldn't accidentally be able to specify -o vfat and get non-vfat. Thats
> > asking for incompatibility, data loss and unpleasant unwarned of suprises.
> 
> There really was no such thing as "vfat" anyway.  VFAT in the Windows

In the eyes of the end user there is such a thing as vfat. This is about
expectations not technical issues.
 
> Arguably what we should have done is kept it as a single filesystem,
> with a mount options "lfn" and "nolfn", but that's water under the
> bridge now.

Well we didn't so now we need to add "lfat" or similar for our fat style.
Doesn't need new code just making sure that USSA_COMPLIANCE_MODE=y
causes mount -o lfat to work and without it both lfat and vfat work.

> The other big user I can think of are digital cameras, but (a)
> normally most users read from them and then delete the pictures, and
> rarely write to media meant for a digital camera, and (b) the DCIM

Except when they hit save instead of "save as" and they get a long file
name and invisible loss of space on the camera.

> standard for digital cameras explicitly only supports 8.3 filenames
> and so digital camera manufacturers explicitly don't need to deal with
> Long File Names at all.  (Hmm.... I wonder why....)  

Can't think - but HAL should clearly mount those 8.3 to avoid the
problem. It seems to use the dcim to find them.

> This suggests that some userspace mechanism for detecting media cards
> used for cameras and explicitly mounting them with FAT might be useful

HAL is very good at that already.

> Ultimately, though, requiring that every single possible device be
> tested is probably not reasonable, so the best way to do this testing
> is the way do most of our testing; we do basic due diligence, but then
> we merge it into mainline and let our huge user community try it out.
> If there are regressions we can work through those issues if and when
> they arise.

>From the funnies we've had in the past with FAT my gut impression is
there are only a few implementations out there. Psion seems to have their
own but most of the rest behave remarkably similarly which makes me
suspect they all licensed a tiny number of implementations (DRDOS one
perhaps ?). If we can keep most of those devices mounted 8.3 we nicely
sidestep the issue anyway.

Alan

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Alan,

 > > There really was no such thing as "vfat" anyway.  VFAT in the Windows
 > 
 > In the eyes of the end user there is such a thing as vfat. This is about
 > expectations not technical issues.

Do you have an example of an end user device that would be affected by
this change, and which is never distributed in any of the countries
where a varient of this patent exists?

 > From the funnies we've had in the past with FAT my gut impression is
 > there are only a few implementations out there. Psion seems to have their
 > own but most of the rest behave remarkably similarly which makes me
 > suspect they all licensed a tiny number of implementations (DRDOS one
 > perhaps ?).

Do you happen to have a Psion and/or DRDOS based device in your
collection of toys so you can test with the proposed patch and see if
there are any problems?

Apart from Windows varients and MacOSX I have also tested on the
following mobile phones (via a microSD card):

  Samsumg SGH-E250
  Nokia 6120

They both worked fine with long filename images saved on a Linux
system with this patch. I don't know what OSes these phones are based
on.

Maybe you could test on the various devices with removable media you
have and see how it goes?

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

On Thu, 2 Jul 2009 11:42:39 +1000
tridge@samba.org wrote:

> Hi Alan,
> 
>  > > There really was no such thing as "vfat" anyway.  VFAT in the Windows
>  > 
>  > In the eyes of the end user there is such a thing as vfat. This is about
>  > expectations not technical issues.
> 
> Do you have an example of an end user device that would be affected by
> this change, and which is never distributed in any of the countries
> where a varient of this patent exists?

You gave one: Microsoft Windows XP. Risk of very rare system crashes.

Thats enough in a high reliability environment to require the file system
in question is banned from use site wide at may locations. Can't have a
flash card bringing down a windows dependant US destroyer can they.

Getting "not vfat" by suprise is thus really really bad

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Alan,

 > You gave one: Microsoft Windows XP. Risk of very rare system crashes.
 > 
 > Thats enough in a high reliability environment to require the file system
 > in question is banned from use site wide at may locations. Can't have a
 > flash card bringing down a windows dependant US destroyer can they.

I think it could indeed be very wise not to insert FAT removable media
into 'high reliability' WindowsXP machines, but not because of the
patch I've proposed. A prudent IT manager might take a look at how
many fastfat.sys crashes are being reported on WindowsXP and realise
that viewing ones holiday photos on your heart bypass machine isn't a
great idea :-)

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

> > Ultimately, though, requiring that every single possible device be
> > tested is probably not reasonable, so the best way to do this testing
> > is the way do most of our testing; we do basic due diligence, but then
> > we merge it into mainline and let our huge user community try it out.
> > If there are regressions we can work through those issues if and when
> > they arise.
> 
> From the funnies we've had in the past with FAT my gut impression is
> there are only a few implementations out there. Psion seems to have their
> own but most of the rest behave remarkably similarly which makes me
> suspect they all licensed a tiny number of implementations (DRDOS one
> perhaps ?). If we can keep most of those devices mounted 8.3 we nicely
> sidestep the issue anyway.

I'm pretty sure there's more. There's stuff such as 'make your own mp3
player with pic and few more circuits'. I'd actually expect many mp3
players to be affected by this; many are really cheap & nasty.

								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Jamie Lokier]

Alan Cox wrote:
> From the funnies we've had in the past with FAT my gut impression is
> there are only a few implementations out there. Psion seems to have their
> own but most of the rest behave remarkably similarly which makes me
> suspect they all licensed a tiny number of implementations (DRDOS one
> perhaps ?). If we can keep most of those devices mounted 8.3 we nicely
> sidestep the issue anyway.

>From recently witnessing someone write another FAT implementation, for
an 8-bit AVR microcontroller reading SD cards, based in part on
someone elses public domain FAT implementation, fixing significant
bugs in that (including directory parsing), and then seeing it
translated from C to hand-coded AVR assembly language to save space,
with me helping teach C to the person doing it, for a commercial
product which is to be released in the next couple of months, and will
be updated with SD cards written by Linux and Windows...

>From witnessing that, for one obscure product among millions, I'd
guess there are more FAT implementations than a "few".  Perhaps there
aren't many VFAT implementations, but my gut says there are quite a
lot of independently written 8.3-only FAT ones.

-- Jamie

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Stefan Richter]

Theodore Tso wrote:
> I'll note that many of the modern-day
> products actually use Linux (i.e., the Sony eReader, the Kindle,
> etc.

Many MP3/Video players surely do not run an embedded Linux.

Besides, it doesn't matter what "many" devices run if the talk is about 
defaults, or even about ripping the working, proven code out of the 
mainline.  (Well, that suggestion was absurd.)

> The other big user I can think of are digital cameras, but (a)
> normally most users read from them and then delete the pictures,

I use my SD cards not only in the camera but also as generic mobile 
storage with laptops and media player.

There is no way for software to detect that a medium is going to be used 
with a 8.3-only device exclusively.
-- 
Stefan Richter
-=====-==--= -=== ----=
http://arcgraph.de/sr/

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Stefan Richter]

Theodore Tso wrote:
> I'll note that many of the modern-day
> products actually use Linux (i.e., the Sony eReader, the Kindle,
> etc.

Many MP3/Video players surely do not run an embedded Linux.

Besides, it doesn't matter what "many" devices run if the talk is about 
defaults, or even about ripping the working, proven code out of the 
mainline.  (Well, that suggestion was absurd.)

> The other big user I can think of are digital cameras, but (a)
> normally most users read from them and then delete the pictures,

I use my SD cards not only in the camera but also as generic mobile 
storage with laptops and media player.

There is no way for software to detect that a medium is going to be used 
with a 8.3-only device exclusively.
-- 
Stefan Richter
-=====-==--= -=== ----=
http://arcgraph.de/sr/

Re: CONFIG_VFAT_FS_DUALNAMES regression

[Jan Engelhardt]

On Wednesday 2009-07-01 16:05, Theodore Tso wrote:
>On Wed, Jul 01, 2009 at 12:25:58PM +0100, Alan Cox wrote:
>
>> (most *FAT using products don't use Microsofts
>> implementation). Testing it versus Windows and saying it works is
>> not really adequate. Thats what ACPI and BIOS people do that *we*
>> moan about all the time.
>
>[...]
>The other big user I can think of are digital cameras, but (a)
>normally most users read from them and then delete the pictures, and
>rarely write to media meant for a digital camera, and (b) the DCIM
>standard for digital cameras explicitly only supports 8.3 filenames
>and so digital camera manufacturers explicitly don't need to deal with
>Long File Names at all.  (Hmm.... I wonder why....)
>[...]
>Ultimately, though, requiring that every single possible device be
>tested is probably not reasonable, so the best way to do this testing
>is the way do most of our testing; we do basic due diligence, but then
>we merge it into mainline and let our huge user community try it out.


Yes, here is your precedent case. The DUALNAMES patch breaks the
operation on my Fujifilm Finepix A210 digital camera.

Here, dscf4159.jpg is a preexisting file created by the camera itself 
(and subsequently, Finepix's FAT code) as a result of taking a picture. 
Then I just copied that file with two different kernels in succession.

 * 2.6.31-rc1 + dualnames=n
   # mount /dev/sdc1 /mnt
   # cd /mnt/dcim/100_fuji/; cp dscf4159.jpg dscf3000.jpg
   # umount /mnt

 * 2.6.29.5 w/o patch
   # same procedure
   cp dscf4159.jpg dscf3001.jpg

End result is that picture ID 3000 is not selectable in the camera's
built-in menu (OSD), as if the file did not exist. 3001 was shown,
however.

Re: CONFIG_VFAT_FS_DUALNAMES regression

[Jan Engelhardt]

On Wednesday 2009-07-01 16:05, Theodore Tso wrote:
>On Wed, Jul 01, 2009 at 12:25:58PM +0100, Alan Cox wrote:
>
>> (most *FAT using products don't use Microsofts
>> implementation). Testing it versus Windows and saying it works is
>> not really adequate. Thats what ACPI and BIOS people do that *we*
>> moan about all the time.
>
>[...]
>The other big user I can think of are digital cameras, but (a)
>normally most users read from them and then delete the pictures, and
>rarely write to media meant for a digital camera, and (b) the DCIM
>standard for digital cameras explicitly only supports 8.3 filenames
>and so digital camera manufacturers explicitly don't need to deal with
>Long File Names at all.  (Hmm.... I wonder why....)
>[...]
>Ultimately, though, requiring that every single possible device be
>tested is probably not reasonable, so the best way to do this testing
>is the way do most of our testing; we do basic due diligence, but then
>we merge it into mainline and let our huge user community try it out.


Yes, here is your precedent case. The DUALNAMES patch breaks the
operation on my Fujifilm Finepix A210 digital camera.

Here, dscf4159.jpg is a preexisting file created by the camera itself 
(and subsequently, Finepix's FAT code) as a result of taking a picture. 
Then I just copied that file with two different kernels in succession.

 * 2.6.31-rc1 + dualnames=n
   # mount /dev/sdc1 /mnt
   # cd /mnt/dcim/100_fuji/; cp dscf4159.jpg dscf3000.jpg
   # umount /mnt

 * 2.6.29.5 w/o patch
   # same procedure
   cp dscf4159.jpg dscf3001.jpg

End result is that picture ID 3000 is not selectable in the camera's
built-in menu (OSD), as if the file did not exist. 3001 was shown,
however.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regression

[tridge]

Hi Jan,

 > Yes, here is your precedent case. The DUALNAMES patch breaks the
 > operation on my Fujifilm Finepix A210 digital camera.

interesting, but I think you may have been trapped by differences in
the VFAT mounting options between the two examples.

Would you mind retesting again, but looking carefully at the VFAT
mount options. I don't think what you've described can happen without
differing VFAT options between your two examples.

I'll try and expain:

 > Here, dscf4159.jpg is a preexisting file created by the camera itself 
 > (and subsequently, Finepix's FAT code) as a result of taking a picture. 
 > Then I just copied that file with two different kernels in succession.

ok, dscf4159.jpg may or may not be a 8.3 name depending on the
"shortname=" mount option. As this is a digital camera, I suspect that
it was created as an 8.3 name, although you'd have to confirm that by
inspecting the disk image. If you made a copy of the image available
to me somewhere (please don't email it unless it compresses to a small
size!) then I can tell you what the camera did in this case.

 >  * 2.6.31-rc1 + dualnames=n
 >    # mount /dev/sdc1 /mnt
 >    # cd /mnt/dcim/100_fuji/; cp dscf4159.jpg dscf3000.jpg
 >    # umount /mnt

Now we would need to know what shortname= option was used here. If the
shortname= option was such that the lowercase name dscf3000.jpg was
considered as 8.3, then the patch I have posted would have had zero
effect on what was stored on disk. If it was not considered a short
name (because you have the shortname= option set to not allow
lowercase short names) then it would have been created as a long
name. As you said the file became invisible then I suspect that for
this test you had the shortname= option set to treat lowercase names
as not being allowed in 8.3.

 >  * 2.6.29.5 w/o patch
 >    # same procedure
 >    cp dscf4159.jpg dscf3001.jpg
 > 
 > End result is that picture ID 3000 is not selectable in the camera's
 > built-in menu (OSD), as if the file did not exist. 3001 was shown,
 > however.

ok, if 3001 was shown, then in this case you must have had different
VFAT mount options. I don't know if this is because the different base
kernel has different defaults, or if perhaps you used a different
system that had different mounting options.

If you had used the same mount options as the previous test, then the
3001 name could not have shown up I think. It would have shown up as
something like DSCF3~01.JPG. 

So I think all you've found is two things we already knew:

 1) VFAT mount options make for an easy trap to fall into :-)

 2) as the config text in the patch explains:

        "That means that long filenames created with this option
	 disabled will not be accessible at all to operating systems
	 that do not understand the VFAT extensions."

If you mount with the right shortname= options then I think you'll
find that your above test will work (for some value of work!)

But please do keep testing. We want to eliminate the possibility that
you have found a real example of where the patch causes a problem with
some embedded systems.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regression

[Jan Engelhardt]

On Friday 2009-07-03 01:37, tridge@samba.org wrote:
>
> > Yes, here is your precedent case. The DUALNAMES patch breaks the
> > operation on my Fujifilm Finepix A210 digital camera.
>
>interesting, but I think you may have been trapped by differences in
>the VFAT mounting options between the two examples.

No special options — all defaults to whatever the Linux kernel and
util-linux 2.14.1 gave me. Whereby util-linux should not make a
difference because it just passes on the mount request for there is
no special vfat helper.

Given my second report about the audio player (just sent out), I even
compared dualnames=n with =y on the same kernel — which would rule
out changed defaults between 2.6.29 and 2.6.31-rc.

/proc/mounts contains, on .31-rc:

/dev/sdc1 /mnt vfat rw,relatime,fmask=0022,dmask=0022,codepage=437
iocharset=utf8,errors=remont-ro 0 0

> > Here, dscf4159.jpg is a preexisting file created by the camera itself 
> > (and subsequently, Finepix's FAT code) as a result of taking a picture. 
> > Then I just copied that file with two different kernels in succession.
>
>ok, dscf4159.jpg may or may not be a 8.3 name depending on the
>"shortname=" mount option. As this is a digital camera, I suspect that
>it was created as an 8.3 name, although you'd have to confirm that by
>inspecting the disk image. If you made a copy of the image available
>to me somewhere (please don't email it unless it compresses to a small
>size!) then I can tell you what the camera did in this case.

http://jengelh.medozas.de/files/finepix.img.bz2
(Includes an unreadable dscf3000 entry.)

Then, I just tried everything to get an overview:

 (All with DUALNAMES=n)
 # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=lower
 # cp /mnt/dcim/100_fuji/dscf{4160,3004}.jpg
 # umount /mnt
 # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=win95
 # cp /mnt/dcim/100_fuji/dscf{4160,3005}.jpg
 # umount /mnt
 # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=winnt
 # cp /mnt/dcim/100_fuji/dscf{4160,3006}.jpg
 # umount /mnt
 # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=mixed
 # cp /mnt/dcim/100_fuji/dscf{4160,3007}.jpg
 # umount /mnt

Result? The camera only displays 3006.

The dualnames patch's filling filenames with random illegal
chars does seems to have a really foul side-effect after all.

Re: CONFIG_VFAT_FS_DUALNAMES regression

[Jan Engelhardt]

On Friday 2009-07-03 01:37, tridge@samba.org wrote:
>
> > Yes, here is your precedent case. The DUALNAMES patch breaks the
> > operation on my Fujifilm Finepix A210 digital camera.
>
>interesting, but I think you may have been trapped by differences in
>the VFAT mounting options between the two examples.

No special options — all defaults to whatever the Linux kernel and
util-linux 2.14.1 gave me. Whereby util-linux should not make a
difference because it just passes on the mount request for there is
no special vfat helper.

Given my second report about the audio player (just sent out), I even
compared dualnames=n with =y on the same kernel — which would rule
out changed defaults between 2.6.29 and 2.6.31-rc.

/proc/mounts contains, on .31-rc:

/dev/sdc1 /mnt vfat rw,relatime,fmask=0022,dmask=0022,codepage=437
iocharset=utf8,errors=remont-ro 0 0

> > Here, dscf4159.jpg is a preexisting file created by the camera itself 
> > (and subsequently, Finepix's FAT code) as a result of taking a picture. 
> > Then I just copied that file with two different kernels in succession.
>
>ok, dscf4159.jpg may or may not be a 8.3 name depending on the
>"shortname=" mount option. As this is a digital camera, I suspect that
>it was created as an 8.3 name, although you'd have to confirm that by
>inspecting the disk image. If you made a copy of the image available
>to me somewhere (please don't email it unless it compresses to a small
>size!) then I can tell you what the camera did in this case.

http://jengelh.medozas.de/files/finepix.img.bz2
(Includes an unreadable dscf3000 entry.)

Then, I just tried everything to get an overview:

 (All with DUALNAMES=n)
 # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=lower
 # cp /mnt/dcim/100_fuji/dscf{4160,3004}.jpg
 # umount /mnt
 # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=win95
 # cp /mnt/dcim/100_fuji/dscf{4160,3005}.jpg
 # umount /mnt
 # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=winnt
 # cp /mnt/dcim/100_fuji/dscf{4160,3006}.jpg
 # umount /mnt
 # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=mixed
 # cp /mnt/dcim/100_fuji/dscf{4160,3007}.jpg
 # umount /mnt

Result? The camera only displays 3006.

The dualnames patch's filling filenames with random illegal
chars does seems to have a really foul side-effect after all.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regression

[tridge]

Hi Jan,

 > Then, I just tried everything to get an overview:

thanks!

 >  (All with DUALNAMES=n)
 >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=lower
 >  # cp /mnt/dcim/100_fuji/dscf{4160,3004}.jpg
 >  # umount /mnt
 >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=win95
 >  # cp /mnt/dcim/100_fuji/dscf{4160,3005}.jpg
 >  # umount /mnt
 >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=winnt
 >  # cp /mnt/dcim/100_fuji/dscf{4160,3006}.jpg
 >  # umount /mnt
 >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=mixed
 >  # cp /mnt/dcim/100_fuji/dscf{4160,3007}.jpg
 >  # umount /mnt
 > 
 > Result? The camera only displays 3006.
 > 
 > The dualnames patch's filling filenames with random illegal
 > chars does seems to have a really foul side-effect after 

actually, you've just proved the patch works as intended on your
device!

Only the shortname=winnt would create a 8.3 name when you use the
filename dscfNNNN.jpg. That is why my first patch in May forced 

   shortname_flags = VFAT_SFN_CREATE_WINNT;

when the patch triggered. I think this is the only sane behaviour for
a VFAT filesystem on Linux. Unfortunately it isn't the default, and at
least on my Ubuntu system the automatic mounting doesn't get this
right.

I'd love to change the default or do an override, at least when
dualnames is diabled. Hirofumi-san, what do you think?

Cheers, Tridge

PS: I'm assuming you actually typed "shortname=" not
"shortnames=". The option is not plural.

Re: CONFIG_VFAT_FS_DUALNAMES regression

[Jan Engelhardt]

On Friday 2009-07-03 02:25, tridge@samba.org wrote:
>
> >  (All with DUALNAMES=n)
> >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=lower
> >  # cp /mnt/dcim/100_fuji/dscf{4160,3004}.jpg
> >  # umount /mnt
> >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=win95
> >  # cp /mnt/dcim/100_fuji/dscf{4160,3005}.jpg
> >  # umount /mnt
> >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=winnt
> >  # cp /mnt/dcim/100_fuji/dscf{4160,3006}.jpg
> >  # umount /mnt
> >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=mixed
> >  # cp /mnt/dcim/100_fuji/dscf{4160,3007}.jpg
> >  # umount /mnt
> > 
> > Result? The camera only displays 3006.
> > 
> > The dualnames patch's filling filenames with random illegal
> > chars does seems to have a really foul side-effect after 
>
>actually, you've just proved the patch works as intended on your
>device!
>
>Only the shortname=winnt would create a 8.3 name when you use the
>filename dscfNNNN.jpg.

The mount(8) manpage becomes pretty ambiguous with the dualnames
patch. By default, "store a long name when the short name is not all
upper case" would apply - and is invoked because I am a lazy typist
who wrote "cp $this dscf3000" instead of "cp $this DSCF3000". Now
since dualnames=n is in effect, no long name would need to be written
because it still fits into 8.3 — subsequently, one would assume an
8.3 is generated for all shortname= cases.

>That is why my first patch in May forced 
>
>   shortname_flags = VFAT_SFN_CREATE_WINNT;

Making WINNT the default would cause many a `ls` output to just
scream at me in uppercaps because there are programs that
create long names with all-uppers.

shortname really needs to be split into "shortname that is applied
during creation" and "shortname that is applied upon readdir".

Re: CONFIG_VFAT_FS_DUALNAMES regression

[Jan Engelhardt]

On Friday 2009-07-03 02:25, tridge@samba.org wrote:
>
> >  (All with DUALNAMES=n)
> >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=lower
> >  # cp /mnt/dcim/100_fuji/dscf{4160,3004}.jpg
> >  # umount /mnt
> >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=win95
> >  # cp /mnt/dcim/100_fuji/dscf{4160,3005}.jpg
> >  # umount /mnt
> >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=winnt
> >  # cp /mnt/dcim/100_fuji/dscf{4160,3006}.jpg
> >  # umount /mnt
> >  # mount /dev/sdc1 /mnt -o iocharset=utf8,shortnames=mixed
> >  # cp /mnt/dcim/100_fuji/dscf{4160,3007}.jpg
> >  # umount /mnt
> > 
> > Result? The camera only displays 3006.
> > 
> > The dualnames patch's filling filenames with random illegal
> > chars does seems to have a really foul side-effect after 
>
>actually, you've just proved the patch works as intended on your
>device!
>
>Only the shortname=winnt would create a 8.3 name when you use the
>filename dscfNNNN.jpg.

The mount(8) manpage becomes pretty ambiguous with the dualnames
patch. By default, "store a long name when the short name is not all
upper case" would apply - and is invoked because I am a lazy typist
who wrote "cp $this dscf3000" instead of "cp $this DSCF3000". Now
since dualnames=n is in effect, no long name would need to be written
because it still fits into 8.3 — subsequently, one would assume an
8.3 is generated for all shortname= cases.

>That is why my first patch in May forced 
>
>   shortname_flags = VFAT_SFN_CREATE_WINNT;

Making WINNT the default would cause many a `ls` output to just
scream at me in uppercaps because there are programs that
create long names with all-uppers.

shortname really needs to be split into "shortname that is applied
during creation" and "shortname that is applied upon readdir".

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regression

[tridge]

Hi Jan,

 > The mount(8) manpage becomes pretty ambiguous with the dualnames
 > patch.

yes, the man page will need an update certainly. 

 > Making WINNT the default would cause many a `ls` output to just
 > scream at me in uppercaps because there are programs that
 > create long names with all-uppers.

well, you could also argue that having WINNT in effect does the
'correct' thing. It causes ls to display the name that is actually in
the filesystem. 

I think the current default for VFAT on Linux is rather misleading. It
always displays 8.3 names as lowercase, so MAIL.dat, foo.TXT and other
combinations always display as lowercase, regardless of what the
application that created the file chose for the name.

With the winnt option the names are shown as the application that
created them chose. That is also what all current versions of Windows
do as far as I know.

yes, this means that for a digital camera card, where the camera is
always choosing all uppercase for the filenames, it can look like
'shouting' on a Linux system where lowercase is more usual. It is
however an accurate representation and maximises compatibility.

I think the current defaults were chosen a long time ago, when Win95
was more the norm, and 16 bit apps that weren't written for case
preserving filesystems were common. That era is long gone.

 > shortname really needs to be split into "shortname that is applied
 > during creation" and "shortname that is applied upon readdir".

perhaps, but I think (as you discovered!) the current VFAT options are
already too complex, and very few people set them correctly. Adding
more options to get yet more varients to the case handling behaviour
doesn't seem like a good move to me.

Other OSes with FAT implementations just choose a sane default (which
probably matches our shortname=winnt pretty closely) and the user
never gets to choose. Choice isn't always a good thing!

I should also point out that if we followed Alan's reasoning then we'd
have to actually make all these options separate filesystems, and we'd
only be able to call the shortname=winnt one "VFAT" as that is the
only one that matches Windows behaviour. The others all treat the
filesystem differently to what Windows does. (no, I'm not seriously
proposing this, just pointing out that the "it has to be a different
filesystem name" has not been always applied previously).

I wonder if Pavel still thinks that VFAT was perfect before I came
along :-)

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regression

[Jan Engelhardt]

On Friday 2009-07-03 03:26, tridge@samba.org wrote:
>
> > Making WINNT the default would cause many a `ls` output to just
> > scream at me in uppercaps because there are programs that
> > create long names with all-uppers.
>
>well, you could also argue that having WINNT in effect does the
>'correct' thing. It causes ls to display the name that is actually in
>the filesystem. 
>
>I think the current default for VFAT on Linux is rather misleading. It
>always displays 8.3 names as lowercase,

There is no misleading in that, since VFAT is rather case-insensitive.
Certainly, lowering all 8.3 names is more appalling to the eyes than
keeping all-caps-longnames that way. I think I would even add a new
heuristic for case transformation on display to fit my personal
guidelines which would be:

  ($file, $ext) = ($filename =~ /^(.*)\.(.*)$/);
  $ext = lc $ext;
  if ($file =~ /^[A-Z]+$/) {
      $file = lc $ext;
  }

I think none of the shortcase= options does that at this time.

>I should also point out that if we followed Alan's reasoning then we'd
>have to actually make all these options separate filesystems[...]

Linux has been doing case conversion all years long so I do not think
Alan's request to make new filesystems was based solely upon
shortname=; otherwise, I suspect, it would have raised it earlier.

Re: CONFIG_VFAT_FS_DUALNAMES regression

[Jamie Lokier]

Jan Engelhardt wrote:
> 
> On Friday 2009-07-03 03:26, tridge@samba.org wrote:
> >
> > > Making WINNT the default would cause many a `ls` output to just
> > > scream at me in uppercaps because there are programs that
> > > create long names with all-uppers.
> >
> >well, you could also argue that having WINNT in effect does the
> >'correct' thing. It causes ls to display the name that is actually in
> >the filesystem. 
> >
> >I think the current default for VFAT on Linux is rather misleading. It
> >always displays 8.3 names as lowercase,
> 
> There is no misleading in that, since VFAT is rather case-insensitive.

It's case-insensitive for searches, but:

 - With the "posix" option, you can have multiple names with different cases.

 - It still preserves case.  I've got (Linux!) devices which copy
   files from FAT flash media onto their internal hard disk which is
   ext3, and then fail to work if the files on ext3 have the wrong case.

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 01:17, Jan Engelhardt wrote:

>
>On Wednesday 2009-07-01 16:05, Theodore Tso wrote:
>>On Wed, Jul 01, 2009 at 12:25:58PM +0100, Alan Cox wrote:
>>
>>> (most *FAT using products don't use Microsofts
>>> implementation). Testing it versus Windows and saying it works is
>>> not really adequate. Thats what ACPI and BIOS people do that *we*
>>> moan about all the time.
>>
>>[...]
>>The other big user I can think of are digital cameras, but (a)
>>normally most users read from them and then delete the pictures, and
>>rarely write to media meant for a digital camera, and (b) the DCIM
>>standard for digital cameras explicitly only supports 8.3 filenames
>>and so digital camera manufacturers explicitly don't need to deal with
>>Long File Names at all.  (Hmm.... I wonder why....)
>>[...]
>>Ultimately, though, requiring that every single possible device be
>>tested is probably not reasonable, so the best way to do this testing
>>is the way do most of our testing; we do basic due diligence, but then
>>we merge it into mainline and let our huge user community try it out.
>
>
>Yes, here is your precedent case. The DUALNAMES patch breaks the
>operation on my Fujifilm Finepix A210 digital camera.

Adding to that, I've just run it against every possible USB device I 
could dig up in my room (i.e. one more for now).

It also breaks with the Tekmax T-1000/TMP-1000 (nicknamed 
"ioneit"/"i-one-it"/"i-want-it") digital audio player. Its built-in 
directory browser does display the copied file (see below), but does not 
want to play it nor enqueue it into the playlist.

Again, a simple copy operation was performed — this time for both 
8.3 and long names, i.e.:

  $ cp working.mp3 TEST0000.mp3
  $ cp working.mp3 TEST12345678.mp3

None of the dualnames=n-created files were accepted.

(Combinations that worked: 2.6.29.5-w/o-dualnames-patch, and
2.6.31-rc1+dualnames=y)

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 01:17, Jan Engelhardt wrote:

>
>On Wednesday 2009-07-01 16:05, Theodore Tso wrote:
>>On Wed, Jul 01, 2009 at 12:25:58PM +0100, Alan Cox wrote:
>>
>>> (most *FAT using products don't use Microsofts
>>> implementation). Testing it versus Windows and saying it works is
>>> not really adequate. Thats what ACPI and BIOS people do that *we*
>>> moan about all the time.
>>
>>[...]
>>The other big user I can think of are digital cameras, but (a)
>>normally most users read from them and then delete the pictures, and
>>rarely write to media meant for a digital camera, and (b) the DCIM
>>standard for digital cameras explicitly only supports 8.3 filenames
>>and so digital camera manufacturers explicitly don't need to deal with
>>Long File Names at all.  (Hmm.... I wonder why....)
>>[...]
>>Ultimately, though, requiring that every single possible device be
>>tested is probably not reasonable, so the best way to do this testing
>>is the way do most of our testing; we do basic due diligence, but then
>>we merge it into mainline and let our huge user community try it out.
>
>
>Yes, here is your precedent case. The DUALNAMES patch breaks the
>operation on my Fujifilm Finepix A210 digital camera.

Adding to that, I've just run it against every possible USB device I 
could dig up in my room (i.e. one more for now).

It also breaks with the Tekmax T-1000/TMP-1000 (nicknamed 
"ioneit"/"i-one-it"/"i-want-it") digital audio player. Its built-in 
directory browser does display the copied file (see below), but does not 
want to play it nor enqueue it into the playlist.

Again, a simple copy operation was performed — this time for both 
8.3 and long names, i.e.:

  $ cp working.mp3 TEST0000.mp3
  $ cp working.mp3 TEST12345678.mp3

None of the dualnames=n-created files were accepted.

(Combinations that worked: 2.6.29.5-w/o-dualnames-patch, and
2.6.31-rc1+dualnames=y)
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

I should also mention that in the first patch I sent back in May, I
added some code that made the 8.3 name problem easier on the user. It
basically forced the shortname= option, overriding the mount option,
when the patch was triggered. The end result was that a name like
dcis3000.jpg would always be stored as a 8.3 name, and thus be visible
to digital cameras, even if the mount options were set to say that
lowercase names should be considered as "long" names.

Hirofumi objected, on the basis that I was overriding the VFAT
mount option. I think that overriding in this case would in fact be
worthwhile, as I don't think many users ever think carefully about
what shortname= option they are using, and unless you get it right
then you can get some real surprises.

Hirofumi-san, do you think that same objection you expressed
previously also applies to this new patch? For example, would you
think it reasonable to modify the patch to store the filename as 8.3
if it can fit in 8.3 in a case mapped manner? Or store as 8.3 if it is
all uppercase or all lowercase?

I think we will hit cases like the one Jan has pointed out where the
exising shortname= options will become even more confusing than they
were already when this new patch is applied, especially as I think
most distros these days magically supply a shortname= option when you
insert some VFAT formatted media.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[OGAWA Hirofumi]

tridge@samba.org writes:

> I should also mention that in the first patch I sent back in May, I
> added some code that made the 8.3 name problem easier on the user. It
> basically forced the shortname= option, overriding the mount option,
> when the patch was triggered. The end result was that a name like
> dcis3000.jpg would always be stored as a 8.3 name, and thus be visible
> to digital cameras, even if the mount options were set to say that
> lowercase names should be considered as "long" names.
>
> Hirofumi objected, on the basis that I was overriding the VFAT
> mount option. I think that overriding in this case would in fact be
> worthwhile, as I don't think many users ever think carefully about
> what shortname= option they are using, and unless you get it right
> then you can get some real surprises.
>
> Hirofumi-san, do you think that same objection you expressed
> previously also applies to this new patch? For example, would you
> think it reasonable to modify the patch to store the filename as 8.3
> if it can fit in 8.3 in a case mapped manner? Or store as 8.3 if it is
> all uppercase or all lowercase?

I have no objection to improve. And in this case, for this option, to
force the shortname= sounds reasonable.

Thanks.

> I think we will hit cases like the one Jan has pointed out where the
> exising shortname= options will become even more confusing than they
> were already when this new patch is applied, especially as I think
> most distros these days magically supply a shortname= option when you
> insert some VFAT formatted media.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

 > I have no objection to improve. And in this case, for this option, to
 > force the shortname= sounds reasonable.

Great! 

Which varient of this would you prefer I implement? I think the two
obvious choices are:

 1) do what I did in the May patch, which is to force
    VFAT_SFN_CREATE_WINNT when the patch code triggers.

 2) force VFAT_SFN_CREATE_WINNT, but also if the name is mixed case
 within either prefix or extension (such as "Mixed.TXT") then force
 that part of the name to all lowercase. So "Mixed.TXT" would become
 "mixed.TXT" on disk.

The 2nd choice maximises compatibility with devices that don't
understand long filenames (such as Jan's camera). The first option
keeps case preservation for devices that do understand long filenames.

What do you think?

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 03:11, tridge@samba.org wrote:
>
> 1) do what I did in the May patch, which is to force
>    VFAT_SFN_CREATE_WINNT when the patch code triggers.
>
> 2) force VFAT_SFN_CREATE_WINNT, but also if the name is mixed case
> within either prefix or extension (such as "Mixed.TXT") then force
> that part of the name to all lowercase. So "Mixed.TXT" would become
> "mixed.TXT" on disk.
>
>The 2nd choice maximises compatibility with devices that don't
>understand long filenames (such as Jan's camera). The first option
>keeps case preservation for devices that do understand long filenames.

The cam does not have a problem with long filenames — as said, it works 
on any to-date vanilla/distro Linux kernel, which, if I understood 
mount(8) right, always create a long filename due to me using a 
lowercase filename in conjunction with the default shortname=mixed.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 03:11, tridge@samba.org wrote:
>
> 1) do what I did in the May patch, which is to force
>    VFAT_SFN_CREATE_WINNT when the patch code triggers.
>
> 2) force VFAT_SFN_CREATE_WINNT, but also if the name is mixed case
> within either prefix or extension (such as "Mixed.TXT") then force
> that part of the name to all lowercase. So "Mixed.TXT" would become
> "mixed.TXT" on disk.
>
>The 2nd choice maximises compatibility with devices that don't
>understand long filenames (such as Jan's camera). The first option
>keeps case preservation for devices that do understand long filenames.

The cam does not have a problem with long filenames — as said, it works 
on any to-date vanilla/distro Linux kernel, which, if I understood 
mount(8) right, always create a long filename due to me using a 
lowercase filename in conjunction with the default shortname=mixed.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

 > The cam does not have a problem with long filenames

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 03:59, tridge@samba.org wrote:
>
> > The cam does not have a problem with long filenames 牴 
>
>That is curious. When you use a long filename (ie. a real long
>filename, not a mixed case 8.3 name), if you ask the camera for
>details on the photo,

No-no, as previously mentioned it ignores filenames not in the
format of dscf????.jpg. But if I interpret the hexdump right,
standard Linux creates a long name *anyway*, even for names
fitting in 8.3, if it's lowercase:

│00bce510 79 39 79 39 03 00 f5 5b-79 39 95 4b 00 00 00 00 |y9y9? ?[y9?K    |    
│00bce520 41 67 00 6f 00 64 00 64-00 61 00 0f 00 27 6d 00 |Ag o d d a ? 'm |    
│00bce530 6e 00 2e 00 7a 00 69 00-70 00 00 00 00 00 ff ff |n . z i p     ??|    
│00bce540 47 4f 44 44 41 4d 4e 20-5a 49 50 20 00 64 d2 20 |GODDAMN ZIP  d? |    

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 03:59, tridge@samba.org wrote:
>
> > The cam does not have a problem with long filenames 牴 
>
>That is curious. When you use a long filename (ie. a real long
>filename, not a mixed case 8.3 name), if you ask the camera for
>details on the photo,

No-no, as previously mentioned it ignores filenames not in the
format of dscf????.jpg. But if I interpret the hexdump right,
standard Linux creates a long name *anyway*, even for names
fitting in 8.3, if it's lowercase:

│00bce510 79 39 79 39 03 00 f5 5b-79 39 95 4b 00 00 00 00 |y9y9? ?[y9?K    |    
│00bce520 41 67 00 6f 00 64 00 64-00 61 00 0f 00 27 6d 00 |Ag o d d a ? 'm |    
│00bce530 6e 00 2e 00 7a 00 69 00-70 00 00 00 00 00 ff ff |n . z i p     ??|    
│00bce540 47 4f 44 44 41 4d 4e 20-5a 49 50 20 00 64 d2 20 |GODDAMN ZIP  d? |    
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

 > No-no, as previously mentioned it ignores filenames not in the
 > format of dscf????.jpg. But if I interpret the hexdump right,
 > standard Linux creates a long name *anyway*, even for names
 > fitting in 8.3, if it's lowercase:
 > 
 > 

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[OGAWA Hirofumi]

tridge@samba.org writes:

> right, that is not unexpected. It creates both the long name and the
> short name in this case as the short name is constrained to not be
> case preserving, and the filesystem normally tries to be both
> case-insensitive and case-preserving. As the long name has no
> constraints on the case of characters it creates both so that devices
> (like your camera I think) that only look at the 8.3 entries will see
> a file.
>
> That is why setting shortname=winnt is desirable when using the "no
> dualnames" patch I posted. That tells Linux to behave more like what
> current windows versions behave, which is to try and put it in the 8.3
> entry if it can, even if it is all lowercase.
>
> What I would like to see us do for maximum compatibility with devices
> like this, while still maximising the legal safety, is to take mixed
> case prefixes or suffixes that fit in 8.3 and uppercase them and set
> the CASE_LOWER_BASE and CASE_LOWER_EXT bits appropriately in the flags
> field, and not store a long name. That would makes us really nicely
> compatible with devices like your camera, at the price of losing mixed
> case preservation for 8.3 names.
>
> It would be another departure from previous behaviour however, so I
> want to get some feedback from Hirofumi-san first. 

I guess it would be depending on how many shortname only devices are
there though. At least for now, I'd like to keep that small.

Thanks.

> btw, if you still think your camera really is looking at the long
> filename entries, then try creating a file like DsCf2000.jpg with a
> unpatched kernel, mounted with shortname=winnt. That will create two
> directory entries for the file, one being DSCF2000.JPG and the other
> being DsCf2000.jpg.
>
> Then view the list of photos on the LCD back panel of your
> camera. Assuming it has a way to see the filename, does it display the
> mixed case name DsCf2000.jpg, or does it display a non-mixed name like
> dscf2000.jpg or DSCF2000.JPG.
>
> If it displays a mixed name (ie. DsCf) then your camera can indeed
> read long filename entries. If it displays a all uppercase or all
> lowercase name then I think it is only capable of reading short names
> from FAT filesystems. That is what I think you will find.
>
> Cheers, Tridge

-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 05:25, tridge@samba.org wrote:
>
>That is why setting shortname=winnt is desirable when using the "no
>dualnames" patch I posted. That tells Linux to behave more like what
>current windows versions behave, which is to try and put it in the 8.3
>entry if it can, even if it is all lowercase.

Right. That in itself is fine, but I'd still like to have readdir
return lowercase names here then. This is sort of what I seem to
remember from Windows: files only having a total-uppercase 8.3
name were still displayed as lowercase. This is why I am all for
splitting the shortcase option. (Maybe patches speak louder but
ATM I am so into Netfilter ;)

>btw, if you still think your camera really is looking at the long
>filename entries,

I toyed with the hexeditor a bit and modified the FAT entry's
filename directly. (I just hope there is not any checksum?)

 * if a filename is of the form /dscf????.jpg/i, the A210 requires
   that 8.3 names be all uppercase (even in the persence of a
   freeform long name) otherwise the OSD will (explicitly) display
   "READ ERROR".

 * if the 8.3 name is, for example, DsCE1234.jpg, the OSD seems to
   be confused and displays "00000000" as the picture index.

 => So uppercase 8.3 is a hard requirement for _any_ filename.

I also looked again at the finepix.img.bz2 I made available earlier.
It has a long entry "dscf3000.jpg", but "no shortname" (i.e. an
invalid one as generated by the patch), and that is most likely the
reason that the camera ignores this file completely.

>then try creating a file like DsCf2000.jpg with a
>unpatched kernel, mounted with shortname=winnt. That will create two
>directory entries for the file, one being DSCF2000.JPG and the other
>being DsCf2000.jpg.
>
>Then view the list of photos on the LCD back panel of your
>camera. Assuming it has a way to see the filename, does it display the
>mixed case name DsCf2000.jpg, or does it display a non-mixed name like
>dscf2000.jpg or DSCF2000.JPG.

It does not display the filename, just the index number. But the
image is present/selectable, because the 8.3 name is present and is
all-uppercase (see above).

>If it displays a mixed name (ie. DsCf) then your camera can indeed
>read long filename entries. If it displays a all uppercase or all
>lowercase name then I think it is only capable of reading short names
>from FAT filesystems. That is what I think you will find.

Capable or not — it only /needs/ 8.3 for DCIM operation :)

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 05:25, tridge@samba.org wrote:
>
>That is why setting shortname=winnt is desirable when using the "no
>dualnames" patch I posted. That tells Linux to behave more like what
>current windows versions behave, which is to try and put it in the 8.3
>entry if it can, even if it is all lowercase.

Right. That in itself is fine, but I'd still like to have readdir
return lowercase names here then. This is sort of what I seem to
remember from Windows: files only having a total-uppercase 8.3
name were still displayed as lowercase. This is why I am all for
splitting the shortcase option. (Maybe patches speak louder but
ATM I am so into Netfilter ;)

>btw, if you still think your camera really is looking at the long
>filename entries,

I toyed with the hexeditor a bit and modified the FAT entry's
filename directly. (I just hope there is not any checksum?)

 * if a filename is of the form /dscf????.jpg/i, the A210 requires
   that 8.3 names be all uppercase (even in the persence of a
   freeform long name) otherwise the OSD will (explicitly) display
   "READ ERROR".

 * if the 8.3 name is, for example, DsCE1234.jpg, the OSD seems to
   be confused and displays "00000000" as the picture index.

 => So uppercase 8.3 is a hard requirement for _any_ filename.

I also looked again at the finepix.img.bz2 I made available earlier.
It has a long entry "dscf3000.jpg", but "no shortname" (i.e. an
invalid one as generated by the patch), and that is most likely the
reason that the camera ignores this file completely.

>then try creating a file like DsCf2000.jpg with a
>unpatched kernel, mounted with shortname=winnt. That will create two
>directory entries for the file, one being DSCF2000.JPG and the other
>being DsCf2000.jpg.
>
>Then view the list of photos on the LCD back panel of your
>camera. Assuming it has a way to see the filename, does it display the
>mixed case name DsCf2000.jpg, or does it display a non-mixed name like
>dscf2000.jpg or DSCF2000.JPG.

It does not display the filename, just the index number. But the
image is present/selectable, because the 8.3 name is present and is
all-uppercase (see above).

>If it displays a mixed name (ie. DsCf) then your camera can indeed
>read long filename entries. If it displays a all uppercase or all
>lowercase name then I think it is only capable of reading short names
>from FAT filesystems. That is what I think you will find.

Capable or not — it only /needs/ 8.3 for DCIM operation :)
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

 > Right. That in itself is fine, but I'd still like to have readdir
 > return lowercase names here then.

you are in luck!

It just so happens that this is what happens when we use the patch
that Hirofumi and I just agreed on (ie. the case handling part of my
patch from May, combined with the current patch). 

I include it below. Can you please test it? It should be applied on
top of the previous patch. Note that you will not need to specify
shortname=winnt. In fact, if you don't specify any special options and
let the kernel default then you will get the behaviour you just
described.

Hirofumi-san, can you have a look at this too and see if it is how you
want to handle it? I built the patch against your fatfs-2.6 tree.

Cheers, Tridge

>From 244a8ebe72288a261e68b5aea04a86793a2f11f4 Mon Sep 17 00:00:00 2001
From: Andrew Tridgell <tridge@samba.org>
Date: Fri, 3 Jul 2009 22:15:02 +1000
Subject: [PATCH] Force case handling when dualnames are disabled for greater compatibility

When CONFIG_VFAT_FS_DUALNAMES is not set and a 8.3 name is created,
force the use of the shortname=winnt option to allow for lowercase or
uppercase prefix and extensions without using a long name.

Signed-off-by: Andrew Tridgell <tridge@samba.org>
---
 fs/fat/namei_vfat.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/fs/fat/namei_vfat.c b/fs/fat/namei_vfat.c
index 894f44d..9555a46 100644
--- a/fs/fat/namei_vfat.c
+++ b/fs/fat/namei_vfat.c
@@ -317,6 +317,11 @@ static int vfat_create_shortname(struct inode *dir, struct nls_table *nls,
 	int sz = 0, extlen, baselen, i, numtail_baselen, numtail2_baselen;
 	int is_shortname;
 	struct shortname_info base_info, ext_info;
+	unsigned opts_shortname = opts->shortname;
+
+#ifndef CONFIG_VFAT_FS_DUALNAMES
+	opts_shortname = VFAT_SFN_CREATE_WINNT;
+#endif
 
 	is_shortname = 1;
 	INIT_SHORTNAME_INFO(&base_info);
@@ -429,9 +434,9 @@ static int vfat_create_shortname(struct inode *dir, struct nls_table *nls,
 		if (vfat_find_form(dir, name_res) == 0)
 			return -EEXIST;
 
-		if (opts->shortname & VFAT_SFN_CREATE_WIN95) {
+		if (opts_shortname & VFAT_SFN_CREATE_WIN95) {
 			return (base_info.upper && ext_info.upper);
-		} else if (opts->shortname & VFAT_SFN_CREATE_WINNT) {
+		} else if (opts_shortname & VFAT_SFN_CREATE_WINNT) {
 			if ((base_info.upper || base_info.lower) &&
 			    (ext_info.upper || ext_info.lower)) {
 				if (!base_info.upper && base_info.lower)
-- 
1.6.0.4

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[OGAWA Hirofumi]

tridge@samba.org writes:

> Hirofumi-san, can you have a look at this too and see if it is how you
> want to handle it? I built the patch against your fatfs-2.6 tree.

Looks good to me. I've applied the patch + Cc:. 

>>From 244a8ebe72288a261e68b5aea04a86793a2f11f4 Mon Sep 17 00:00:00 2001
> From: Andrew Tridgell <tridge@samba.org>
> Date: Fri, 3 Jul 2009 22:15:02 +1000
> Subject: [PATCH] Force case handling when dualnames are disabled for greater compatibility

[PATCH] fat: Force case handling...

Please add "fat: " prefix at next time.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Friday 2009-07-03 14:24, tridge@samba.org wrote:
>Hi Jan,
>
> > Right. That in itself is fine, but I'd still like to have readdir
> > return lowercase names here then.
>
>you are in luck!

Dunno.
(It finally turns Monday...)

>I include it below. Can you please test it? It should be applied on
>top of the previous patch. Note that you will not need to specify
>shortname=winnt. In fact, if you don't specify any special options and
>let the kernel default then you will get the behaviour you just
>described.

With this extra patch:

 * "cp dscf4160.jpg dscf3010.jpg" works

 * "cp dscf4160.jpg dscF3011.jpg" does not work - i.e. cam ignores the
   file without error.
   Reason for that is that dscF3011 has an illegal 8.3 name

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

 > With this extra patch:
 > 
 >  * "cp dscf4160.jpg dscf3010.jpg" works

good

 >  * "cp dscf4160.jpg dscF3011.jpg" does not work - i.e. cam ignores the
 >    file without error.
 >    Reason for that is that dscF3011 has an illegal 8.3 name

yep, that's expected. I suggested a possible way of handling this to
Hirofumi-san (ie. change the case of the name), but he prefers to keep
things simple for now.

It is probably not unreasonable to say that you should not be using
mixed case names when you are putting files onto a device that doesn't
understand mixed case. The reason it worked before my patch is that
the kernel forced the case of the 8.3 name, and only retained the
mixed case for the long name.

How did things go with your mp3 players?

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[David Newall]

tridge@samba.org wrote:
> Hi Jan,
>
>  >  * "cp dscf4160.jpg dscF3011.jpg" does not work - i.e. cam ignores the
>  >    file without error.
>  >    Reason for that is that dscF3011 has an illegal 8.3 name
>
> yep, that's expected. I suggested a possible way of handling this to
> Hirofumi-san (ie. change the case of the name), but he prefers to keep
> things simple for now.
>   

Expected or not, it's a serious bug. If it doesn't canonify the
filename, it should return an error. Lying by returning success is
horridly wrong. And I might just suggest that as people are now
accustomed to typing mixed-case filenames, it's a bug that's likely to
occur more frequently than it might at first seem.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[David Newall]

tridge@samba.org wrote:
> Hi Jan,
>
>  >  * "cp dscf4160.jpg dscF3011.jpg" does not work - i.e. cam ignores the
>  >    file without error.
>  >    Reason for that is that dscF3011 has an illegal 8.3 name
>
> yep, that's expected. I suggested a possible way of handling this to
> Hirofumi-san (ie. change the case of the name), but he prefers to keep
> things simple for now.
>   

Expected or not, it's a serious bug. If it doesn't canonify the
filename, it should return an error. Lying by returning success is
horridly wrong. And I might just suggest that as people are now
accustomed to typing mixed-case filenames, it's a bug that's likely to
occur more frequently than it might at first seem.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

David Newall wrote:
> tridge@samba.org wrote:
> > Hi Jan,
> >
> >  >  * "cp dscf4160.jpg dscF3011.jpg" does not work - i.e. cam ignores the
> >  >    file without error.
> >  >    Reason for that is that dscF3011 has an illegal 8.3 name
> >
> > yep, that's expected. I suggested a possible way of handling this to
> > Hirofumi-san (ie. change the case of the name), but he prefers to keep
> > things simple for now.
> >   
> 
> Expected or not, it's a serious bug. If it doesn't canonify the
> filename, it should return an error. Lying by returning success is
> horridly wrong. And I might just suggest that as people are now
> accustomed to typing mixed-case filenames, it's a bug that's likely to
> occur more frequently than it might at first seem.

It's no different from writing a long name, though, is it?

"cp dscf4160.jpg dscf3011.jpeg" would also be ignored by the
camera, because there is no corresponding 8.3 file name.

It's a bit clearer when mounted with shortname=winnt or
shortname=mixed, because then I'd expect files created by the camera
to show as upper case in Linux, and it would be obvious that you might
have to create new files for the camera that look the same as the camera's.

The current Linux default shows 8.3 short-name files in lower case
(i.e. pure FAT shows as lower case), but files created in Linux with
8.3 lower case names create a long name.  Which is bound to cause some
confusion.

I agree with tridge, shortname=winnt makes sense when dualnames are off.

Unfortunately it doesn't preserve case when creating in Linux and
reading in Windows 95/98/ME and devices with a similar implementation.

For that reason, shortname=mixed is better when dualnames are on.

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Monday 2009-07-06 15:05, tridge@samba.org wrote:
>
> > With this extra patch:
> > 
> >  * "cp dscf4160.jpg dscf3010.jpg" works
> >  * "cp dscf4160.jpg dscF3011.jpg" does not work - i.e. cam ignores the
> >    file without error.
> >    Reason for that is that dscF3011 has an illegal 8.3 name
>
>It is probably not unreasonable to say that you should not be using
>mixed case names when you are putting files onto a device that doesn't
>understand mixed case. The reason it worked before my patch is that
>the kernel forced the case of the 8.3 name, and only retained the
>mixed case for the long name.
>
>How did things go with your mp3 players?

Not too different from previously[1].

Entirely-lowercase fits-8-3-names work as usual, but of course are 
displayed with an uppercase filename in the OSD (technically 
correct, but still ick!).

Whenever a long name is created and an 8.3 name is omitted (or rather, 
an invalid one is generated), files are displayed in the browser but 
fail to enqueue/play.

As it stands, my two devices always want a valid 8.3 name. That in 
stark conflict to your patch. And I am not going to do without long 
filenames.



[1] http://osdir.com/ml/linux-kernel/2009-07/msg01314.html

Random statistic of the day about audio file's name length,
as output by R:
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   6.00   18.00   25.00   26.62   33.00   77.00 

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

 > Whenever a long name is created and an 8.3 name is omitted (or rather, 
 > an invalid one is generated), files are displayed in the browser but 
 > fail to enqueue/play.

Can you confirm whether without my patch, if you put a file on the
device that has a long name like ThisIsALongName.mp3 that the correct
long name is displayed in the browser on the device?

If so then it would have to be a long name capable device, and in that
case I'll have to see if I can buy one on eBay so I can have a play :-)

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Monday 2009-07-06 22:26, tridge@samba.org wrote:
>
> > Whenever a long name is created and an 8.3 name is omitted (or rather, 
> > an invalid one is generated), files are displayed in the browser but 
> > fail to enqueue/play.
>
>Can you confirm whether without my patch, if you put a file on the
>device that has a long name like ThisIsALongName.mp3 that the correct
>long name is displayed in the browser on the device?

Without the two of your patches everything Works As Expected™.

(With yours, the file is displayed - as in "existence" -, with longname,
but does not play.)

>If so then it would have to be a long name capable device, and in that
>case I'll have to see if I can buy one on eBay so I can have a play :-)

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Monday 2009-07-06 22:26, tridge@samba.org wrote:
>
> > Whenever a long name is created and an 8.3 name is omitted (or rather, 
> > an invalid one is generated), files are displayed in the browser but 
> > fail to enqueue/play.
>
>Can you confirm whether without my patch, if you put a file on the
>device that has a long name like ThisIsALongName.mp3 that the correct
>long name is displayed in the browser on the device?

Without the two of your patches everything Works As Expected™.

(With yours, the file is displayed - as in "existence" -, with longname,
but does not play.)

>If so then it would have to be a long name capable device, and in that
>case I'll have to see if I can buy one on eBay so I can have a play :-)
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

 > Without the two of your patches everything Works As Expected

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Monday 2009-07-06 20:55, Jan Engelhardt wrote:
>>How did things go with your mp3 players?
>[...]
>As it stands, my two devices always want a valid 8.3 name.

On or about June 26, James Bottomley exchanged these words to Andrew Tridgell:
>So the patch has been tested with Vista, Windows 7 and Windows XP


Vista, 7... nothing special.

So let me fill in.

Windows 98 can't make anything of the stunted vfat entries either[3],
and there's blanks for 16-bit programs[4]. They too, it seems, always
want an 8.3 entry in any case.
It does not crash, but neither of these results is usable.

This dualnames patch just won't fly in practice.


[3] http://picpaste.de/w98dualnames.png
[4] http://picpaste.de/xpwith16bit.png
(pics kept for 7 days from now)

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

Jan Engelhardt wrote:
> 
> On Monday 2009-07-06 20:55, Jan Engelhardt wrote:
> >>How did things go with your mp3 players?
> >[...]
> >As it stands, my two devices always want a valid 8.3 name.
> 
> On or about June 26, James Bottomley exchanged these words to Andrew Tridgell:
> >So the patch has been tested with Vista, Windows 7 and Windows XP
> 
> 
> Vista, 7... nothing special.
> 
> So let me fill in.
> 
> Windows 98 can't make anything of the stunted vfat entries either[3],
> and there's blanks for 16-bit programs[4]. They too, it seems, always
> want an 8.3 entry in any case.
> It does not crash, but neither of these results is usable.
>
> This dualnames patch just won't fly in practice.
> 
> 
> [3] http://picpaste.de/w98dualnames.png
> [4] http://picpaste.de/xpwith16bit.png
> (pics kept for 7 days from now)

Summary of pics, for posterity:

[3] shows an MS-DOS window on a Windows 98 desktop, listing the A:
    drive:

      Microsoft(R) Windows 98
         (C)Copyright Microsoft Corp 1981-1998.

      A:\>Dir

       Datantrager in Laufwerk A: hat keine Bezeichnung
       Seriennummer des Datentragers: 4A52-570A
       Verzeichnis von A:\

      SHORT    TXT            12  06.07.09  21:57 SHORT.TXT
                              12  06.07.09  21:57 
                              12  06.07.09  21:57 
               3 Datai(en)                    36 Bytes
               0 Verzeichnis(se)       1.456.128 Bytes frei

      A:\>

Question: In Windows 98, is it just the MS-DOS box which cannot see
some of the filenames, or is the normal file explorer affected too?

[4] shows a old-looking Windows application, presumably 16-bit running
    on XP, with a File Selection box listing the a:\ drive.  Four
    files are shown on the a:\ drive in a selection box, but the first
    of them is completely blank.

In Windows XP, it suggests to me that the Windows API function
GetShortPathName() returns the 8.3 file name entry.  Call it an
educated guess, and I'll guess the same applies in Server 2003, Vista,
Server 2008 and Windows 7.

All those Windows versions also have GetLongPathName() which does the
reverse transformation.  Presumably that also expects the 8.3 entry to
exist.

The fact these were added in Windows 2000 implies they're not an
ancient thing from the Windows 95 era, but specifically used by
applications since Windows 2000.

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Monday 2009-07-06 22:58, Jamie Lokier wrote:
>> On Monday 2009-07-06 20:55, Jan Engelhardt wrote:
>> >>How did things go with your mp3 players?
>> >[...]
>> >As it stands, my two devices always want a valid 8.3 name.
>> 
>> On or about June 26, James Bottomley exchanged these words to Andrew Tridgell:
>> >So the patch has been tested with Vista, Windows 7 and Windows XP
>> 
>> [Test with Windows 98, and 16-bit GDI programs under XP]
>> [3] http://picpaste.de/w98dualnames.png
>> [4] http://picpaste.de/xpwith16bit.png
>> (pics kept for 7 days from now)
>
>Summary of pics, for posterity:
>[3] shows an [...] Question: In Windows 98, is it just the MS-DOS box
>which cannot see some of the filenames, or is the normal file explorer
>affected too?

[3] shows an MS-DOS window on a Win98 desktop, having just run the
"dir" command. In the background is a Win98 explorer window, for
comparison with the dir command.
Neither of the two ways of listing the contents of a directory show
any filenames for <illegal 8.3, long name present> entries as produced
by tridge's vfat patches.

>[4] shows a old-looking Windows application, presumably 16-bit running
>    on XP, with a File Selection box listing the a:\ drive.  Four
>    files are shown on the a:\ drive in a selection box, but the first
>    of them is completely blank.

As the filename and the intro text had said..

[4] 16-bit program (Windows 3.x era) running on XP (but probably also
happens earlier). File dialog shows blank line where an entry has an
illegal 8.3 name; also as a result of the vfat aptches.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

Jan Engelhardt wrote:
> 
> On Monday 2009-07-06 22:58, Jamie Lokier wrote:
> >> On Monday 2009-07-06 20:55, Jan Engelhardt wrote:
> >> >>How did things go with your mp3 players?
> >> >[...]
> >> >As it stands, my two devices always want a valid 8.3 name.
> >> 
> >> On or about June 26, James Bottomley exchanged these words to Andrew Tridgell:
> >> >So the patch has been tested with Vista, Windows 7 and Windows XP
> >> 
> >> [Test with Windows 98, and 16-bit GDI programs under XP]
> >> [3] http://picpaste.de/w98dualnames.png
> >> [4] http://picpaste.de/xpwith16bit.png
> >> (pics kept for 7 days from now)
> >
> >Summary of pics, for posterity:
> >[3] shows an [...] Question: In Windows 98, is it just the MS-DOS box
> >which cannot see some of the filenames, or is the normal file explorer
> >affected too?
> 
> [3] shows an MS-DOS window on a Win98 desktop, having just run the
> "dir" command. In the background is a Win98 explorer window, for
> comparison with the dir command.
> Neither of the two ways of listing the contents of a directory show
> any filenames for <illegal 8.3, long name present> entries as produced
> by tridge's vfat patches.

Indeed.  I missed the explorer window behind the MS-DOS box, because
it has a few icons but the file names are missing.  They're missing
because of modified FAT.

> >[4] shows a old-looking Windows application, presumably 16-bit running
> >    on XP, with a File Selection box listing the a:\ drive.  Four
> >    files are shown on the a:\ drive in a selection box, but the first
> >    of them is completely blank.
>
> As the filename and the intro text had said..
>
> [4] 16-bit program (Windows 3.x era) running on XP (but probably also
> happens earlier). File dialog shows blank line where an entry has an
> illegal 8.3 name; also as a result of the vfat aptches.

Your original mail didn't say about the	MS-DOS box in the 98 pic,
which is why I explained it (since the pics will disappear but the l-k
mail will be archived).

I wanted to point that out because MS-DOS file access (even in a DOS
box) is different from Windows file access.

It wouldn't be surprising if modified FAT broke on MS-DOS but worked
in 16-bit Windows - but thanks for showing that it breaks 16-bit Windows too.

And then I wanted to say the XP pic wasn't an MS-DOS box like the 98 pic...

That's all.

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 542 bytes --]


On Tuesday 2009-07-07 00:24, Jamie Lokier wrote:
>
>Your original mail didn't say about the	MS-DOS box in the 98 pic,
>which is why I explained it (since the pics will disappear but the l-k
>mail will be archived).

I could attach it as MIME, it's not overly large (2x8KB unmimed) thanks to
optipng postcompression. Then again I wonder which archives allow
access to such, sometimes they seem stunted enough to only handle .diffs.
I could host them myself too. Hm.
I'll allow myself to strip Cc since it's just for the archives - here 
goes:

[-- Attachment #2: Type: IMAGE/png, Size: 8489 bytes --]

[-- Attachment #3: Type: IMAGE/png, Size: 8845 bytes --]

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

 > Windows 98 can't make anything of the stunted vfat entries either[3],
 > and there's blanks for 16-bit programs[4]. They too, it seems, always
 > want an 8.3 entry in any case.
 > It does not crash, but neither of these results is usable.

Interesting! Thanks for the testing.

I didn't have a win98 test image, but I've scrounged up a copy of the
install disks now from an old MSDN subscription, and I've built a
win98 virtual machine. I stopped testing Win9X against Samba a long
time ago (along with OS/2) so I didn't have the right setup to hand.

I've now done some experiments, and I have reproduced the problem you
saw. I've also experimented with changing the
vfat_build_dummy_83_buffer() to try some other combinations. I've
found that with a simple memset(msdos_name, ' ', 11) that Win98 works
pretty well:

  http://picpaste.com/Win98-longnames.png

It does show one error though. In the DOS box directory listing on the
left notice that it shows both a long name and a short name for the
files, with the long name being a truncated form of the short
name. The normal commands like xcopy, notepad etc all seem to work
fine though, so practical compatibility seems pretty good.

The problem with that simple memset() approach with spaces is that it
would cause more crashes with WinXP. It does show that there might be
some other combination that works with both though. I'll play a bit
more and see what I can find.

 > This dualnames patch just won't fly in practice.

well, "won't fly" depends on your POV I guess. Unless we're hoping
that all the Microsoft lawyers take early retirement, I think we do
need to have some strategy to avoiding more companies having the same
problems that TomTom had. 

Would it be worth it to lose VFAT compatibility with a dead OS like
Win9X?  Quite possibly it would be for a pretty large number of
vendors.

What we need to explore now is the limits of compatibility that we can
reach. Your testing has really helped with that! I'm trying to track
down a Tekmax or 'IOneIt' mp3 player but no luck on eBay yet - I don't
suppose anyone on this list will sell me one? Are they sold in Germany
somewhere Jan?

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 2848 bytes --]

Am Dienstag 07 Juli 2009 schrieb tridge@samba.org:
> I've now done some experiments, and I have reproduced the problem you
> saw. I've also experimented with changing the
> vfat_build_dummy_83_buffer() to try some other combinations. I've
> found that with a simple memset(msdos_name, ' ', 11) that Win98 works
> pretty well:
>
>   http://picpaste.com/Win98-longnames.png
>
> It does show one error though. In the DOS box directory listing on the
> left notice that it shows both a long name and a short name for the
> files, with the long name being a truncated form of the short
> name. The normal commands like xcopy, notepad etc all seem to work
> fine though, so practical compatibility seems pretty good.
>
> The problem with that simple memset() approach with spaces is that it
> would cause more crashes with WinXP. It does show that there might be
> some other combination that works with both though. I'll play a bit
> more and see what I can find.
>
>  > This dualnames patch just won't fly in practice.
>
> well, "won't fly" depends on your POV I guess. Unless we're hoping
> that all the Microsoft lawyers take early retirement, I think we do
> need to have some strategy to avoiding more companies having the same
> problems that TomTom had.

Following this thread I get more and more the impression that this patch 
is playing roulette regarding compatibility with countless devices which 
use fat/vfat, with different Windows versions and with countless Windows 
applications including but not limited to low level filesystem check, 
repair and cloning tools. And I start to get the impression that it 
becomes unmanageably complex to make a clear assessment on compatibility 
in all those circumstances. Thus I can't figure how a realistic assessment 
on the impact of this patch could be made.

Have low level filesystem check, repair and cloning tools been checked 
against the patch at all?

I think the patch actively *corrupts* perfectly fine shortname FAT 
filesystems and at least for certain use scenarios even those with long 
name support.

Thus I would certainly not enable it unless forced too - already just for 
*technical* reasons. I probably would even default to compile my own 
kernel when I find that the distribution of my choice (Debian) would enable 
it.

Politically spoken I think the patch tries to workaround a problem that 
better is fixed properly and causes a precedence for other politically, 
juristically motivated patches. If the Linux kernel would be changed to 
avoid any software patent issues I am not sure whether it would even be 
able to boot anymore. As such I think the patch should not be part of 
vanilla kernels.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Martin Steigerwald]

Am Dienstag 07 Juli 2009 schrieb Martin Steigerwald:
> I think the patch actively corrupts perfectly fine shortname FAT
> filesystems and at least for certain use scenarios even those with long
> name support.

Well for short name FAT only if it gets a chance to. I.e. when the 
filesystem is mounted as vfat.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Martin,

 > Have low level filesystem check, repair and cloning tools been checked 
 > against the patch at all?

I have tested chkdsk.exe obvious, and I have reported the bug in
chkdsk that this testing has found to Microsoft.

I haven't tested tools like ghost or 3rd party tools. I don't actually
have any of those tools myself, although I guess I could go out and
buy them. Does anyone on this list have those tools and can let me
know if they show any problems?

 > I think the patch actively *corrupts* perfectly fine shortname FAT 
 > filesystems and at least for certain use scenarios even those with long 
 > name support.

The patch only changes the values stored for new files created by
Linux, so I think it is going a bit far to say it "actively corrupts"
filesystems. 

I am looking into the Win9X issued raised by Jan. As I mentioned in my
mail to him, it seems to work better with some different values in the
11 bytes. I'll keep looking at that, although I don't think Win9X
support is a high priority for anyone any more. After installing Win98
in a virtual machine I connected it to the windows update service to
see if there had been updates to the old install images I had, and I
found it couldn't even connect to windows update, it just throws a
page full of html errors:

  http://samba.org/tridge/Win98-update.png

When the vendor of an operating system doesn't even bother to display
a clean "sorry, you don't get updates any more" page for their OS then
I think it is safe to say that the operating system is dead and
buried.

That leaves just the 'IOneIt' MP3 player that Jan tested. I have
ordered a bunch of different cheap MP3 players from Hong Kong on eBay
to see if I can reproduce that.

 > Thus I would certainly not enable it unless forced too - already
 > just for *technical* reasons. I probably would even default to
 > compile my own kernel when I find that the distribution of my
 > choice (Debian) would enable it.

I can't speak for Debian, but I would be not surprised if they do
enable this patch. Your opinion certainly lends weight to the idea
that it should be configurable, but I don't think it leads to the
conclusion that the patch should not be in the upstream kernel at all.

 > Politically spoken I think the patch tries to workaround a problem that 
 > better is fixed properly and causes a precedence for other politically, 
 > juristically motivated patches.

But what is 'fixed properly'? Exactly what should be done beyond the
activism that is already trying to change the patent systems around
the world?

I am a conscientious objector when it comes to patents, so I have
never filed any patents myself. I have also fought patents in other
ways, such as the partial win that I helped with in relation to
patents in the EU anti-trust case, and I have helped quite a few
people with patent workarounds.

It would be great if the Linux community had enough clout to force
through a change in patent law, but as yet we don't. When there was
the possibility of all of the politicians blackberries in the US
getting switched off due to a patent dispute I thought that perhaps we
finally had something that would get the attention of the people who
make these silly laws. That didn't end up resulting in any significant
change, so I don't think it is reasonable to think that the threat of
Linux desktops not being able to share USB keys with Windows will
cause politicians around the world to suddenly take an interest.

 > If the Linux kernel would be changed to avoid any software patent
 > issues I am not sure whether it would even be able to boot
 > anymore.

That argument can be used with pretty much any software (both
proprietary and free), not just the Linux kernel.

The answer is that patents that are being actively litigated, such as
the VFAT patents, fall into an entirely different class than the rest
of the patents out there. 

In the case of the GIF patents the correct answer was a concerted
effort to switch to a new format. That was a fantastic campaign and
largely successful.

We don't, as yet, have any equivalent campaign to get behind for these
VFAT patents. The calls for changing to a different filesystem format
are great, but they fall down in an even worse way than what I have
proposed on exactly the same issues. This hypothetical new format
won't work with cheap MP3 players, won't work with Win9X, and almost
certainly won't work with existing Windows boxes (yes, I know about
UDF, but if you actually try it you'll see it isn't the panacea some
have claimed). So in what way is it a solution, even if the new format
existed?

If you can propose a truly workable alternative then I would be
delighted to never have to think about FAT filesystems again in my
life. 

Meanwhile, I have proposed what I think is the best alternative we
have at the moment. It is certainly unpalatable to have to deal with
patents at all, but I think it is better that we work around them than
to see the problems that TomTom faced spread across the Linux world.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

>   http://samba.org/tridge/Win98-update.png
> 
> When the vendor of an operating system doesn't even bother to display
> a clean "sorry, you don't get updates any more" page for their OS then
> I think it is safe to say that the operating system is dead and
> buried.

There are lots of Win98 systems still alive out there. The "last few" in
the Win98 world is rather a lot due to the original enormous user base.

> that it should be configurable, but I don't think it leads to the
> conclusion that the patch should not be in the upstream kernel at all.

I think we already proved it had no use upstream. Vendors will remove
the code from their source tree if worried about patents so including it
in the base tree is really irrelevant. So I find your argument about this
less than convincing.

The patch serves no purpose but to confuse users and increasingly it is
shown to break systems horribly.

There might have been a limited case for a not-quite-vfat-fs "tridgefat"
etc if it was both more compatible and if the vendors would use the
option. But given its not compatible and vendors won't why bother at all ?

I also note you keep talking about vendors. This is an open list yet I
don't hear a word from the vendors you claim to represent in support of
this patch set, and saying they would enable it. Not one voice seems to
have appeared.

The decision sequence goes something like this

	- do we want to ship the feature because of patent concern
	> do not ship
	- is it less risk to remove the source from our build tree or
	  configure it out
	> remove from the tree
	- is there a functionality difference to the user between
	  removing or unconfiguring it
	> No

At that point nobody managing risk is going to do anything but remove the
code that worries them. It's additional risk with no return.


Alan

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Alan,

 > There are lots of Win98 systems still alive out there. The "last few" in
 > the Win98 world is rather a lot due to the original enormous user base.

I find it rather ironic that you are advocating leaving the Linux
kernel open to further legal attacks by Microsoft because if we didn't
then we might lose compatibility with an operating system which
Microsoft themselves abandoned a long time ago.

 > I think we already proved it had no use upstream. Vendors will remove
 > the code from their source tree if worried about patents so including it
 > in the base tree is really irrelevant. So I find your argument about this
 > less than convincing.

we've been round this loop before. See my previous emails where I
explained why I think having it upstream is an advantage. See also the
reply from James.

 > I also note you keep talking about vendors. This is an open list yet I
 > don't hear a word from the vendors you claim to represent in support of
 > this patch set, and saying they would enable it. Not one voice seems to
 > have appeared.

I never claimed to represent any vendors. I said that it was my
opinion that many vendors will want to apply this patch, and you seem
to support that opinion as you've several times stated that you think
that vendors will apply it in an unconditional form.

 > At that point nobody managing risk is going to do anything but remove the
 > code that worries them. It's additional risk with no return.

Some vendors may well do that. Some may decide to keep it as a compile
time option. The legal advice that I've seen is that keeping it as a
compile time option is fine. 

If it's pulled out of kernel.org trees then:

 - each vendor may end up with slightly different varients. That means
   we could have Linux devices behaving inconsistently.

 - the testing and discussion may end up happening in a less open
   manner. I think the testing and discussion on lkml has been very
   valuable. As you've noted, vendors have not been actively
   participating in these discussions. Do you think they'll suddenly
   decide to start discussing things openly if we move it out of a
   kernel.org tree? You must know how reluctant vendors are to draw
   attention to themselves when it comes to patents.

 - some vendors may decide not to use Linux, and switch to embedded
   windows instead if we don't have a clearly supported way of
   avoiding these patents in the Linux kernels.

 - we'd be setting up the kernel to have a deliberate long term
   difference between what a large part of the user base runs and what
   is tested for kernel.org kernels. As I said previously, I think
   that is poor software engineering practice. I know you disagree,
   but I also know that some other people do agree. I suspect you
   would be more inclined to agree if this patch had nothing to do
   with patents.

I suspect you are right that many vendors would apply it anyway if it
wasn't in a kernel.org kernel. Is that sufficient to stop a new round
of lawsuits? Maybe. If that is what is decided then I guess we'll find
out over the next year or two.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> I find it rather ironic that you are advocating leaving the Linux
> kernel open to further legal attacks by Microsoft because if we didn't

Now you are being totally inconsistent.

Either leaving the code in the source is not a problem, _or_ if it could
be a problem (for US citizens) then vendors will need to remove the code
from their tree completely.

> we've been round this loop before. See my previous emails where I
> explained why I think having it upstream is an advantage. See also the
> reply from James.

James reply was based on the fallacy that vendors will keep the source in
their tree. They don't do that *today*, look at their trees.

>  > this patch set, and saying they would enable it. Not one voice seems to
>  > have appeared.
> 
> I never claimed to represent any vendors. I said that it was my
> opinion that many vendors will want to apply this patch, and you seem

So why have the vendors not commented on list ? If they will want to
apply the patch why don't I see supporting email from them ?

>  > At that point nobody managing risk is going to do anything but remove the
>  > code that worries them. It's additional risk with no return.
> 
> Some vendors may well do that. Some may decide to keep it as a compile
> time option. The legal advice that I've seen is that keeping it as a
> compile time option is fine. 

I can't comment on the advice I've seen directly, but I will point out
that every vendor I am aware of *removes completely* any source material
which they view with concern. You can download the packages and review
them if you doubt it.

> If it's pulled out of kernel.org trees then:
> 
>  - each vendor may end up with slightly different varients. That means
>    we could have Linux devices behaving inconsistently.

The Linux Foundation can manage a reference patch if it wants, or someone
can set up "Linux-USA" or similar git trees on top of the real kernel
one. Git is actually rather good at that.

>  - the testing and discussion may end up happening in a less open
>    manner. I think the testing and discussion on lkml has been very

Less open than "I'm sorry our lawyer says we can't use a non colliding
entry but we can't tell you why", plus of course to one person a "we will
tell you off list", which I understand they haven't.

>    valuable. As you've noted, vendors have not been actively
>    participating in these discussions. Do you think they'll suddenly
>    decide to start discussing things openly if we move it out of a
>    kernel.org tree? You must know how reluctant vendors are to draw
>    attention to themselves when it comes to patents.

I don't actually think they will participate either way - so this
argument of yours doesn't hold water either.

>  - some vendors may decide not to use Linux, and switch to embedded
>    windows instead if we don't have a clearly supported way of
>    avoiding these patents in the Linux kernels.

That is a business problem not a technical one. The people who keep the
out of tree patch need to share and discuss what they are doing. They
need to do that in a framework which allows them to do it right. That
means the Linux Foundation or similar with a lawyer present is the
nearest they can do to "open" any way.

>  - we'd be setting up the kernel to have a deliberate long term
>    difference between what a large part of the user base runs and what
>    is tested for kernel.org kernels. As I said previously, I think

That has always been the case. That is the established practice for all
packages in the open source world today. The kernel is not different. If
you put in special American treatment you need to do the same for Chinese
compliance (No reference to Taiwan as a country or implication thereof in
code page naming etc) and every other country in the world.

It's also an incredibly complicated specialist field. The vendors have
teams of experts working on it who deal with everything. Dumping that on
the community is neither appropriate nor fair. You want to sell Linux
products in the USA, you do the compliance work. Given that deciding what
do about FAT is a tiny spec of the vast red tape you must complete (from
crypto exports to liability insurance) its no real overhead to the vendor.

>    that is poor software engineering practice. I know you disagree,
>    but I also know that some other people do agree. I suspect you
>    would be more inclined to agree if this patch had nothing to do
>    with patents.

No. Red Hat have lots of patches that belong in their products that I
know about - from removing country flags from some versions of KDE to
removing certain crypto algorithms from ssh. None are upstream, none
belong upstream. The kernel is *NOT* different to every other of the
hundreds of packages shipped by vendors.

There is an established practice over many years for this stuff, and it
is contrary to your model of messing with base reference code to meet
random national legislation.

> I suspect you are right that many vendors would apply it anyway if it
> wasn't in a kernel.org kernel. Is that sufficient to stop a new round
> of lawsuits? Maybe. If that is what is decided then I guess we'll find
> out over the next year or two.

Maybe kernel.org shouldn't be hosted in the USA. That seems to be what
the "reduce risk" argument you are making is really saying if you continue
it to its logical conclusion.

No action you take except closing down and going home determines whether
patent lawsuits get filed. You know that. In the US and quite a few other
states patent lawsuits are rarely based on fact, evidence or process.
They are a business tool used to raise costs and hamper rivals. You file
patent lawsuits to force people to use your hardware, to slow down
business rivals who are ahead in a key market and to slow adoption by
creating fear. None of it has anything to do with validity or actual
infringement.

You have about as much need for an actual infringement to file a US
patent law suit as to actually find chemical weapons in Iraq to start a
war.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Alan,

 > James reply was based on the fallacy that vendors will keep the source in
 > their tree. They don't do that *today*, look at their trees.

You keep talking about what is done about patents in other parts of
the free software world as though it is a good model to follow. It
isn't - it is a complete disaster.

The only weapon we have to fight patents right now is our collective
ability to find smart solutions to problems. The "every vendor for
themselves" approach that you seem to be so keen on makes that
collective approach pretty much impossible.

You talk about media players and other software being crippled in the
US and other countries as though this is a good thing. You talk about
vendors ripping out large slabs of functionality as though you want it
to happen.

I don't want that. I want to *fight*.

So how do we fight these and all the other patents that threaten
Linux? We implement the damn functionality anyway, finding a smart way
to make it work despite the patents. It has been so disheartening over
the years to see people say things like "oh, the XYZ codec is
patented, we can't use that". Patents don't work like that. They
patent very specific steps, and if you learn to read them properly
then you can find ways to get the functionality you need without just
applying an axe to a whole slab of useful features.

The patch I've posted is a good first step in showing the kernel
community how to do this. I'd like to see us apply the same approach
outside the kernel too. I'd like to see us produce implementations of
the 'patented' codecs that meet the same high standards for being
clearly non-infringing as the patch that I've posted for VFAT does.

I've been applying this approach in Samba for years, and it
*works*. Samba is extremely functional. We have not once had a user
complaint about a missing feature because of a patent workaround. How
do you reckon we achieved that? 

We, as a technical community, have the very best possible set of
resources for actually beating the patent system at its own game. Far
better than any company in the world by a very large margin.

We can also teach patent holders that filing a patent suit against the
Linux community is a _really_ bad idea. Patent holders will soon find
out that when they do that then there will quickly be a public
workaround to their patent posted. That public workaround can then be
used by not just the free software community, but also by the normal
proprietary vendors that these agressive patent holders make their
licensing money off.

So patent holders will learn that taking on the free software
community over patents loses them income from patent licensing. They
risk a huge, very talented, very motivated community of engineers
focussing its attention on their patent and finding a way around
it. They will learn that it is better to go after softer targets.

Short of a political miracle, that is the only way we are going to
beat the patent system. 

Let's stop whining about the unfair patent system and start fighting
back!

 > So why have the vendors not commented on list ? If they will want to
 > apply the patch why don't I see supporting email from them ?

I'm pretty sure you were at RedHat during a couple of patent
suits. What is the first thing that happened? I bet the lawyers told
everyone not to talk publicly about the patents.

That is standard practice within companies. How should we react to
that? We should react by helping them when they need the help. When we
need the help some day perhaps they will help us.

 > I can't comment on the advice I've seen directly, but I will point out
 > that every vendor I am aware of *removes completely* any source material
 > which they view with concern. You can download the packages and review
 > them if you doubt it.

oh yes, I've seen it, but unlike you I didn't jump with joy at the
sight of Linux users losing features because of the patent system. I
fumed at the fact that the vendors and projects involved did not have
the knowledge and skills to apply a more subtle, and much more
effective, approach to beating patents. 

We can teach each other those skills, and we can learn to beat
patents.

So please Alan, think about whether the approach that has been taken
for so long is working. Think about how it stops us using our brains
to solve problems that we're good at solving. Think about how much
worse things may get.

Then join me in trying to fight the patent system for real.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> You keep talking about what is done about patents in other parts of
> the free software world as though it is a good model to follow. It
> isn't - it is a complete disaster.

Its been a very successful disaster for many years then.

> The only weapon we have to fight patents right now is our collective
> ability to find smart solutions to problems. The "every vendor for
> themselves" approach that you seem to be so keen on makes that
> collective approach pretty much impossible.

The vendors can only talk to one another with lawyers present, so this is
an irrelevant argument to the public kernel. We've proved that already in
the fact you can't even discuss implementation details with the list.

> You talk about media players and other software being crippled in the
> US and other countries as though this is a good thing. You talk about
> vendors ripping out large slabs of functionality as though you want it
> to happen.

It is what the vendors will do regardless.

> to make it work despite the patents. It has been so disheartening over
> the years to see people say things like "oh, the XYZ codec is
> patented, we can't use that". Patents don't work like that. They
> patent very specific steps, and if you learn to read them properly
> then you can find ways to get the functionality you need without just
> applying an axe to a whole slab of useful features.

This misses the fact that if the vendor thinks sueing you is a business
opprtunity they will do so anyway. Why should they care if the patent is
valid - its not relevant to most of the legal process so it still has the
desired slowdown and expend money approach

> Let's stop whining about the unfair patent system and start fighting
> back!

How is shipping something labelled as vfat that doesn't work properly
"fighting back". It's "trying to make the users think our code is crap".

If you wanted reliable working approach surely tridgefat should read long
names but only create new short names ? That keeps you in spec, as I
understand it avoids the alleged possible patent infringement and works
for all devices ?

> oh yes, I've seen it, but unlike you I didn't jump with joy at the
> sight of Linux users losing features because of the patent system. I

I find the assumption that losing features to patents makes me "jump for
joy" rather offensive.

> fumed at the fact that the vendors and projects involved did not have
> the knowledge and skills to apply a more subtle, and much more
> effective, approach to beating patents. 

Many of the things removed are not patent related. Patents are only one
issue. Stuff gets removed for many reasons including a few cases of "they
sue everyone who does this regardless of validity so we don't XYZ"

> Then join me in trying to fight the patent system for real.

I've spent over ten years doing that. Those ten years have taught me

- if you give in to a patent thug they will simply try to bully you
  further
- if you limit functionality silently or in an unreliable way the blame
  falls on the software supplier not the patent holder or the governments
  who made the bad laws (or the judges of course as software patents
  in many states aren't really of political origin but judicial
  over-reach)
- if you delete stuff from base packages you hurt people in the rest of
  the world unneccessarily
- patents are a tiny part of the process anyone shipping any software
  commercially on a scale worth lawsuits goes through reviewing.

That is why I want to avoid any process which cripples the base kernel,
and anything which gives users broken software without them knowing it is
feature limited and why.

The patches you've proposed break stuff. It's mildly funny that MS is
trying to force us into a workaround that randomly crashes MS boxes but
that doesn't make it a good idea to merge such a change. Nor is it a good
idea to call something vfat which is not vfat.

This isn't about pain either - it is about users knowing what they are
getting.

So instead of regressions can we be a little bit less clever and a little
bit more reliable and do "FAT which reads and honours existing longnames"
and give it a name other than vfat.

That would give us absolute on media compatibility, accessibility from
all devices (which is the important bit) but some naming limits for new
file creation.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Alan,

 > The vendors can only talk to one another with lawyers present, so this is
 > an irrelevant argument to the public kernel. We've proved that already in
 > the fact you can't even discuss implementation details with the list.

We have discussed implementation details on this list. We've shown the
code, and a FAQ on how the code works. 

The situation is much more subtle with regard to what can be publicly
discussed than what you make out. Think of it like the media training
that many engineers get in companies these days. You need to learn
what words can and cannot be used. I think it is perfectly possible to
have public discussions of patents such as we are having right now. 

I grant you that it is very unusual to do this, but I would like to
see that change.

 > > You talk about media players and other software being crippled in the
 > > US and other countries as though this is a good thing. You talk about
 > > vendors ripping out large slabs of functionality as though you want it
 > > to happen.
 > 
 > It is what the vendors will do regardless.

no, when the vendors are given a clearly non-infringing way of
implementation then I think they will go for it. The proof is that
they still ship Samba, which includes patent workarounds, and I'm sure
there are plenty of other free software packages which similarly have
patent workarounds and that are shipped with distros all the time.

 > This misses the fact that if the vendor thinks sueing you is a business
 > opprtunity they will do so anyway. Why should they care if the patent is
 > valid - its not relevant to most of the legal process so it still has the
 > desired slowdown and expend money approach

you are mixing up "validity" and "non-infringment". If my argument was
in any way based on a patent being "invalid" then you would be right,
because a court is required to start from the assumption that a patent
is valid. Trying to fight a patent based on invalidity is really
really hard.

The court is not required to start with the assumption that a piece of
code infringes a patent - quite the opposite in fact. If the defendent
can show a clear argument as to why it is not infringing then the
patent holder quickly loses.

Do you know of any example where a defendent had a really good
non-infringment argument and still lost a case?

 > If you wanted reliable working approach surely tridgefat should read long
 > names but only create new short names ? That keeps you in spec, as I
 > understand it avoids the alleged possible patent infringement and works
 > for all devices ?

'works' for all devices, except that any Linux users copying a file
with a long name to the device get an error. That is not my idea of
'works'.

My patch in May proposed that solution because of it's simplicity, not
because it was the best solution. It was the first solution I thought
of, and I'm sure others thought of it too, but it loses so much
functionality that it would be a very painful solution for Linux
vendors.

The dualnames patch achieves a much higher degree of functionality for
a much wider number of users. It isn't perfect, but I think it is much
better.

 > Many of the things removed are not patent related. Patents are only one
 > issue. Stuff gets removed for many reasons including a few cases of "they
 > sue everyone who does this regardless of validity so we don't XYZ"

again, this is validity, not non-infringment. I am not advocating a
"you're patent is invalid" argument, as that is not certain enough. I
am advocating a "we don't implement that" approach.

 > I've spent over ten years doing that. Those ten years have taught me
 > 
 > - if you give in to a patent thug they will simply try to bully you
 >   further

and what did TomTom do on these patents? They had to give in, because
they didn't have a workaround. What has that taught the patent holders
of the world?

I'd like to teach patent holders that if you assert a patent against a
free software implementation of any bit of technology that you will
have a huge group of engineers working on stomping all over your
patent, thus threatening the nice little revenue stream you have from
other proprietary licensees.

We need to be seen as the guys with the big club who even the bullies
don't mess with. 

The great thing about this is that it works even better with patent
trolls. Their sole purpose in life is to derive revenue from their
patents. If we become known as the guys who threaten that income then
the trolls will give us a wide berth. 

This is an advantage that only the free software community has. If a
proprietary vendor finds a patent workaround then they will keep it
secret, as they don't want the other proprietary vendors to be able to
avoid the license fee. That means the trolls lose just one victim if a
proprietary vendor finds a workaround. With the free software
community our open code means that the trolls will be risking their
entire revenue stream from the patent by attacking any vendor based on
free software. That puts us in a uniquely strong position in the
patent world.

 > - if you limit functionality silently or in an unreliable way the blame
 >   falls on the software supplier not the patent holder or the governments
 >   who made the bad laws (or the judges of course as software patents
 >   in many states aren't really of political origin but judicial
 >   over-reach)

And if we make it so that copying files to a FAT filesystem return
-1/ENAMETOOLONG then what will that do to users? You really think
users will have sympathy with the situation then?

 > - if you delete stuff from base packages you hurt people in the rest of
 >   the world unneccessarily

that impacts only what the default should be, not whether the patch
should be in at all.

 > - patents are a tiny part of the process anyone shipping any software
 >   commercially on a scale worth lawsuits goes through reviewing.

so? For all the other parts of the process there are ways to manage
it. For patents, the solution of removing large amounts of
functionality is non-workable for many packages.

 > The patches you've proposed break stuff. It's mildly funny that MS is
 > trying to force us into a workaround that randomly crashes MS boxes but
 > that doesn't make it a good idea to merge such a change. Nor is it a good
 > idea to call something vfat which is not vfat.

and how is your preffered 'return -1/ENAMETOOLONG' for valid long
names 'vfat'? A new filesystem name could be used, but I think that is
largely cosmetic. How many Linux users, other than developers,
actually type:

  mount -t vfat .....

I think nearly all users just rely on the auto-detection of the fs
type. 

 > So instead of regressions can we be a little bit less clever and a little
 > bit more reliable and do "FAT which reads and honours existing longnames"
 > and give it a name other than vfat.
 > 
 > That would give us absolute on media compatibility, accessibility from
 > all devices (which is the important bit) but some naming limits for new
 > file creation.

It is the 'some naming limits' bit which is the problem. As was
pointed out when I posted the patch that did this in May, those limits
impact a whole lot of normal use cases.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> no, when the vendors are given a clearly non-infringing way of
> implementation then I think they will go for it. The proof is that

Even if they go for the non-infringing tweak they will still end up with
patches to rip out the potentially infringing code paths.

> 'works' for all devices, except that any Linux users copying a file
> with a long name to the device get an error. That is not my idea of
> 'works'.

It fails in a controlled manner and always produces valid output as
defined by the existing devices. At the end of the day it guarantees that
I can get my file back off the USB device on any system.

What is the fundamental purpose of a USB stick - to store and *transfer*
files. Long names are a (useful) luxury. Absolute certainty of
compatibility and reliability matter more for a file system.

> functionality that it would be a very painful solution for Linux
> vendors.

How about the users - is crashing XP, confusing Win98 and other suprises
on MP3 players not painful ?

> The dualnames patch achieves a much higher degree of functionality for
> a much wider number of users. It isn't perfect, but I think it is much
> better.

If it had controlled failure cases I would agree - but it doesn't. We are
still seeing bizarre failure cases and incompatibilities with the tiny
number of people currently testing it. Who knows what it does on some
random chinese market only mp3 player ?

> again, this is validity, not non-infringment. I am not advocating a
> "you're patent is invalid" argument, as that is not certain enough. I
> am advocating a "we don't implement that" approach.

Ok - that is important and I accept that.

> The great thing about this is that it works even better with patent
> trolls. Their sole purpose in life is to derive revenue from their
> patents. If we become known as the guys who threaten that income then
> the trolls will give us a wide berth. 

Agreed

>  > - if you limit functionality silently or in an unreliable way the blame
>  >   falls on the software supplier not the patent holder or the governments
>  >   who made the bad laws (or the judges of course as software patents
>  >   in many states aren't really of political origin but judicial
>  >   over-reach)
> 
> And if we make it so that copying files to a FAT filesystem return
> -1/ENAMETOOLONG then what will that do to users? You really think
> users will have sympathy with the situation then?

You'll get a discussion "Why can't I" "Because" "Gee that sucks how do I"

Users may not be great fans of it but they do understand whose fault it
is.

"You crashed my XP server" isn't a good basis for enlightenment nor is
"Where has my thesis gone"

> and how is your preffered 'return -1/ENAMETOOLONG' for valid long
> names 'vfat'? A new filesystem name could be used, but I think that is

I didn't say it was. It isn't vfat either.

> largely cosmetic. How many Linux users, other than developers,
> actually type:
> 
>   mount -t vfat .....
> 
> I think nearly all users just rely on the auto-detection of the fs
> type. 

Fine then giving it an accurate name won't trouble them too much.

> It is the 'some naming limits' bit which is the problem. As was
> pointed out when I posted the patch that did this in May, those limits
> impact a whole lot of normal use cases.

As clearly does this patch. But the May patch at least impacted them in a
controlled defined and predictable manner. We don't know the failure
cases and we can't predict them for this newer approach.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[James Bottomley]

On Thu, 2009-07-09 at 10:42 +0100, Alan Cox wrote:
> > no, when the vendors are given a clearly non-infringing way of
> > implementation then I think they will go for it. The proof is that
> 
> Even if they go for the non-infringing tweak they will still end up with
> patches to rip out the potentially infringing code paths.

You keep saying this and I keep pointing out to you it's not true.

Vendors would only rip the code out if they truly feared a suit for
contributory infringement based on shipping the code.  I don't actually
believe that a compile time option they turn on in the binary kernel to
make it non infringing coupled with shipping code where the user could
recompile with it off is sufficient to rise to the tests under the
contributory infringement doctrine.

Even if I'm wrong on this, vendors still won't rip out the code in this
case principally because this is a weak set of patents they universally
regard as invalid.  The way to troll with a weak patent is to go after
the small fish (namely end users) who can't sustain a business shutdown
under TRO or ITU actions.

The point of this patch is to alter the code sufficiently to forestall
troll actions against small redistributors or end users, not to shield
vendors from contributory infringement suits which would never get filed
because the patents are too weak for such a large scale attack.

James

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> > Even if they go for the non-infringing tweak they will still end up with
> > patches to rip out the potentially infringing code paths.
> 
> You keep saying this and I keep pointing out to you it's not true.

You keep saying this is not true and you keep ignoring the fact that a)
it is true and b) you can look at lots of existing RPM packages and see
that today. The evidence is out there - no hearsay, no question marks, go
look.

> contributory infringement based on shipping the code.  I don't actually
> believe that a compile time option they turn on in the binary kernel to
> make it non infringing coupled with shipping code where the user could
> recompile with it off is sufficient to rise to the tests under the
> contributory infringement doctrine.

Its a simple risk test. Anything which reduces risk but does not change
functionality is something you do. Its basically a zero cost way to
reduce the chances of being shot at.

The decision they have to make is full vfat v tridgefat v fat. Once you
make that decision and any part of that decision involves not enabling a
feature because of a theoretical concern about a dubious patent you rip
the code out of your tree as well. So if a vendor picked tridgefat they'd
rip out the source bits outside of the ifdefs that switched tridgefat v
vfat.

Alan

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Theodore Tso]

On Thu, Jul 09, 2009 at 03:10:09PM +0100, Alan Cox wrote:
> > contributory infringement based on shipping the code.  I don't actually
> > believe that a compile time option they turn on in the binary kernel to
> > make it non infringing coupled with shipping code where the user could
> > recompile with it off is sufficient to rise to the tests under the
> > contributory infringement doctrine.
> 
> Its a simple risk test. Anything which reduces risk but does not change
> functionality is something you do. Its basically a zero cost way to
> reduce the chances of being shot at.

Look, even if what you say is valid (despite the advice of lawyers),
it is still useful for us to apply the patch (with the CONFIG option)
in the upstream sources.  It means that support for the workaround
stays in the mainstream sources, so we don't have to worry about
separate patch going no longer applying as time goes by and the
upstream sources change over time.  *If* a vendor really wants to
strip out the source code, they can do that easily enough using
unifdef to strip out the one specific CONFIG option.

However, I don't think they will think it's necessary, and I'll tell
you why.  Regardless of whether or not source code with an #ifdef'ed
out (but still present) source code might be considered "contributory
infringment" (and we have legal advice saying that this would not be
the case), either way, the *binary*, which is to say the **product**
does not contain any of the infringing functionality or code.  Hence,
the patent troll won't be able to use an ITC action to stop the
product at the border, thus putting the potentially weak company out
of business.  Suing for contributory infringment takes time;
potentially years, so there is plenty of time for (a) the patent
troll, if it is a company like Microsoft, to suffer all sorts of
public relations damange, thus revealing the claims of people like Sam
Ramji as empty and cynical, and (b) for the community to rally around
said company and provide defense.  In addition, healing a cliam of
contributory infringment is easy enough; simply releasing a version of
the source code with unifdef applied won't changing the resulting
binary.   

In any case, the fact that the patent troll won't be able to put a
company out of business, but instead might have to wage a long legal
war (consider how long the SCO lawsuits have dragged on), will tend to
dissuade the rational troll from pursuing such a path; and even if we
do have an irrational actor, (a) I don't really think Microsoft is
irrational, and (b) in the U.S. you can get a lawyer to sue a ham
sandwich, so there's no real proactive steps you can take to protect
yourself against an irrational actor who is bound and determined to
abuse the legal system; and most companies and lawyers (as opposed to
people who like to play lawyers on mailing lists and TV) understand
that.

But in any case, even for a very risk-averse company, getting this
proposed patch into mainline is useful, since at any point in time the
company can get a version of the code without any code that might be
claimable as being infringing by using unifdef.  So it's still a net
win.  And if people are worried about the very small chances of
problems (which perhaps we can improve), we can fix that as future
patches against the mainline --- which is the right way to do OSS
development.  Given that Hirofumi-san has already decided to take this
patch, so unless Linus decides to override his decision, this
discussion is rapidly becoming moot in any case.

Regards,

						- Ted

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Christoph Hellwig]

On Thu, Jul 09, 2009 at 11:25:41AM -0400, Theodore Tso wrote:
> Look, even if what you say is valid (despite the advice of lawyers),
> it is still useful for us to apply the patch (with the CONFIG option)
> in the upstream sources.  It means that support for the workaround
> stays in the mainstream sources, so we don't have to worry about
> separate patch going no longer applying as time goes by and the
> upstream sources change over time.  *If* a vendor really wants to
> strip out the source code, they can do that easily enough using
> unifdef to strip out the one specific CONFIG option.

By putting it in the kernel tree we would only give the bogus patent
claims more credit.  And so far no one has but IBM has actually care
about the patch.  Tridge can put the patch into the IBM kernel tree for
the service processors if you really deeply care, but can we leave the
rest of the world alone?

If someone really wants a patch to corrupt their filesystems they know
where to find it.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[David Newall]

Christoph Hellwig wrote:
> If someone really wants a patch to corrupt their filesystems they know
> where to find it.
>   

No need for pettiness. Andrew's already intimated that he's still
working the patch, and he's a very clever lad and knows if it corrupts
it needs more work.

What I don't understand is how anybody could be satisfied with the
status quo. We cannot leave vfat unchanged, for that will perpetuate a
pool of victims to be sued, and Linux loses credibility every time that
happens. Something *must* change.

What is especially attractive about Andrew's position (he said this more
eloquently than me) is that developing a solution to avoid the patent
will impact Microsoft revenues; and that will be most instructive to
them. That's almost sufficient reason by itself!

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 4527 bytes --]

Am Donnerstag 09 Juli 2009 schrieb David Newall:
> Christoph Hellwig wrote:
> > If someone really wants a patch to corrupt their filesystems they
> > know where to find it.
>
> No need for pettiness. Andrew's already intimated that he's still
> working the patch, and he's a very clever lad and knows if it corrupts
> it needs more work.
>
> What I don't understand is how anybody could be satisfied with the
> status quo. We cannot leave vfat unchanged, for that will perpetuate a
> pool of victims to be sued, and Linux loses credibility every time that
> happens. Something *must* change.
>
> What is especially attractive about Andrew's position (he said this
> more eloquently than me) is that developing a solution to avoid the
> patent will impact Microsoft revenues; and that will be most
> instructive to them. That's almost sufficient reason by itself!

Pardon me, but I just don't get how adapting software to software patents 
will contribute into solving the problems they cause. Instead of 
implementing a feature like long name + short name support straight 
forwardly and simply one has to invent utterly complex, error prone and 
ugly work-arounds that actually even limit the functionality. Actually I 
do not envy Tridge for doing that job.

To me it seems that Microsoft has won if it can get Linux kernel 
developers to cripple the upstream vanilla kernel in order to avoid 
software patents. If it can get Linux kernel developers to accept a patch 
that if activated limits interoperatibility and compatibily with uncounted 
implementations of the one and only widely spread multiplatform and 
multidevice data exchange filesystem out there currently. If it can get 
Linux kernel developers to accept a patch that if activated actually risks 
more bugs and errors and thus makes the implementation less reliable than 
before.

To me it seems challenging the FAT standard by a modern replacement would 
still be the best way to go. Any hours wasted into a patch like 
CONFIG_VFAT_FS_DUALNAMES IMHO should be better spent with thinking about 
and implementing a replacement filesystem. Of course everyone is free to 
spend their time as they wish, but thats my oppinion.

The legal action of Microsoft against TomTom seems to create fear, 
uncertainity and doubts once again. And giving in to that IMHO would mean 
that Microsoft had already (partly) won.

Microsoft sued only TomTom regarding FAT patents upto now. Why? If they 
acted like SCO they would have gone against IBM, big Linux distributors 
like Novell and especially against several companies at once. I think this 
might be cause that Microsoft just knows that their software patent claim 
would break down if really tested. I do think that Microsoft does not want 
those FAT patents to be tested in court. Cause I think they know they 
would not stand a chance.

Accepting such a patch IMHO would help Microsoft to get away with seeding 
fear, uncertainity and doubt and not having the software patent tested and 
be made void. Actively adapting to software patents gives them a place to 
be, gives those who support them your energy.

Actually I think just ignoring them would be better. Don't give your 
energy to software patents. Ignore them and only defend were attacked 
unless the world sees how ridiculous they really are. TomTom can remove 
long name FAT from the Linux kernels they use to defend themselves if they 
can't stand the trial. This would be defending where necessary. Putting a 
patch in upstream Linux kernel would just be overreacting. Microsoft did 
not yet attack the upstream Linux kernel. And I think they won't. At least 
when they act rational. Thus IMHO there is no need to even think about 
adapting it.

And I think its not the job of the upstream kernel to protect companies 
that can not stand a patent trial or do not like to stand it. Its open-
source. They can remove parts of it.

Shall Microsoft attack IBM or other big companies. Shall Microsoft attack 
big Linux distributors. Shall they attack the upstream Linux kernel - 
however, I can't imagine an easy way to do that. Shall they ruin their 
image completely - the did quite well on that with Vista already. Don't 
help them not doing that. If Microsoft lawyers so desire let them go and 
make Microsoft a parody like lawyers of SCO managed to do.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[David Newall]

Martin Steigerwald wrote:
> Am Donnerstag 09 Juli 2009 schrieb David Newall:
>   
>> What I don't understand is how anybody could be satisfied with the
>> status quo. We cannot leave vfat unchanged, for that will perpetuate a
>> pool of victims to be sued, and Linux loses credibility every time that
>> happens. Something *must* change.
>>
>> What is especially attractive about Andrew's position (he said this
>> more eloquently than me) is that developing a solution to avoid the
>> patent will impact Microsoft revenues; and that will be most
>> instructive to them. That's almost sufficient reason by itself!
>>     
>
> Pardon me, but I just don't get how adapting software to software patents 
> will contribute into solving the problems they cause.

As Andrew said, if we work around a patent without losing function we
destroy the economic value of that patent.  Nobody would pay licence
fees for a patent if there was a good work around.  As patent holders
attack us, we devalue their patents.  Eventually (probably quickly)
they'll learn not to attack us.


> Instead of 
> implementing a feature like long name + short name support straight 
> forwardly and simply one has to invent utterly complex, error prone and 
> ugly work-arounds that actually even limit the functionality. Actually I 
> do not envy Tridge for doing that job.
>   

It's not a given that working around a patent will result in something
ugly, error-prone or complex, nor that it will limit function.  But it
is true that we have to work around them.  We can't permit Linux to
violate patents (nor can we permit a serious claim of such.)  That would
cause no end of legal trouble, and would drive vendors away.  Since vfat
violates Microsoft's patent, or at least they seriously claim it does,
we have to do something.  We really have no choice.

> To me it seems that Microsoft has won if it can get Linux kernel 
> developers to cripple the upstream vanilla kernel in order to avoid 
> software patents.

It's still not certain that we have to cripple Linux to work around the
long filename patent.  Granted it looks sad, but Andrew is still working
on it.  If it turns out that we do have to cripple Linux, well sometimes
we win, sometimes we lose; but if don't at least try we always lose. 
When we work around a patent without losing function we win big, and
Andrew said that has already happened for other patents.

> To me it seems challenging the FAT standard by a modern replacement would 
> still be the best way to go.

Perhaps, but FAT is entrenched everywhere so I doubt that challenge
would even be noticed.

> Microsoft sued only TomTom regarding FAT patents upto now. Why? If they 
> acted like SCO they would have gone against IBM, big Linux distributors 
> like Novell and especially against several companies at once. I think this 
> might be cause that Microsoft just knows that their software patent claim 
> would break down if really tested. I do think that Microsoft does not want 
> those FAT patents to be tested in court. Cause I think they know they 
> would not stand a chance.
>   

I think you're right that Microsoft does not want their patent tested in
court, but as they have more money than anyone, they could keep a patent
case in court forever, costing both them and those they sue more and
more money.  If the other party keeps fighting they stand a real chance
of running out of money (and thus out of business); or they could
settle, as Tom Tom did.

> Accepting such a patch IMHO would help Microsoft to get away with seeding 
> fear, uncertainity and doubt and not having the software patent tested and 
> be made void. Actively adapting to software patents gives them a place to 
> be, gives those who support them your energy.
>   
This is wrong.  Doing nothing is what helps Microsoft.

Working around their patent forces Microsoft to sue someone or else
tacitly accept that their patent has been bypassed.  If they sue, and
assuming the patent really has been bypassed, the courts can very
quickly determine that there is no violation, and then it becomes
explicit that the patent has been beaten.  Either way Microsoft would lose.


> Actually I think just ignoring them would be better.
This is also wrong.  Microsoft have sued Tom Tom, and they won't stop
there.  They'll keep picking businesses to sue; and each time they do
they'll win; and each time they win, Linux's reputation becomes
tarnished.  Eventually the manufacturers that build on Linux will move
to some other platform, which would be a disaster for us.

> And I think its not the job of the upstream kernel to protect companies 
> that can not stand a patent trial or do not like to stand it. Its open-
> source. They can remove parts of it.
>   
They can remove parts, and it's odd that Tom Tom didn't do that.  Surely
8.3 filenames would have met their needs, and perhaps they will meet our
needs, too, in which case Alan's suggestion would be appropriate, namely
that we use long filenames where they already exist, but create new
files with 8.3 only.  I could live with that, and I suspect Andrew
could, too.  But I think he'll look for a better solution until he
satisfies himself that none is there to be found.  I hope he find one.

> Shall Microsoft attack IBM or other big companies. Shall Microsoft attack 
> big Linux distributors. Shall they attack the upstream Linux kernel
>   

IBM?  Of course they would, and then Microsoft and IBM would
cross-licence their patents.  They've probably already done this.

Big Linux distributors?  I don't see Red Hat or Novel fighting, should
Microsoft sue.  They'd know the score, and either settle or remove the
function.

The upstream kernel developers?  I don't know.  They would if they felt
they needed to, but the truth is that they can do just as well by
attacking manufacturers who build on Linux, such as Tom Tom.  How many
more Tom Toms do you think it would take to drive manufacturers away
from Linux?  I don't think the number is large.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 10215 bytes --]

Am Freitag 10 Juli 2009 schrieb David Newall:
> Martin Steigerwald wrote:
> > Am Donnerstag 09 Juli 2009 schrieb David Newall:
> >> What I don't understand is how anybody could be satisfied with the
> >> status quo. We cannot leave vfat unchanged, for that will perpetuate
> >> a pool of victims to be sued, and Linux loses credibility every time
> >> that happens. Something *must* change.
> >>
> >> What is especially attractive about Andrew's position (he said this
> >> more eloquently than me) is that developing a solution to avoid the
> >> patent will impact Microsoft revenues; and that will be most
> >> instructive to them. That's almost sufficient reason by itself!
> >
> > Pardon me, but I just don't get how adapting software to software
> > patents will contribute into solving the problems they cause.
>
> As Andrew said, if we work around a patent without losing function we
> destroy the economic value of that patent.  Nobody would pay licence
> fees for a patent if there was a good work around.  As patent holders
> attack us, we devalue their patents.  Eventually (probably quickly)
> they'll learn not to attack us.

To me your thoughts appear to be *within* the patent approach and *thus* 
implicitely saying "yes" software patents as such. Going this way I think 
actually strengthen patents. The FAT patents have not yet been tested. 
They might easily just be void and invalid.

But going the way you outline would be giving in to the patent claim 
*before* it has even been tried and tested for validity. And thus IMHO 
this strengthens the patent and it holder. If you change the upstream 
kernel Microsoft can always say: "Look Linux kernel developers think that 
the kernel infringed our patents." 

The linux kernel has not yet been proven to infringe any FAT patent. 
Microsoft claims that it does. But so did SCO claim that it contains 
source of UNIX that they said they have a license for. SCO could never 
prove that claim. The linux kernel community ignored SCO for exactly that 
reason. Why on Earth should it handle Microsoft differently?  Just due to 
its sheer size and weight in capital? The trial with TomTom just proves 
nothing right now.

> > Instead of
> > implementing a feature like long name + short name support straight
> > forwardly and simply one has to invent utterly complex, error prone
> > and ugly work-arounds that actually even limit the functionality.
> > Actually I do not envy Tridge for doing that job.
>
> It's not a given that working around a patent will result in something
> ugly, error-prone or complex, nor that it will limit function.  But it
> is true that we have to work around them.  We can't permit Linux to
> violate patents (nor can we permit a serious claim of such.)  That
> would cause no end of legal trouble, and would drive vendors away. 
> Since vfat violates Microsoft's patent, or at least they seriously
> claim it does, we have to do something.  We really have no choice.

Right now to me it seems that it is nowhere but clear that Linux violates 
valid patents about the inner workings of FAT. Currently any work to adapt 
the kernel to the FAT patents is based on the assumption that they might 
be valid, not on facts.

> > To me it seems that Microsoft has won if it can get Linux kernel
> > developers to cripple the upstream vanilla kernel in order to avoid
> > software patents.
>
> It's still not certain that we have to cripple Linux to work around the
> long filename patent.  Granted it looks sad, but Andrew is still
> working on it.  If it turns out that we do have to cripple Linux, well
> sometimes we win, sometimes we lose; but if don't at least try we
> always lose. When we work around a patent without losing function we
> win big, and Andrew said that has already happened for other patents.

I just relate to what is currently available. If Tridge posts a different 
patch things might be different. But right now this is just pure 
speculation. So I would like to stick to what is actually there. This 
patch.

> > Microsoft sued only TomTom regarding FAT patents upto now. Why? If
> > they acted like SCO they would have gone against IBM, big Linux
> > distributors like Novell and especially against several companies at
> > once. I think this might be cause that Microsoft just knows that
> > their software patent claim would break down if really tested. I do
> > think that Microsoft does not want those FAT patents to be tested in
> > court. Cause I think they know they would not stand a chance.
>
> I think you're right that Microsoft does not want their patent tested
> in court, but as they have more money than anyone, they could keep a
> patent case in court forever, costing both them and those they sue more
> and more money.  If the other party keeps fighting they stand a real
> chance of running out of money (and thus out of business); or they
> could settle, as Tom Tom did.

I think you omit that doing this would cost Microsoft really lots of 
reputation. I believe that Microsoft fears testing the patent in trial and 
having a long trial just as much if not more than the company it sues.

Would you buy a Unix or something else from SCO? I wouldn't. Would a 
company do it? I think only if forced to by compatibility constraints. Now 
SCO OpenServer or what it was called may be more replaceable than Windows, 
but Windows has become replaceable in more and more use cases as well.

If Microsoft would be serious about playing the bad guy they IMHO would 
pay a forbiddenly high price for it. Actually I believe this would impose 
a high, probably existential risk to Microsoft. Yes, Microsoft has lots of 
money. But the current economic crisis has shown how easily insane amounts 
of money can be disintegrated in no time. I do think that Microsoft is 
neither almighty nor immune to that.

And then they would also show even more clearly than ever the absurdities 
of the current software patent law in the USA. This would strengthen 
forces that work to revert that law. More politicians would see that the 
software patent law must be changed.

> > Accepting such a patch IMHO would help Microsoft to get away with
> > seeding fear, uncertainity and doubt and not having the software
> > patent tested and be made void. Actively adapting to software patents
> > gives them a place to be, gives those who support them your energy.
>
> This is wrong.  Doing nothing is what helps Microsoft.

I can't continue aguing on that basis. I just stated my oppinion. You say 
it is wrong. I can say your oppinion is wrong. But nothing gained by that. 
No need to continue to cycle this. My oppinion is my oppinion and yours is 
yours. And I don't buy your oppinion even when you present it as factual 
thing.

> > Actually I think just ignoring them would be better.
>
> This is also wrong.  Microsoft have sued Tom Tom, and they won't stop

Same here. Just:

> there.  They'll keep picking businesses to sue; and each time they do
> they'll win; and each time they win, Linux's reputation becomes
> tarnished.  Eventually the manufacturers that build on Linux will move
> to some other platform, which would be a disaster for us.

You seem to know more than me. Sure I can speculate about what Microsoft 
will be doing and I did. But I am pretty sure that I can not really *know* 
it. Your statements sounds bold enough to suggest that you have looked 
into a crystal orb of foresay. Did you?

I just point out to you: They didn't do anything else than suing TomTom 
just yet. Thats fact, unless I did not notice any other trials going on.

> > Shall Microsoft attack IBM or other big companies. Shall Microsoft
> > attack big Linux distributors. Shall they attack the upstream Linux
> > kernel
>
> IBM?  Of course they would, and then Microsoft and IBM would
> cross-licence their patents.  They've probably already done this.

You really think they would? Have they been already that successful at 
seeding fear, uncertainity and doubt? I bet that they would not dare to 
challenge IBM on such ridiculous patents.

> Big Linux distributors?  I don't see Red Hat or Novel fighting, should
> Microsoft sue.  They'd know the score, and either settle or remove the
> function.

Sure? What about indemnification assurances regarding SCO claims that 
AFAIK both Novell and RedHat offered to their customers? SCO is smaller 
than Microsoft, granted. But still, those patents, the whole software 
patent law in USA are ridiculous. Giving in to them would probably loose 
Novell and RedHat more than is gained by testing them and having them 
declared as invalid and void as they should be declared.

> The upstream kernel developers?  I don't know.  They would if they felt
> they needed to, but the truth is that they can do just as well by
> attacking manufacturers who build on Linux, such as Tom Tom.  How many
> more Tom Toms do you think it would take to drive manufacturers away
> from Linux?  I don't think the number is large.

Directly attacking upstream kernel developers IMHO would backfire even 
more than suing companies. Do you really think Microsoft can risk to loose 
that much reputation? Do you really think that Microsoft is ready to 
handle the response of the open-source community, big news sites, 
companies supporting open-source, lawyers supporting open-source? I think 
they aren't and I think they know that. And thus I think they would not 
dare to do it. Right know they only had a go at TomTom. To me this looks 
like the behavior of coward. Apart from I think Linux including all its 
surroundings isn't that weak that it needs to fear Microsoft. I think 
right know its more the other way around.

Granted, speculation, but IMHO a quite plausible one. Anyway I will end it 
here. And as I think there is not much more to be said without recycling 
arguments I will try to refrain from any comments on arguments that have 
already been rised.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jonathan Corbet]

On Fri, 10 Jul 2009 20:49:00 +0200
Martin Steigerwald <Martin@lichtvoll.de> wrote:

> To me your thoughts appear to be *within* the patent approach and *thus* 
> implicitely saying "yes" software patents as such. Going this way I think 
> actually strengthen patents. 

One deals with reality on reality's terms.  For the moment, software
patents exist in some parts of the world.  People are working on that,
but one cannot just stick one's fingers into one's ears and wish the
problem away.

> The FAT patents have not yet been tested. 
> They might easily just be void and invalid.

The FAT patents *have* been tested, invalidated, and then revalidated.
Yes, they might still be invalid - I believe they are.  This helps a
Linux user whose products have been seized at the US border exactly
how? 

> But going the way you outline would be giving in to the patent claim 
> *before* it has even been tried and tested for validity. And thus IMHO 
> this strengthens the patent and it holder. If you change the upstream 
> kernel Microsoft can always say: "Look Linux kernel developers think that 
> the kernel infringed our patents." 

The point is not to give in to the claim; the point is to make it
irrelevant.  That "invalidates" the patent in a way which (1) doesn't
cost several million dollars and (2) is rather more certain in its
outcome.

> The linux kernel has not yet been proven to infringe any FAT patent. 
> Microsoft claims that it does. But so did SCO claim that it contains 
> source of UNIX that they said they have a license for. SCO could never 
> prove that claim. The linux kernel community ignored SCO for exactly that 
> reason. 

The Linux community did not ignore SCO.  We looked very hard at our
code and our processes, and adopted things like the DCO very much in
response to SCO.  The whole SCO affair has, IMO, made us a lot
stronger.

SCO is a very different story, though.  Copyright infringement is
usually pretty easy to avoid - just do your own work.  Patent
infringement happens regardless of how original your work is.

> Why on Earth should it handle Microsoft differently?  Just due to 
> its sheer size and weight in capital? The trial with TomTom just proves 
> nothing right now.

The TomTom trial proves that companies like TomTom have no real hope of
withstanding patent attacks and nicely invalidating obnoxious patents
as a favor for the rest of us.

Perhaps (a future version of) this patch isn't the right solution to
the VFAT patent problem.  Be we do need to figure out how we can
respond to patents which are being used to attack our users.  Telling
them to take the bullet and try to invalidate the patent for us isn't
going to fly.  Neither is ignoring the problem.

I have a lot of sympathy for Alan's assertion that we can't muck up our
upstream code with hackarounds for every bit of legal obnoxiousness we
encounter.  But outright refusal of things like patent workarounds does
not help us either.  Linux is a pragmatic system; somehow, I believe,
we can find a way to minimize our patent hassles without messing up the
system as a whole.

jon

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Bartlomiej Zolnierkiewicz]

On Friday 10 July 2009 21:31:27 Jonathan Corbet wrote:

> I have a lot of sympathy for Alan's assertion that we can't muck up our
> upstream code with hackarounds for every bit of legal obnoxiousness we
> encounter.  But outright refusal of things like patent workarounds does
> not help us either.  Linux is a pragmatic system; somehow, I believe,
> we can find a way to minimize our patent hassles without messing up the
> system as a whole.

Alan, Pavel and other people are suggesting pragmatic *long-term* solutions,
like moving kernel.org outside problematic geopolitical area.  Tainting *our*
code-base with (confusing for *our* users) workarounds doesn't belong in this
class of solutions.. actually no.. it is a PURE F*CKING IDIOCY..

Microsoft having patents on their *obsolete* filesystem should be *their*
problem, not *ours*.  We keep said filesystem strictly for compatibility
reasons and not because we would like to encourage hardware/software vendors
or users to use it.  Lets just update VFAT help text to mention potential
risks of using it, default it to N, rip it from defconfigs, pass it through
lawyers, remove CONFIG_VFAT_FS_DUALNAMES and move on..

When it comes to distributions if they really need they can easily deal with
the problem by simply allowing the 3-rd parties to host the repositories with
risky package (like they do with mp3 etc).

The end result is that end-users can still use full vfat, vendors are safe,
kernel developers don't waste their time on artificial problems and the blame
goes where it belong..

[ The validity of potentially bogus patents (or problems of hardware vendors
  caused by *their* decisions to use *obsolete* and patent-risky filesystem in
  their products) is an entirely different matter/discussion and it shouldn't
  be mixed with technical solutions.. ]

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jörn Engel]

[ Ignoring all legal and moral aspects...]

On Fri, 10 July 2009 22:40:14 +0200, Bartlomiej Zolnierkiewicz wrote:
> 
> Microsoft having patents on their *obsolete* filesystem should be *their*

FAT is far from obsolete.  It is practically always the filesystem of
choice and often the only filesystem when trying to move data from one
system to another.  The next best alternatives are isofs and ext2.  And
I don't know a single digital camera, mp3 player or cellphone that
speaks either.

Jörn

-- 
Joern's library part 5:
http://www.faqs.org/faqs/compression-faq/part2/section-9.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jörn Engel]

[ Ignoring all legal and moral aspects...]

On Fri, 10 July 2009 22:40:14 +0200, Bartlomiej Zolnierkiewicz wrote:
> 
> Microsoft having patents on their *obsolete* filesystem should be *their*

FAT is far from obsolete.  It is practically always the filesystem of
choice and often the only filesystem when trying to move data from one
system to another.  The next best alternatives are isofs and ext2.  And
I don't know a single digital camera, mp3 player or cellphone that
speaks either.

Jörn

-- 
Joern's library part 5:
http://www.faqs.org/faqs/compression-faq/part2/section-9.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Sunday 2009-07-12 13:21, Jörn Engel wrote:
>[ Ignoring all legal and moral aspects...]
>
>On Fri, 10 July 2009 22:40:14 +0200, Bartlomiej Zolnierkiewicz wrote:
>> 
>> Microsoft having patents on their *obsolete* filesystem should be *their*
>
>FAT is far from obsolete.  It is practically always the filesystem of
>choice and often the only filesystem when trying to move data from one
>system to another.  The next best alternatives are isofs and ext2.  And
>I don't know a single digital camera, mp3 player or cellphone that
>speaks either.

The next best would probably be UDF, which is already used on DVDs
and thus is implemented in a number of devices - though probably
only disc-reading ones.
There's your market hole, dear vendors.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Sunday 2009-07-12 13:21, Jörn Engel wrote:
>[ Ignoring all legal and moral aspects...]
>
>On Fri, 10 July 2009 22:40:14 +0200, Bartlomiej Zolnierkiewicz wrote:
>> 
>> Microsoft having patents on their *obsolete* filesystem should be *their*
>
>FAT is far from obsolete.  It is practically always the filesystem of
>choice and often the only filesystem when trying to move data from one
>system to another.  The next best alternatives are isofs and ext2.  And
>I don't know a single digital camera, mp3 player or cellphone that
>speaks either.

The next best would probably be UDF, which is already used on DVDs
and thus is implemented in a number of devices - though probably
only disc-reading ones.
There's your market hole, dear vendors.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

Jan Engelhardt wrote:
> 
> On Sunday 2009-07-12 13:21, Jörn Engel wrote:
> >[ Ignoring all legal and moral aspects...]
> >
> >On Fri, 10 July 2009 22:40:14 +0200, Bartlomiej Zolnierkiewicz wrote:
> >> 
> >> Microsoft having patents on their *obsolete* filesystem should be *their*
> >
> >FAT is far from obsolete.  It is practically always the filesystem of
> >choice and often the only filesystem when trying to move data from one
> >system to another.  The next best alternatives are isofs and ext2.  And
> >I don't know a single digital camera, mp3 player or cellphone that
> >speaks either.
> 
> The next best would probably be UDF, which is already used on DVDs
> and thus is implemented in a number of devices - though probably
> only disc-reading ones.
> There's your market hole, dear vendors.

UDF has already been mentioned on this thread, next to "looks usable
in theory, now try it and find out it isn't".

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

Jan Engelhardt wrote:
> 
> On Sunday 2009-07-12 13:21, Jörn Engel wrote:
> >[ Ignoring all legal and moral aspects...]
> >
> >On Fri, 10 July 2009 22:40:14 +0200, Bartlomiej Zolnierkiewicz wrote:
> >> 
> >> Microsoft having patents on their *obsolete* filesystem should be *their*
> >
> >FAT is far from obsolete.  It is practically always the filesystem of
> >choice and often the only filesystem when trying to move data from one
> >system to another.  The next best alternatives are isofs and ext2.  And
> >I don't know a single digital camera, mp3 player or cellphone that
> >speaks either.
> 
> The next best would probably be UDF, which is already used on DVDs
> and thus is implemented in a number of devices - though probably
> only disc-reading ones.
> There's your market hole, dear vendors.

UDF has already been mentioned on this thread, next to "looks usable
in theory, now try it and find out it isn't".

-- Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Tuesday 2009-07-14 00:20, Jamie Lokier wrote:
>> The next best would probably be UDF, which is already used on DVDs
>> and thus is implemented in a number of devices - though probably
>> only disc-reading ones.
>> There's your market hole, dear vendors.
>
>UDF has already been mentioned on this thread, next to "looks usable
>in theory, now try it and find out it isn't".
>

Sorry if I was showing a lack of enthusiasm discussing, let alone
reading, the legalese subthreads.
Please do not expect everybody to follow LKML as close as akpm/jcm.

The fact that people even dropped the mailing list from Cc on a
subthread effectively rendered my ~/.procmailrc sort-to-folder rules
useless so that I just decided to hit Thread-Delete to get rid of it
from the master INBOX.

-jeng

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> The FAT patents *have* been tested, invalidated, and then revalidated.
> Yes, they might still be invalid - I believe they are.  This helps a
> Linux user whose products have been seized at the US border exactly
> how? 

The same way as one who shipped mp3 software.

Of course they can simply remove the code that concerns them as companies
*already do*. They have specialists for this, experts who know the very
complex field of import and export regulation.

> SCO is a very different story, though.  Copyright infringement is
> usually pretty easy to avoid - just do your own work.  Patent
> infringement happens regardless of how original your work is.

Actually this is changing, large corporations stealing photographic
images and the threatening to sue the original photographer is becoming
rather common. I'm sure the same will happen in other fields too. SCO
probably did it by accident, but folks will do it deliberately too.

> I have a lot of sympathy for Alan's assertion that we can't muck up our
> upstream code with hackarounds for every bit of legal obnoxiousness we
> encounter.  But outright refusal of things like patent workarounds does
> not help us either.  Linux is a pragmatic system; somehow, I believe,
> we can find a way to minimize our patent hassles without messing up the
> system as a whole.

If you don't believe VFAT is permitted in your shipping locale you turn
it off (actually you cut the code from your source tar before build).
Utterly routine practice every day for companies shipping product into
the USA. Happens with tons of other open source software that isn't
permitted in the USA.

The pragmatic approach would be to either avoid the patent without
breaking stuff (which Tridge hasn't managed, although maybe one day he
might), or to produce something which doesn't break stuff, is reliable,
meets our quality goals but which is alternative and clearly presented as
an alternative - such as Tridge's shortname creating tridgefat variant,
which at least didn't randomly explode users data in new and novel ways.
It still needs to be an alternative not a replacement for the real thing.

Alan

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[David Newall]

Martin Steigerwald wrote:
> The FAT patents have not yet been tested. 
> They might easily just be void and invalid.
>   

It might be invalid, but it would take a lot of money to get just the US
Patent invalidated.  Since it's probably also patented in Australia, UK,
Japan, Germany and so on, you'd need to invalidate it there, too, to
wipe out the problem, and a US decision to revoke wouldn't help much.

Don't worry about whether the patent is valid, because we can't fight
that.  Tom Tom chose not to fight the patent, which let's you know how
big we could be and still be unable to fight it.  Instead, think of ways
to side-step it.



> But going the way you outline would be giving in to the patent claim 
> *before* it has even been tried and tested for validity. And thus IMHO 
> this strengthens the patent and it holder. If you change the upstream 
> kernel Microsoft can always say: "Look Linux kernel developers think that 
> the kernel infringed our patents." 
>   

Working around a patent in no way strengthens the patent claims.  Even
if it did, if we come up with an alternative that gives long and short
file names without going through the steps that Microsoft described in
their patent, then we wouldn't be violating their patent.  Remember
this: Microsoft did not patent a file system with long and short file
names.  I understand that is not even a patentable matter.  Microsoft
patented a *method* for creating long and short files, and if we don't
follow their method then their patent doesn't cover what we do.

Let's find a solution that Microsoft haven't patented, and then give
that to the world.  While it might still look like a duck, walk like a
duck and quack like a duck, we need only put the feathers on *before*
the feet, and then it isn't a duck under patent law.


>> I think you're right that Microsoft does not want their patent tested
>> in court, but as they have more money than anyone, they could keep a
>> patent case in court forever, costing both them and those they sue more
>> and more money.  If the other party keeps fighting they stand a real
>> chance of running out of money (and thus out of business); or they
>> could settle, as Tom Tom did.
>>     
>
> I think you omit that doing this would cost Microsoft really lots of 
> reputation. I believe that Microsoft fears testing the patent in trial and 
> having a long trial just as much if not more than the company it sues.
>   

You think Microsoft fear testing in court, yet we know otherwise because
they already did initiate action against Tom Tom.  Think what you like,
but I will stick, unimaginative thought it might be, to the real world.

> Would you buy a Unix or something else from SCO?
Let me put it this way: going back only a few years, we had no choice
over who provided local calls.  Even though almost everybody agreed they
were expensive, unfriendly and disinterested in their customers' needs,
we still bought from them.


One thing I suspect none of us would deny is that the patent system is
seriously flawed.  Sadly, that's completely irrelevant.  It's long been
accepted that the Law is an ass, but it's still the law and if you
disobey it it will slap you so hard and quickly your head will spin.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> However, I don't think they will think it's necessary, and I'll tell
> you why.  Regardless of whether or not source code with an #ifdef'ed

They do it for existing cases in existing packages. So I'd say the
evidence is very very clearly that they will and they do, already, now,
at this moment. No speculation - go look at the packages.

> the case), either way, the *binary*, which is to say the **product**
> does not contain any of the infringing functionality or code.  Hence,

You have a GPL obligation to supply the sources used to build the binary.

> But in any case, even for a very risk-averse company, getting this
> proposed patch into mainline is useful, since at any point in time the
> company can get a version of the code without any code that might be
> claimable as being infringing by using unifdef.  So it's still a net
> win.  And if people are worried about the very small chances of

That aspect of the argument makes sense and I would agree with you. Its
one way to keep a reference implementation in tree as a patch.

> problems (which perhaps we can improve), we can fix that as future
> patches against the mainline --- which is the right way to do OSS
> development.  Given that Hirofumi-san has already decided to take this
> patch, so unless Linus decides to override his decision, this
> discussion is rapidly becoming moot in any case.

I've not seen any evidence to support this, and there seem to be other fs
maintainers deeply unhappy with the longname corrupting patch - Christoph
included.

Alan

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Wednesday 2009-07-08 12:48, tridge@samba.org wrote:
>
> - some vendors may decide not to use Linux, and switch to embedded
>   windows instead if we don't have a clearly supported way of
>   avoiding these patents in the Linux kernels.

I perceive that:
Linux is primarily about doing things the right way, not about making
the largest userbase (it is a welcomed side-effect though).

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[James Bottomley]

On Wed, 2009-07-08 at 11:04 +0100, Alan Cox wrote:
> > that it should be configurable, but I don't think it leads to the
> > conclusion that the patch should not be in the upstream kernel at all.
> 
> I think we already proved it had no use upstream. Vendors will remove
> the code from their source tree if worried about patents so including it
> in the base tree is really irrelevant. So I find your argument about this
> less than convincing.

You have asserted such, but that's not proof.  If your assertion were
valid, vendors would already have removed all the msdos/vfat code, which
they haven't.

Obviously I have no idea what your ex-employer will do. I also can't
speak for Novell, but very likely what we'll do is leave the decision
for OpenSUSE up to the OpenSUSE community and likely follow the kernel
default for SLE.

> The patch serves no purpose but to confuse users and increasingly it is
> shown to break systems horribly.
> 
> There might have been a limited case for a not-quite-vfat-fs "tridgefat"
> etc if it was both more compatible and if the vendors would use the
> option. But given its not compatible and vendors won't why bother at all ?
> 
> I also note you keep talking about vendors. This is an open list yet I
> don't hear a word from the vendors you claim to represent in support of
> this patch set, and saying they would enable it. Not one voice seems to
> have appeared.

Why would vendors wish to comment?  Their position universally is that
the FAT32 patents are invalid.  However, they also recognise that
trolling with invalid patents is increasingly becoming a nasty problem
for their customers, and with TomTom Microsoft has shown willing to do
this, so anything that lowers the risk and potential costs to customers
would be a good thing for vendors.

> The decision sequence goes something like this
> 
> 	- do we want to ship the feature because of patent concern
> 	> do not ship
> 	- is it less risk to remove the source from our build tree or
> 	  configure it out
> 	> remove from the tree
> 	- is there a functionality difference to the user between
> 	  removing or unconfiguring it
> 	> No
> 
> At that point nobody managing risk is going to do anything but remove the
> code that worries them. It's additional risk with no return.

I think you might be confusing two sources of risk.  The risk of
actually infringing a real patent is what you're covering above.  The
source of risk in this case is the risk of being trolled with an invalid
patent but have to spend millions of dollars to demonstrate such if it
goes to trial.  The patch mitigates the latter risk by making it
demonstrable at a preliminary hearing that the invalid patent doesn't
read upon the kernel implementation.

James

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> > I think we already proved it had no use upstream. Vendors will remove
> > the code from their source tree if worried about patents so including it
> > in the base tree is really irrelevant. So I find your argument about this
> > less than convincing.
> 
> You have asserted such, but that's not proof.  If your assertion were
> valid, vendors would already have removed all the msdos/vfat code, which
> they haven't.

That represents loss of functionality for loss of risk (a trade off). If
they are going to disable stuff then they'll pull the lot (loss of risk
for free). Remember the GPL requires you provide the *sources* you used
to create the binaries, not redacted ones so if you don't want to provide
feature X for some reason you need to pull it out of your source tree
before you build.

> > 	- do we want to ship the feature because of patent concern
> > 	> do not ship
> > 	- is it less risk to remove the source from our build tree or
> > 	  configure it out
> > 	> remove from the tree
> > 	- is there a functionality difference to the user between
> > 	  removing or unconfiguring it
> > 	> No
> > 
> > At that point nobody managing risk is going to do anything but remove the
> > code that worries them. It's additional risk with no return.
> 
> I think you might be confusing two sources of risk.  The risk of
> actually infringing a real patent is what you're covering above.  The
> source of risk in this case is the risk of being trolled with an invalid
> patent but have to spend millions of dollars to demonstrate such if it
> goes to trial.  The patch mitigates the latter risk by making it
> demonstrable at a preliminary hearing that the invalid patent doesn't
> read upon the kernel implementation.

At that point we get into legal advice I can't comment on in public.
However based on previous discussions about source code and other patterns
suffice to say I don't agree.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[James Bottomley]

On Wed, 2009-07-08 at 16:37 +0100, Alan Cox wrote:
> > > I think we already proved it had no use upstream. Vendors will remove
> > > the code from their source tree if worried about patents so including it
> > > in the base tree is really irrelevant. So I find your argument about this
> > > less than convincing.
> > 
> > You have asserted such, but that's not proof.  If your assertion were
> > valid, vendors would already have removed all the msdos/vfat code, which
> > they haven't.
> 
> That represents loss of functionality for loss of risk (a trade off).

Precisely ... that's what this whole thread is about.  Accepting the
lawyer's argument that this patch eliminates the risk of suit under the
FAT32 patents, does the loss of functionality it comes with represent an
adequate reward?

>  If
> they are going to disable stuff then they'll pull the lot (loss of risk
> for free).

I'd agree with that, the problem is the loss of functionality ... and
actually, the best outcome for this would be to come up with a true
light weight filesystem that could replace the rather crap DOS/FAT32 one
in embedded applications based on fully free source and no patent
encumbrances and have it adopted as a universal standard for flash/USB
stick/mmc interchange.

> Remember the GPL requires you provide the *sources* you used
> to create the binaries, not redacted ones so if you don't want to provide
> feature X for some reason you need to pull it out of your source tree
> before you build.

OK, so we disagree on whether providing source code constitutes
contributory infringement.

> > > 	- do we want to ship the feature because of patent concern
> > > 	> do not ship
> > > 	- is it less risk to remove the source from our build tree or
> > > 	  configure it out
> > > 	> remove from the tree
> > > 	- is there a functionality difference to the user between
> > > 	  removing or unconfiguring it
> > > 	> No
> > > 
> > > At that point nobody managing risk is going to do anything but remove the
> > > code that worries them. It's additional risk with no return.
> > 
> > I think you might be confusing two sources of risk.  The risk of
> > actually infringing a real patent is what you're covering above.  The
> > source of risk in this case is the risk of being trolled with an invalid
> > patent but have to spend millions of dollars to demonstrate such if it
> > goes to trial.  The patch mitigates the latter risk by making it
> > demonstrable at a preliminary hearing that the invalid patent doesn't
> > read upon the kernel implementation.
> 
> At that point we get into legal advice I can't comment on in public.
> However based on previous discussions about source code and other patterns
> suffice to say I don't agree.

OK, so rather than get into a what my lawyer says versus what your
lawyer says argument, can we get back to the technical aspects of this
and leave the lawyers retained by the vendors to sort out what they
actually do?

James

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> > That represents loss of functionality for loss of risk (a trade off).
> 
> Precisely ... that's what this whole thread is about.  Accepting the
> lawyer's argument that this patch eliminates the risk of suit under the
> FAT32 patents, does the loss of functionality it comes with represent an
> adequate reward?

You are missing the point still.

Ship with problem feature enabled	risk high    features high
Ship with problem off but in source	risk lower   features low
Ship with it ripped out			risk lowest  features low

So the only two logical positions to operate are low feature/lowest risk
and high feature/high risk. So if you decide not to ship the feature it'll
get ripped out entirely.

> OK, so we disagree on whether providing source code constitutes
> contributory infringement.

We disagree whether there is zero risk involved. If there is any risk
then the logical position if you've decided not to enable the feature is
not to ship source for it.

> OK, so rather than get into a what my lawyer says versus what your
> lawyer says argument, can we get back to the technical aspects of this
> and leave the lawyers retained by the vendors to sort out what they
> actually do?

Sure and I'd say the technical issues are simple

- Tridge's patch breaks stuff
- Tridge's patch masquerades as vfat but isn't.

We can fix those by only creating short names (but honouring existing
long ones) and by not claiming its vfat.

Alan

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Alan,

Can you explain what standard you think should be applied to patent
workaround patches for them to be acceptable? I'd like to know if
there is the possibility of us finding some agreement in the future or
not.

For example, some possibilities might be:

  1) no patent workarounds allowed in upstream kernel at all

  2) the workaround must be shown to have 100% compatibility with all
  current and possible future devices

  3) the workaround must be shown to have a high degree of
  compatibility with all the devices we have available to test with

  4) the workaround must have the highest degree of compatibility that
  we can achieve with the most used devices, but some degree of
  interoperability problems are OK for less used devices.

There are lots of possible levels in between these of course. I don't
think you are advocating (1) or (2), as you seem happier with the "no
long name creation" patch from May.

I also know you want whatever is done to be a different filesystem
name. 

I'm advocating (4) as a reasonable standard, although I'd like to
achieve (3) if we can. Whether we can actually achieve (3) will depend
on the results of further testing (see my messages to Jan on that for
example).

I apologise if you don't think this is a reasonable way to phrase the
question. As you are the most vocal opponent of the patch, I'd like to
better understand what it would take for you to find a patch
acceptable.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[OGAWA Hirofumi]

tridge@samba.org writes:

> Can you explain what standard you think should be applied to patent
> workaround patches for them to be acceptable? I'd like to know if
> there is the possibility of us finding some agreement in the future or
> not.

You are talking different thing than patch. Please stop it.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Pavel Machek]

On Thu 2009-07-09 14:27:39, OGAWA Hirofumi wrote:
> tridge@samba.org writes:
> 
> > Can you explain what standard you think should be applied to patent
> > workaround patches for them to be acceptable? I'd like to know if
> > there is the possibility of us finding some agreement in the future or
> > not.
> 
> You are talking different thing than patch. Please stop it.

Uff, the patch is obviously trash -- takes working code, replaces it
with known broken code -- so some justification is neccessary.

Outside of U.S., the patch makes obviously no sense.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[David Newall]

Pavel Machek wrote:
> On Thu 2009-07-09 14:27:39, OGAWA Hirofumi wrote:
>   
>> You are talking different thing than patch. Please stop it.
>>     
>
> Outside of U.S., the patch makes obviously no sense.

I think it does make sense outside of the USA.  Australia's previous
Prime Minister, Bonsai, signed a free trade agreement with Warmerica and
therefore the patent should be considered an active threat there, too.

While the patch seems too flawed, Andrew's approach, namely looking to
work around the patent, is sound.  In fact it's more than sound, it's a
brilliant attack for the reasons he stated.  Given the negative attitude
displayed towards Andrew's approach, I think it's appropriate for him to
get the policy clarified before he wastes any more of his time on the
technical issues.

I applaud his efforts in this regard.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> Can you explain what standard you think should be applied to patent
> workaround patches for them to be acceptable? I'd like to know if
> there is the possibility of us finding some agreement in the future or
> not.
> 
> For example, some possibilities might be:
> 
>   1) no patent workarounds allowed in upstream kernel at all

I have no problem with working around alleged patents

>   2) the workaround must be shown to have 100% compatibility with all
>   current and possible future devices

If we have a workaround that is truely compatible with stuff and has no
real impact then I am all for applying it.

>   3) the workaround must be shown to have a high degree of
>   compatibility with all the devices we have available to test with

IFF the workaround can be turned off for non problem parts of the world
IFF we can define what the failure models are

That bit is critical

>   4) the workaround must have the highest degree of compatibility that
>   we can achieve with the most used devices, but some degree of
>   interoperability problems are OK for less used devices.

At this point we get into last resorts. I don't believe unpredictable
interoperability problems are acceptable, especially for file systems.

To give an example of the reverse case, a video decoder for some web
sites that only played most but not all content of that format wouldn't
worry me. Why - because the failure case is defined ("Sorry can't play
that because of patents" dialogue box) and there isn't a real risk of
loss.

> There are lots of possible levels in between these of course. I don't
> think you are advocating (1) or (2), as you seem happier with the "no
> long name creation" patch from May.

I am. For the simple reason that

	cp importantstuff.doc /media/wibble
	umount; eject

go to random other system and access the document has to be a reliable
process and it is far better to get the error copying (and use a
shortname) than to arrive at the other end of an eight hour flight to
find your document is invisible on the recipient system. Hence the focus
on definable the failure case. We wouldn't apply an fs patch that
randomly vanished files from some systems, and you wouldn't apply a SAMBA
patch that made documents vanish from some systems without apparent logic.

> I apologise if you don't think this is a reasonable way to phrase the
> question. As you are the most vocal opponent of the patch, I'd like to
> better understand what it would take for you to find a patch
> acceptable.

I want to find an elegant solution to this, that is reliable, safe for
users and doesn't mislead. If asking multi-choice questions like that
helps keeps going the right way carry on.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[OGAWA Hirofumi]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> Sure and I'd say the technical issues are simple
>
> - Tridge's patch breaks stuff
> - Tridge's patch masquerades as vfat but isn't.
>
> We can fix those by only creating short names (but honouring existing
> long ones) and by not claiming its vfat.

As technical stand, I agree with this approach. And my poor brain can't
consider other than technical thing, it is purely my problem though. So
I'll try to create the patch based on first version, and I'll apply it
instead.

And could you please stop talking about other than technical issues? My
poor brain is already overflowed.

I'm a fool?  Yes, I agree.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Alan Cox]

> And could you please stop talking about other than technical issues? My
> poor brain is already overflowed.

Sorry but the two really do overlap in this case.

> I'm a fool?  Yes, I agree.

You are no fool. I certainly could not hold a debate about patents and
ethics in a second language.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[OGAWA Hirofumi]

OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:

> Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
>
>> Sure and I'd say the technical issues are simple
>>
>> - Tridge's patch breaks stuff
>> - Tridge's patch masquerades as vfat but isn't.
>>
>> We can fix those by only creating short names (but honouring existing
>> long ones) and by not claiming its vfat.
>
> As technical stand, I agree with this approach. And my poor brain can't
> consider other than technical thing, it is purely my problem though. So
> I'll try to create the patch based on first version, and I'll apply it
> instead.
>
> And could you please stop talking about other than technical issues? My
> poor brain is already overflowed.
>
> I'm a fool?  Yes, I agree.

My patch is following. Sorry. But I'm still worrying about this myself.

It seems my skill and knowledge for the issue are too small.  My time
for fat was already small before this, and honestly, I don't think I can
maintain/support all issues well.

So, finally, I thought the best for us which I can is to find new
maintainer for fat.

Could you help to find the new maintainer?

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>



From: tridge@samba.org

When this option is enabled this will refuse to create new files with
long names. Accessing existing files with long names will continue to
work.

File names to be created must conform to the 8.3 format.  Mixed case is
not allowed in either the prefix or the suffix.

Signed-off-by: Andrew Tridgell <tridge@samba.org>
[Add "lfat", comment out Kconfig]
Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
---

 fs/fat/Kconfig      |   11 +++++++++++
 fs/fat/namei_vfat.c |   35 ++++++++++++++++++++++++++++++-----
 2 files changed, 41 insertions(+), 5 deletions(-)

diff -puN fs/fat/Kconfig~fat-disable-longname fs/fat/Kconfig
--- fatfs-2.6-2/fs/fat/Kconfig~fat-disable-longname	2009-07-10 16:16:47.000000000 +0900
+++ fatfs-2.6-2-hirofumi/fs/fat/Kconfig	2009-07-12 12:50:52.000000000 +0900
@@ -98,3 +98,14 @@ config FAT_DEFAULT_IOCHARSET
 
 	  Enable any character sets you need in File Systems/Native Language
 	  Support.
+
+#config VFAT_NO_CREATE_WITH_LONGNAMES
+#	bool "Disable creating files with long names"
+#	depends on VFAT_FS
+#	default n
+#	help
+#	  Set this to disable support for creating files or directories with
+#	  names longer than 8.3 (the original DOS maximum file name length)
+#	  e.g. naming a file FILE1234.TXT would be allowed but creating or
+#	  renaming a file to FILE12345.TXT or FILE1234.TEXT would not
+#	  be permitted.  Reading files with long file names is still permitted.
diff -puN fs/fat/namei_vfat.c~fat-disable-longname fs/fat/namei_vfat.c
--- fatfs-2.6-2/fs/fat/namei_vfat.c~fat-disable-longname	2009-07-10 16:16:47.000000000 +0900
+++ fatfs-2.6-2-hirofumi/fs/fat/namei_vfat.c	2009-07-11 16:26:24.000000000 +0900
@@ -316,6 +316,7 @@ static int vfat_create_shortname(struct 
 	int sz = 0, extlen, baselen, i, numtail_baselen, numtail2_baselen;
 	int is_shortname;
 	struct shortname_info base_info, ext_info;
+	unsigned opt_shortname = opts->shortname;
 
 	is_shortname = 1;
 	INIT_SHORTNAME_INFO(&base_info);
@@ -424,13 +425,22 @@ static int vfat_create_shortname(struct 
 	memcpy(name_res, base, baselen);
 	memcpy(name_res + 8, ext, extlen);
 	*lcase = 0;
+
+#ifdef CONFIG_VFAT_NO_CREATE_WITH_LONGNAMES
+	if (is_shortname == 0)
+		return -ENAMETOOLONG;
+	if (!base_info.valid || !ext_info.valid)
+		return -EINVAL;
+	opt_shortname = VFAT_SFN_CREATE_WINNT;
+#endif
+
 	if (is_shortname && base_info.valid && ext_info.valid) {
 		if (vfat_find_form(dir, name_res) == 0)
 			return -EEXIST;
 
-		if (opts->shortname & VFAT_SFN_CREATE_WIN95) {
+		if (opt_shortname & VFAT_SFN_CREATE_WIN95) {
 			return (base_info.upper && ext_info.upper);
-		} else if (opts->shortname & VFAT_SFN_CREATE_WINNT) {
+		} else if (opt_shortname & VFAT_SFN_CREATE_WINNT) {
 			if ((base_info.upper || base_info.lower) &&
 			    (ext_info.upper || ext_info.lower)) {
 				if (!base_info.upper && base_info.lower)
@@ -593,15 +603,19 @@ static int vfat_build_slots(struct inode
 {
 	struct msdos_sb_info *sbi = MSDOS_SB(dir->i_sb);
 	struct fat_mount_options *opts = &sbi->options;
-	struct msdos_dir_slot *ps;
 	struct msdos_dir_entry *de;
-	unsigned char cksum, lcase;
+	unsigned char lcase;
 	unsigned char msdos_name[MSDOS_NAME];
 	wchar_t *uname;
 	__le16 time, date;
 	u8 time_cs;
-	int err, ulen, usize, i;
+	int err, ulen, usize;
+#ifndef CONFIG_VFAT_NO_CREATE_WITH_LONGNAMES
+	int i;
+	struct msdos_dir_slot *ps;
+	unsigned char cksum;
 	loff_t offset;
+#endif
 
 	*nr_slots = 0;
 
@@ -628,6 +642,9 @@ static int vfat_build_slots(struct inode
 		goto shortname;
 	}
 
+#ifdef CONFIG_VFAT_NO_CREATE_WITH_LONGNAMES
+	de = (struct msdos_dir_entry *)slots;
+#else
 	/* build the entry of long file name */
 	cksum = fat_checksum(msdos_name);
 
@@ -645,6 +662,7 @@ static int vfat_build_slots(struct inode
 	}
 	slots[0].id |= 0x40;
 	de = (struct msdos_dir_entry *)ps;
+#endif
 
 shortname:
 	/* build the entry of 8.3 alias name */
@@ -1074,7 +1092,11 @@ static int vfat_get_sb(struct file_syste
 
 static struct file_system_type vfat_fs_type = {
 	.owner		= THIS_MODULE,
+#ifdef CONFIG_VFAT_NO_CREATE_WITH_LONGNAMES
+	.name		= "lfat",
+#else
 	.name		= "vfat",
+#endif
 	.get_sb		= vfat_get_sb,
 	.kill_sb	= kill_block_super,
 	.fs_flags	= FS_REQUIRES_DEV,
@@ -1093,6 +1115,9 @@ static void __exit exit_vfat_fs(void)
 MODULE_LICENSE("GPL");
 MODULE_DESCRIPTION("VFAT filesystem support");
 MODULE_AUTHOR("Gordon Chaffee");
+#ifdef CONFIG_VFAT_NO_CREATE_WITH_LONGNAMES
+MODULE_ALIAS("lfat");
+#endif
 
 module_init(init_vfat_fs)
 module_exit(exit_vfat_fs)
_

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Thanks to everyone who helped with the testing of the previous round
of VFAT patent workaround patches.

I've posted a new set of patches today which tries to address some of
the technical problems found in the last patch.

The new patches:

  - work with Win98
  - work with Jan's IOneIt MP3 player
  - work with all the other FAT capable devices I have available for
    testing
  - work with existing copies of mtools

The remaining issues that I am aware of are:

  - There is a cosmetic issue with the DOS command prompt under
    Win98. A directory listing works, but displays garbage in the
    column where a 8.3 short filename would normally go

  - Similarly, under WinXP, a "dir/x" will show garbage in the 8.3
    column. For example: http://samba.org/~tridge/dir_test.png

  - mtools has a similar cosmetic issue, which is fixed with a small
    patch

  - devices which only support 8.3 filenames will not be able to see
    or use files created with long names with the patch enabled

  - There is a very small chance of WinXP bluescreening if two files
    in the same directory have the same 11 dummy bytes, and are
    accessed in quick succession. The chances of this happening is
    approximately 80x smaller than with the previous patch. As
    previously noted, this is a very difficult problem to reproduce,
    and in fact nobody has managed to reproduce it without modifying
    the patch to use a much smaller number of random bits.

  - Similarly, there is a small chance that chkdsk on Windows will
    rename one file in a directory if they happen to have the same 11
    byte dummy values. The probability of this happening is
    approximately 80x lower than with the previous patch.

Some people have also asked that this patch change the name of the
filesystem to 'lfat' or similar. I have not included that change in
this patch as I think it is counter productive. Instead I have added a
printk_once() to produce a warning like this:

  VFAT: not creating 8.3 short filenames for long names

when the first long filename is created on a VFAT filesystem with this
patch enabled.

The reason I think this is a better option than a filesystem name
change is that a name change will break a unknown number of userspace
tools, scripts and config files. For example, desktop tools for
mounting filesystems, scripts that use -t vfat, module configuration
options in /etc could all be broken without any ability to give the
user feedback on why it broke.

If you have a FAT capable device that you want to test for
compatibility with the new patches, please have a look at the
'Testing' section of the following README:

  http://kernel.org/pub/linux/kernel/people/tridge/VFAT/README

It gives details on how you can do testing without having to change
your kernel.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Boaz Harrosh]

On 07/21/2009 06:37 AM, tridge@samba.org wrote:
> Thanks to everyone who helped with the testing of the previous round
> of VFAT patent workaround patches.
> 
> I've posted a new set of patches today which tries to address some of
> the technical problems found in the last patch.
> 
> The new patches:
> 
>   - work with Win98
>   - work with Jan's IOneIt MP3 player
>   - work with all the other FAT capable devices I have available for
>     testing
>   - work with existing copies of mtools
> 
> The remaining issues that I am aware of are:
> 
>   - There is a cosmetic issue with the DOS command prompt under
>     Win98. A directory listing works, but displays garbage in the
>     column where a 8.3 short filename would normally go
> 
>   - Similarly, under WinXP, a "dir/x" will show garbage in the 8.3
>     column. For example: http://samba.org/~tridge/dir_test.png
> 

I guess you tried putting a zero at first char and it breaks everybody?

>   - mtools has a similar cosmetic issue, which is fixed with a small
>     patch
> 
>   - devices which only support 8.3 filenames will not be able to see
>     or use files created with long names with the patch enabled
> 
>   - There is a very small chance of WinXP bluescreening if two files
>     in the same directory have the same 11 dummy bytes, and are
>     accessed in quick succession. The chances of this happening is
>     approximately 80x smaller than with the previous patch. As
>     previously noted, this is a very difficult problem to reproduce,
>     and in fact nobody has managed to reproduce it without modifying
>     the patch to use a much smaller number of random bits.
> 

I guess (35^6)*8*7 is not that bad

>   - Similarly, there is a small chance that chkdsk on Windows will
>     rename one file in a directory if they happen to have the same 11
>     byte dummy values. The probability of this happening is
>     approximately 80x lower than with the previous patch.
> 

What if we had a user mode utility that does these short-names
renames that a user can optionally run after umount? since it
only writes the (random) short-names it's also safe.

Kind of the cop that can read and the cop that can write e-literacy
problem, No?

> Some people have also asked that this patch change the name of the
> filesystem to 'lfat' or similar. I have not included that change in
> this patch as I think it is counter productive. Instead I have added a
> printk_once() to produce a warning like this:
> 
>   VFAT: not creating 8.3 short filenames for long names
> 
> when the first long filename is created on a VFAT filesystem with this
> patch enabled.
> 
> The reason I think this is a better option than a filesystem name
> change is that a name change will break a unknown number of userspace
> tools, scripts and config files. For example, desktop tools for
> mounting filesystems, scripts that use -t vfat, module configuration
> options in /etc could all be broken without any ability to give the
> user feedback on why it broke.
> 
> If you have a FAT capable device that you want to test for
> compatibility with the new patches, please have a look at the
> 'Testing' section of the following README:
> 
>   http://kernel.org/pub/linux/kernel/people/tridge/VFAT/README
> 
> It gives details on how you can do testing without having to change
> your kernel.
> 
> Cheers, Tridge
> --

Boaz

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Pavel Machek]

> >   - Similarly, there is a small chance that chkdsk on Windows will
> >     rename one file in a directory if they happen to have the same 11
> >     byte dummy values. The probability of this happening is
> >     approximately 80x lower than with the previous patch.
> > 
> 
> What if we had a user mode utility that does these short-names
> renames that a user can optionally run after umount? since it
> only writes the (random) short-names it's also safe.

Actually, why not having dosfsck creating _matching_ short names for
long names? As it only writes short names, it should be safe :-).
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Pavel,

 > Actually, why not having dosfsck creating _matching_ short names for
 > long names? As it only writes short names, it should be safe :-).

As I mentioned to Boaz, what usually matters for claim construction in
patents is the overall system. I'm sure John Lanza would be happy to
run through it with you if you want a more legally detailed answer,
but basically breaking it up within the one system isn't likely to
help us.

Like I mentioned to Boaz though, do keep thinking about it. There may
well be a better solution that nobody has suggested yet. You might
also like to read the file wrapper (availble on the USPTO site) which
shows the discussions between the patent office and the
applicant. That is often a good source of inspiration for patent
workarounds. If you think you've found something then it might be a
good idea to raise it with John Lanza first.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[John Lanza]

All:

I think the email I just sent applies here, as well.  And I echo
Tridge's sentiments about giving me a shout if you have some idea for
qorkarounds.  I'm here to be a sounding board for you.

johnl

On Tue, Jul 21, 2009 at 9:24 AM, <tridge@samba.org> wrote:
> Hi Pavel,
>
>  > Actually, why not having dosfsck creating _matching_ short names for
>  > long names? As it only writes short names, it should be safe :-).
>
> As I mentioned to Boaz, what usually matters for claim construction in
> patents is the overall system. I'm sure John Lanza would be happy to
> run through it with you if you want a more legally detailed answer,
> but basically breaking it up within the one system isn't likely to
> help us.
>
> Like I mentioned to Boaz though, do keep thinking about it. There may
> well be a better solution that nobody has suggested yet. You might
> also like to read the file wrapper (availble on the USPTO site) which
> shows the discussions between the patent office and the
> applicant. That is often a good source of inspiration for patent
> workarounds. If you think you've found something then it might be a
> good idea to raise it with John Lanza first.
>
> Cheers, Tridge
>

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[John Lanza]

All:

I think the email I just sent applies here, as well.  And I echo
Tridge's sentiments about giving me a shout if you have some idea for
qorkarounds.  I'm here to be a sounding board for you.

johnl

On Tue, Jul 21, 2009 at 9:24 AM, <tridge@samba.org> wrote:
> Hi Pavel,
>
>  > Actually, why not having dosfsck creating _matching_ short names for
>  > long names? As it only writes short names, it should be safe :-).
>
> As I mentioned to Boaz, what usually matters for claim construction in
> patents is the overall system. I'm sure John Lanza would be happy to
> run through it with you if you want a more legally detailed answer,
> but basically breaking it up within the one system isn't likely to
> help us.
>
> Like I mentioned to Boaz though, do keep thinking about it. There may
> well be a better solution that nobody has suggested yet. You might
> also like to read the file wrapper (availble on the USPTO site) which
> shows the discussions between the patent office and the
> applicant. That is often a good source of inspiration for patent
> workarounds. If you think you've found something then it might be a
> good idea to raise it with John Lanza first.
>
> Cheers, Tridge
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[John Lanza]

Resending in plain text..........

On Tue, Jul 21, 2009 at 11:08 AM, John Lanza <jdlanza@gmail.com> wrote:
>
> All:
>
> I think the email I just sent applies here, as well.  And I echo
> Tridge's sentiments about giving me a shout if you have some idea for
> qorkarounds.  I'm here to be a sounding board for you.
>
> johnl
>
> On Tue, Jul 21, 2009 at 9:24 AM, <tridge@samba.org> wrote:
> > Hi Pavel,
> >
> >  > Actually, why not having dosfsck creating _matching_ short names for
> >  > long names? As it only writes short names, it should be safe :-).
> >
> > As I mentioned to Boaz, what usually matters for claim construction in
> > patents is the overall system. I'm sure John Lanza would be happy to
> > run through it with you if you want a more legally detailed answer,
> > but basically breaking it up within the one system isn't likely to
> > help us.
> >
> > Like I mentioned to Boaz though, do keep thinking about it. There may
> > well be a better solution that nobody has suggested yet. You might
> > also like to read the file wrapper (availble on the USPTO site) which
> > shows the discussions between the patent office and the
> > applicant. That is often a good source of inspiration for patent
> > workarounds. If you think you've found something then it might be a
> > good idea to raise it with John Lanza first.
> >
> > Cheers, Tridge
> >

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[John Lanza]

Resending in plain text..........

On Tue, Jul 21, 2009 at 11:08 AM, John Lanza <jdlanza@gmail.com> wrote:
>
> All:
>
> I think the email I just sent applies here, as well.  And I echo
> Tridge's sentiments about giving me a shout if you have some idea for
> qorkarounds.  I'm here to be a sounding board for you.
>
> johnl
>
> On Tue, Jul 21, 2009 at 9:24 AM, <tridge@samba.org> wrote:
> > Hi Pavel,
> >
> >  > Actually, why not having dosfsck creating _matching_ short names for
> >  > long names? As it only writes short names, it should be safe :-).
> >
> > As I mentioned to Boaz, what usually matters for claim construction in
> > patents is the overall system. I'm sure John Lanza would be happy to
> > run through it with you if you want a more legally detailed answer,
> > but basically breaking it up within the one system isn't likely to
> > help us.
> >
> > Like I mentioned to Boaz though, do keep thinking about it. There may
> > well be a better solution that nobody has suggested yet. You might
> > also like to read the file wrapper (availble on the USPTO site) which
> > shows the discussions between the patent office and the
> > applicant. That is often a good source of inspiration for patent
> > workarounds. If you think you've found something then it might be a
> > good idea to raise it with John Lanza first.
> >
> > Cheers, Tridge
> >
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Pavel Machek]

Hi!

>  > Actually, why not having dosfsck creating _matching_ short names for
>  > long names? As it only writes short names, it should be safe :-).
> 
> As I mentioned to Boaz, what usually matters for claim construction in
> patents is the overall system. I'm sure John Lanza would be happy to
> run through it with you if you want a more legally detailed answer,
> but basically breaking it up within the one system isn't likely to
> help us.

Hehe, or we could create just long names and mark filesystem as
"needing repair" so that Windows fixes it up for us on next mount ;-).
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Pavel,

 > Hehe, or we could create just long names and mark filesystem as
 > "needing repair" so that Windows fixes it up for us on next mount ;-).

Adding short names on the windows side would be a valid patent
workaround I believe, but I'm not sure that forcing a chkdsk is a very
useful way to do that. The current windows chkdsk chooses .--N as the
filename, for integer N. It doesn't base the added short name on the
long name.

Paul and I also looked into the option of adding a autorun file of
some sort when files are written on the Linux side, so that a windows
tool adds 8.3 names when the media is inserted. It doesn't turn out to
be very practical however, partly because so many windows systems
don't have auto-run enabled and it looks like MS is disabling it in
future versions. This solution also doesn't work for all the other FAT
devices out there (MP3 players, photo frames etc).

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Tuesday 2009-07-21 11:16, Boaz Harrosh wrote:
>>   - Similarly, there is a small chance that chkdsk on Windows will
>>     rename one file in a directory if they happen to have the same 11
>>     byte dummy values. The probability of this happening is
>>     approximately 80x lower than with the previous patch.
>
>What if we had a user mode utility that does these short-names
>renames that a user can optionally run after umount? since it
>only writes the (random) short-names it's also safe.

Mm, user-mode. tridgefat could trigger a usermode process at open(2)
time to rebuild the short name with traditional semantics to counter
the 'breakage' induced by random 8.3 names.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Boaz,

 > I guess you tried putting a zero at first char and it breaks everybody?

It works with some devices, but with many it doesn't. A space followed
by a nul works with quite a lot of devices, but not enough (the last
patch used a space followed by a nul).

I went to a large electronics store and told them I wanted to buy
devices that didn't work with my computer. They were very helpful, and
as a result I was able to test a lot of devices. That is what led to
the design of this patch (plus the feedback from people like Jan and
his IOneIt MP3 player).

 > I guess (35^6)*8*7 is not that bad

yes, but luckily For the WinXP bluescreen the probability of the crash
is actually much lower than that figure would give. With the same
modelling assumptions of WinXP memory slots for 8.3 entries that Paul
used for the last patch, it comes out as less than a 1 in 10k chance
for a full directory (ie. 32767 long filenames). For 100 files in a
directory it is around 1 chance in 10^11. I'm sure Paul will do the
full expansion and modelling if anyone wants more precise numbers.

For the chkdsk rename, the probability is much easier to calculate as
it is just the usual birthday expansion (see wikipedia for simple
formula for that). That is what gives 0.5% for 32767 files in a
directory, and 4.8x10^-8 for for 100 files.

Basically it won't happen very often. In each case the probability is
rougly 75x less than it was for the last patch.

 > What if we had a user mode utility that does these short-names
 > renames that a user can optionally run after umount? since it
 > only writes the (random) short-names it's also safe.

While I will defer to John Lanza if you want a more complete legal
view on this, I think it is likely that separating the steps of the
patent between programs within one system is not a safe enough legal
strategy to be used.

Please do keep thinking about it though. There could well be some
simple combination which is legally safe and also technically
completely satisfactory. If you think you have hit on a winner, you
may wish to discuss it with John Lanza in private first though, so it
can be fine tuned before being presented publicly.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[John Lanza]

If I understand the proposal (which I think I do), Tridge is correct.
If a patent prohibits a system from performing steps "A" and "B",
simply separating the steps into separate modules, or utilities, won't
avoid infringement.

I'm happy to answer specific questions, but it might be best to do
that separately from lkml.

johnl

On Tue, Jul 21, 2009 at 9:04 AM, <tridge@samba.org> wrote:
> Hi Boaz,
>
>  > I guess you tried putting a zero at first char and it breaks everybody?
>
> It works with some devices, but with many it doesn't. A space followed
> by a nul works with quite a lot of devices, but not enough (the last
> patch used a space followed by a nul).
>
> I went to a large electronics store and told them I wanted to buy
> devices that didn't work with my computer. They were very helpful, and
> as a result I was able to test a lot of devices. That is what led to
> the design of this patch (plus the feedback from people like Jan and
> his IOneIt MP3 player).
>
>  > I guess (35^6)*8*7 is not that bad
>
> yes, but luckily For the WinXP bluescreen the probability of the crash
> is actually much lower than that figure would give. With the same
> modelling assumptions of WinXP memory slots for 8.3 entries that Paul
> used for the last patch, it comes out as less than a 1 in 10k chance
> for a full directory (ie. 32767 long filenames). For 100 files in a
> directory it is around 1 chance in 10^11. I'm sure Paul will do the
> full expansion and modelling if anyone wants more precise numbers.
>
> For the chkdsk rename, the probability is much easier to calculate as
> it is just the usual birthday expansion (see wikipedia for simple
> formula for that). That is what gives 0.5% for 32767 files in a
> directory, and 4.8x10^-8 for for 100 files.
>
> Basically it won't happen very often. In each case the probability is
> rougly 75x less than it was for the last patch.
>
>  > What if we had a user mode utility that does these short-names
>  > renames that a user can optionally run after umount? since it
>  > only writes the (random) short-names it's also safe.
>
> While I will defer to John Lanza if you want a more complete legal
> view on this, I think it is likely that separating the steps of the
> patent between programs within one system is not a safe enough legal
> strategy to be used.
>
> Please do keep thinking about it though. There could well be some
> simple combination which is legally safe and also technically
> completely satisfactory. If you think you have hit on a winner, you
> may wish to discuss it with John Lanza in private first though, so it
> can be fine tuned before being presented publicly.
>
> Cheers, Tridge
>

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[John Lanza]

Resending in plain text.......

On Tue, Jul 21, 2009 at 11:06 AM, John Lanza<jdlanza@gmail.com> wrote:
> If I understand the proposal (which I think I do), Tridge is correct.
> If a patent prohibits a system from performing steps "A" and "B",
> simply separating the steps into separate modules, or utilities, won't
> avoid infringement.
>
> I'm happy to answer specific questions, but it might be best to do
> that separately from lkml.
>
> johnl
>
> On Tue, Jul 21, 2009 at 9:04 AM, <tridge@samba.org> wrote:
>> Hi Boaz,
>>
>>  > I guess you tried putting a zero at first char and it breaks everybody?
>>
>> It works with some devices, but with many it doesn't. A space followed
>> by a nul works with quite a lot of devices, but not enough (the last
>> patch used a space followed by a nul).
>>
>> I went to a large electronics store and told them I wanted to buy
>> devices that didn't work with my computer. They were very helpful, and
>> as a result I was able to test a lot of devices. That is what led to
>> the design of this patch (plus the feedback from people like Jan and
>> his IOneIt MP3 player).
>>
>>  > I guess (35^6)*8*7 is not that bad
>>
>> yes, but luckily For the WinXP bluescreen the probability of the crash
>> is actually much lower than that figure would give. With the same
>> modelling assumptions of WinXP memory slots for 8.3 entries that Paul
>> used for the last patch, it comes out as less than a 1 in 10k chance
>> for a full directory (ie. 32767 long filenames). For 100 files in a
>> directory it is around 1 chance in 10^11. I'm sure Paul will do the
>> full expansion and modelling if anyone wants more precise numbers.
>>
>> For the chkdsk rename, the probability is much easier to calculate as
>> it is just the usual birthday expansion (see wikipedia for simple
>> formula for that). That is what gives 0.5% for 32767 files in a
>> directory, and 4.8x10^-8 for for 100 files.
>>
>> Basically it won't happen very often. In each case the probability is
>> rougly 75x less than it was for the last patch.
>>
>>  > What if we had a user mode utility that does these short-names
>>  > renames that a user can optionally run after umount? since it
>>  > only writes the (random) short-names it's also safe.
>>
>> While I will defer to John Lanza if you want a more complete legal
>> view on this, I think it is likely that separating the steps of the
>> patent between programs within one system is not a safe enough legal
>> strategy to be used.
>>
>> Please do keep thinking about it though. There could well be some
>> simple combination which is legally safe and also technically
>> completely satisfactory. If you think you have hit on a winner, you
>> may wish to discuss it with John Lanza in private first though, so it
>> can be fine tuned before being presented publicly.
>>
>> Cheers, Tridge
>>
>

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[John Lanza]

Resending in plain text.......

On Tue, Jul 21, 2009 at 11:06 AM, John Lanza<jdlanza@gmail.com> wrote:
> If I understand the proposal (which I think I do), Tridge is correct.
> If a patent prohibits a system from performing steps "A" and "B",
> simply separating the steps into separate modules, or utilities, won't
> avoid infringement.
>
> I'm happy to answer specific questions, but it might be best to do
> that separately from lkml.
>
> johnl
>
> On Tue, Jul 21, 2009 at 9:04 AM, <tridge@samba.org> wrote:
>> Hi Boaz,
>>
>>  > I guess you tried putting a zero at first char and it breaks everybody?
>>
>> It works with some devices, but with many it doesn't. A space followed
>> by a nul works with quite a lot of devices, but not enough (the last
>> patch used a space followed by a nul).
>>
>> I went to a large electronics store and told them I wanted to buy
>> devices that didn't work with my computer. They were very helpful, and
>> as a result I was able to test a lot of devices. That is what led to
>> the design of this patch (plus the feedback from people like Jan and
>> his IOneIt MP3 player).
>>
>>  > I guess (35^6)*8*7 is not that bad
>>
>> yes, but luckily For the WinXP bluescreen the probability of the crash
>> is actually much lower than that figure would give. With the same
>> modelling assumptions of WinXP memory slots for 8.3 entries that Paul
>> used for the last patch, it comes out as less than a 1 in 10k chance
>> for a full directory (ie. 32767 long filenames). For 100 files in a
>> directory it is around 1 chance in 10^11. I'm sure Paul will do the
>> full expansion and modelling if anyone wants more precise numbers.
>>
>> For the chkdsk rename, the probability is much easier to calculate as
>> it is just the usual birthday expansion (see wikipedia for simple
>> formula for that). That is what gives 0.5% for 32767 files in a
>> directory, and 4.8x10^-8 for for 100 files.
>>
>> Basically it won't happen very often. In each case the probability is
>> rougly 75x less than it was for the last patch.
>>
>>  > What if we had a user mode utility that does these short-names
>>  > renames that a user can optionally run after umount? since it
>>  > only writes the (random) short-names it's also safe.
>>
>> While I will defer to John Lanza if you want a more complete legal
>> view on this, I think it is likely that separating the steps of the
>> patent between programs within one system is not a safe enough legal
>> strategy to be used.
>>
>> Please do keep thinking about it though. There could well be some
>> simple combination which is legally safe and also technically
>> completely satisfactory. If you think you have hit on a winner, you
>> may wish to discuss it with John Lanza in private first though, so it
>> can be fine tuned before being presented publicly.
>>
>> Cheers, Tridge
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Pavel Machek]

>   - Similarly, there is a small chance that chkdsk on Windows will
>     rename one file in a directory if they happen to have the same 11
>     byte dummy values. The probability of this happening is
>     approximately 80x lower than with the previous patch.

The "small chance" seems to be 90% for 30000 files in directory. And
no, it is probably not 80x lower. Do the math.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Pavel,

 > >   - Similarly, there is a small chance that chkdsk on Windows will
 > >     rename one file in a directory if they happen to have the same 11
 > >     byte dummy values. The probability of this happening is
 > >     approximately 80x lower than with the previous patch.
 > 
 > The "small chance" seems to be 90% for 30000 files in directory. And
 > no, it is probably not 80x lower. Do the math.

Perhaps you could double check my math? 

The number of combinations available with the current patch is 
35^6 * 7 * 8 for files (for directories there are more combinations, 
but lets ignore that for now).

That comes to 102942875000 combinations.

Now we can do the exponential birthday approximation, which is:

  p = 1.0 - exp(-(n * (n-1)) / (2 * m))

where n is the number of entries in the directory, and m is the total
number of combinations.

That comes to about 0.0052 for 32767 files in a directory
(ie. maximully full), or about 0.5%.

With the previous patch we had 2^30 combinations, which came to 0.393,
or about 39%. So the new patch has about 75x lower chance of a single
collision than the old one.

Similarly for 100 files, the old patch gave a probability of
4.6*10^-6, whereas the new one gives 4.8x10^-8, which is about 96x
lower chance of a single collision. 

My '80x' was very rough of course, but its close enough to be a
reasonable rule of thumb I think. What matters here is orders of
magnitude, not precise numbers.

The chance of an actual bluescreen is much lower again, as a
bluescreen happens when two files that have the same 8.3 entry are
accessed in quick succession. Paul did some modelling of that, which
is where the numbers I gave in my last email came from.

I do agree that it would be much better if the chance was zero, but I
also think that the chances of a problem due to a collision are not
really very high, and it may well be lost in the noise of the (rather
common) bluescreens of the fastfat driver on WinXP.

Please do check my maths though. I'll be extremely surprised if you
get anything like 90% as I have run it for days with randomised
testing, and even with the lower chances of the last patch I couldn't
make it happen. I could make it happen by greatly reducing the amount
of randomness in the patch, which is why I know it can happen, but the
probability drops off pretty rapidly with more bits.

Still, my combinatorics is a little rusty, so please correct me if I
have made a mistake.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Pavel Machek]

Hi!

>  > >   - Similarly, there is a small chance that chkdsk on Windows will
>  > >     rename one file in a directory if they happen to have the same 11
>  > >     byte dummy values. The probability of this happening is
>  > >     approximately 80x lower than with the previous patch.
>  > 
>  > The "small chance" seems to be 90% for 30000 files in directory. And
>  > no, it is probably not 80x lower. Do the math.
> 
> Perhaps you could double check my math? 
> 
> The number of combinations available with the current patch is 
> 35^6 * 7 * 8 for files (for directories there are more combinations, 
> but lets ignore that for now).
> 
> That comes to 102942875000 combinations.
> 
> Now we can do the exponential birthday approximation, which is:
> 
>   p = 1.0 - exp(-(n * (n-1)) / (2 * m))
> 
> where n is the number of entries in the directory, and m is the total
> number of combinations.
> 
> That comes to about 0.0052 for 32767 files in a directory
> (ie. maximully full), or about 0.5%.
> 
> With the previous patch we had 2^30 combinations, which came to 0.393,
> or about 39%. So the new patch has about 75x lower chance of a single
> collision than the old one.
> 
> Similarly for 100 files, the old patch gave a probability of
> 4.6*10^-6, whereas the new one gives 4.8x10^-8, which is about 96x
> lower chance of a single collision. 

I guess I should apologize here. I did not do the math. It just seemed
to me that from the paradox of twins, difference should be
greater. Sorry for confusion.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 6407 bytes --]

Am Mittwoch 08 Juli 2009 schrieb tridge@samba.org:
> Hi Martin,

Hi Tridge,

>  > Have low level filesystem check, repair and cloning tools been
>  > checked against the patch at all?
>
> I have tested chkdsk.exe obvious, and I have reported the bug in
> chkdsk that this testing has found to Microsoft.
>
> I haven't tested tools like ghost or 3rd party tools. I don't actually
> have any of those tools myself, although I guess I could go out and
> buy them. Does anyone on this list have those tools and can let me
> know if they show any problems?

The question before that would be whether anyone has a comprehensive list 
of those tools, cause I think there are quite many. Well at least those 
from bigger vendors should be tested I think. Paragon, Symantec, ...

And has it been tested with Linux tools such as fsck.msdos, fsck.vfat, 
parted and partimage? I think it probably has not much effect on parted and 
partimage, but what about the fscks?

>  > I think the patch actively *corrupts* perfectly fine shortname FAT
>  > filesystems and at least for certain use scenarios even those with
>  > long name support.
>
> The patch only changes the values stored for new files created by
> Linux, so I think it is going a bit far to say it "actively corrupts"
> filesystems.

I do not. A filesystem is intact when all of its metadata is intact. This 
is at least what I believe and what various fsck tools seem to implement. 
Thus even when the patch only changes the values stored for new - or 
rewritten? - files it actively corrupts the meta consistency of the whole 
filesystem. To me it is like inserting a defective inode into a consistent 
Linux filesystem.

> I am looking into the Win9X issued raised by Jan. As I mentioned in my
> mail to him, it seems to work better with some different values in the
> 11 bytes. I'll keep looking at that, although I don't think Win9X
> support is a high priority for anyone any more. After installing Win98
> in a virtual machine I connected it to the windows update service to
> see if there had been updates to the old install images I had, and I
> found it couldn't even connect to windows update, it just throws a
> page full of html errors:
>
>   http://samba.org/tridge/Win98-update.png

I don't believe that Microsoft is still providing updates for Win98. But I 
think Windows 2000 might still be in use - I for example have a Win 2000 
installation on my ThinkPad T23, although I didn't boot it for about a 
year or so. Has it been tested against Windows 2000? I digged for the mail 
where you said something about against which Windows versions you tested, 
but I didn't find it anymore.

I think information regarding test status of the patch should be collected 
in the FAQ.

> When the vendor of an operating system doesn't even bother to display
> a clean "sorry, you don't get updates any more" page for their OS then
> I think it is safe to say that the operating system is dead and
> buried.

It is safe to say much. But still users might not behave according to your 
saying or might even not be able to. A potential customer asked us to 
migrate a Windows 98 installation into a virtual machine, cause the 
software that is running there would not run with any newer version of 
Windows. Sometimes people are locked / forced to a specific Windows (or 
Linux) version at least is they do not want to pay lots of $$$ to replace 
their proprietary special hardware + software combination by something 
which is supported on a newer version of an operating system. And for a 
coincidence I think digital photos have been involved in that use case.

I skip most of the political stuff. I was not my main point to make. It 
has been discussed before. We obviously have different oppinions and thats 
just how it is. No need to cycle around our different view points.

>  > If the Linux kernel would be changed to avoid any software patent
>  > issues I am not sure whether it would even be able to boot
>  > anymore.
>
> That argument can be used with pretty much any software (both
> proprietary and free), not just the Linux kernel.

I think this doesn't render my argument invalid. To me it merely says that 
also other free software projects shall be careful with politically or 
juristically motivated patches. Drastically spoken if everyone decides to 
jump out of the window, why should I follow?

> In the case of the GIF patents the correct answer was a concerted
> effort to switch to a new format. That was a fantastic campaign and
> largely successful.
>
> We don't, as yet, have any equivalent campaign to get behind for these
> VFAT patents. The calls for changing to a different filesystem format
> are great, but they fall down in an even worse way than what I have
> proposed on exactly the same issues. This hypothetical new format
> won't work with cheap MP3 players, won't work with Win9X, and almost
> certainly won't work with existing Windows boxes (yes, I know about
> UDF, but if you actually try it you'll see it isn't the panacea some
> have claimed). So in what way is it a solution, even if the new format
> existed?

Its challening, agreed. Especially as for it to be truly multi-plattform 
Linux developers, Apple, Microsoft and little device makers would have to 
agree on it. Or it needs to be a free software product that can be 
installed on any OS really easily. Then you could have a small fat16 
partition with a *.exe file and a MacOS X dmg to click on in order to 
install the driver for the real new inter-exchange filesystem. But what 
about device makers. And what about existing devices? I think it can only 
be a slow migration.

> If you can propose a truly workable alternative then I would be
> delighted to never have to think about FAT filesystems again in my
> life.

I did not meant any personal offence. So no need to justify yourself for 
what you do. You are just trying to find a workaround for the issue. And I 
am pretty sure you believe in what you do to be a viable workaround. I am 
not intending to shoot the messenger of a possible workaround. I am just 
not a fan of a patch like yours being in vanilla and thats about it ;).

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

Martin Steigerwald wrote:
> I don't believe that Microsoft is still providing updates for Win98. But I 
> think Windows 2000 might still be in use

Definitely.  The mail server belonging to a company that pays me for
embedded Linux work runs on Windows 2000.  I moved it to a virtual
machine about 1 year ago - it's still in use.

> - I for example have a Win 2000 
> installation on my ThinkPad T23, although I didn't boot it for about a 
> year or so. Has it been tested against Windows 2000? I digged for the mail 
> where you said something about against which Windows versions you tested, 
> but I didn't find it anymore.

Heh.  I still use Windows 95 and Windows 98 occasionally.  I'm a bit
disappointed to find Samba no longer tests against them :-)

I wouldn't be surprised if Windows ME has fewer users than 98.  98 had
a reputation for being the best of the non-NT series.

> > When the vendor of an operating system doesn't even bother to display
> > a clean "sorry, you don't get updates any more" page for their OS then
> > I think it is safe to say that the operating system is dead and
> > buried.
> 
> It is safe to say much. But still users might not behave according to your 
> saying or might even not be able to. A potential customer asked us to 
> migrate a Windows 98 installation into a virtual machine, cause the 
> software that is running there would not run with any newer version of 
> Windows. Sometimes people are locked / forced to a specific Windows (or 
> Linux) version at least is they do not want to pay lots of $$$ to replace 
> their proprietary special hardware + software combination by something 
> which is supported on a newer version of an operating system. And for a 
> coincidence I think digital photos have been involved in that use case.

I think you've described commercial ancient Windows users.

But I suspect there are more non-commercial users - that ancient PC
someone has in their home which is good enough at running Word 2 for
_their_ word processing needs.  You know the sort of thing: ancient
14-inch CRT still going strong, friend probably replaced the disk 5
years ago and cloned the original working OS, fan's getting a bit
noisy but the old clunker isn't worth replacing just yet.

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jeremy Allison]

On Wed, Jul 08, 2009 at 02:53:27PM +0100, Jamie Lokier wrote:

> Heh.  I still use Windows 95 and Windows 98 occasionally.  I'm a bit
> disappointed to find Samba no longer tests against them :-)

We do run regression tests that emulate Win9x against Samba
and ensure we pass. We also fix bugs reported by Win9x and
OS/2 users with Samba.

Jeremy.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan and Martin,

As promised, I have continued to investigate what is happening with
Win98, even though I don't think it is a high priority.

I've discovered two things that are interesting:

  1) the 11 spaces varient works pretty nicely with it

  2) the current dualnames patch works nicely if you change the first
  byte to a '~' instead of a space.

What seems to be happening is that Win98 either wants all spaces, or
it wants to see a '~' in the 8 byte prefix, after taking into account
nul termination. If it doesn't see this '~' then it doesn't even look
for a long name.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

Am Donnerstag 09 Juli 2009 schrieb tridge@samba.org:
> What seems to be happening is that Win98 either wants all spaces, or
       ^^^^^

I can't comment on the technical details, cause I do not know them. But 
this again looks to me like playing roulette on compatibility.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Martin,

 > The question before that would be whether anyone has a comprehensive list 
 > of those tools, cause I think there are quite many. Well at least those 
 > from bigger vendors should be tested I think. Paragon, Symantec, ...

Do you happen to have any of those handy to test with?

 > And has it been tested with Linux tools such as fsck.msdos, fsck.vfat, 
 > parted and partimage? I think it probably has not much effect on parted and 
 > partimage, but what about the fscks?

I tested it with dosfstools (which provides the fsck.vfat on Linux
distros) and with mtools. Both required patches to work correctly. I
have submitted both patches to the maintainers of those packages.

The patch to dosfstools makes it skip the invalid 8.3 entries, just as
windows chkdsk does. The patch is here:

  http://samba.org/tridge/dosfstools.patch1

The patch to mtools is partly cosmetic, and partly to fix a bug in the
VFAT checksum routine. The code in mtools incorrectly treated a nul
byte as special in 8.3 directory entries. The patch is here:

  http://samba.org/tridge/mtools.patch1

 > Thus even when the patch only changes the values stored for new - or 
 > rewritten? - files it actively corrupts the meta consistency of the whole 
 > filesystem. To me it is like inserting a defective inode into a consistent 
 > Linux filesystem.

If the windows implementation is taken as the reference implementation
then the files are not considered defective. The windows chkdsk will
(with a small probability) complain of duplicates, but it doesn't
complain about the entries being defective in any other way.

 > I don't believe that Microsoft is still providing updates for Win98. But I 
 > think Windows 2000 might still be in use - I for example have a Win 2000 
 > installation on my ThinkPad T23, although I didn't boot it for about a 
 > year or so. Has it been tested against Windows 2000? I digged for the mail 
 > where you said something about against which Windows versions you tested, 
 > but I didn't find it anymore.

I haven't tested against w2k yet. I'll need to dig through my old MSDN
CD stack and see if I can find a w2k CD to test with. It's no longer
offered on current MSDN subscriptions.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 3408 bytes --]

Am Donnerstag 09 Juli 2009 schrieb tridge@samba.org:
> Hi Martin,

Hi Tridge,

>  > The question before that would be whether anyone has a comprehensive
>  > list of those tools, cause I think there are quite many. Well at
>  > least those from bigger vendors should be tested I think. Paragon,
>  > Symantec, ...
>
> Do you happen to have any of those handy to test with?

There might be some stuff laying around on some magazine CD/DVDs, but 
honestly I lack the motivation to do any testing. I am still not a fan of 
your patch. And I think the people that are interested in getting the 
patch in should do the testing - not me. Why should I do the work that 
TomTom and other companies not willing or able to fight patents legally 
might be interested in getting done? A motivation would be to think that 
it actually helps towards a software patent free world, but I am not 
convinced that it does.

>  > And has it been tested with Linux tools such as fsck.msdos,
>  > fsck.vfat, parted and partimage? I think it probably has not much
>  > effect on parted and partimage, but what about the fscks?
>
> I tested it with dosfstools (which provides the fsck.vfat on Linux
> distros) and with mtools. Both required patches to work correctly. I
> have submitted both patches to the maintainers of those packages.
>
> The patch to dosfstools makes it skip the invalid 8.3 entries, just as
> windows chkdsk does. The patch is here:
>
>   http://samba.org/tridge/dosfstools.patch1
>
> The patch to mtools is partly cosmetic, and partly to fix a bug in the
> VFAT checksum routine. The code in mtools incorrectly treated a nul
> byte as special in 8.3 directory entries. The patch is here:
>
>   http://samba.org/tridge/mtools.patch1

Okay. But then before activating the your Linux kernel patch as default 
there should a transition time to let distro pick up the newest stuff and 
a safety margin for users to upgrade to them. If it should be activated as 
default at all.

>  > Thus even when the patch only changes the values stored for new - or
>  > rewritten? - files it actively corrupts the meta consistency of the
>  > whole filesystem. To me it is like inserting a defective inode into
>  > a consistent Linux filesystem.

> If the windows implementation is taken as the reference implementation
> then the files are not considered defective. The windows chkdsk will
> (with a small probability) complain of duplicates, but it doesn't
> complain about the entries being defective in any other way.

But you and others consider the characters / bytes your patch put into the 
short names as invalid. How did you come to that conclusion? Either these 
characters are invalid or they are not. An indication would be on what DOS 
would allow me to create. Does it allow me to create files with those 
characters such as blanks or null bytes with simple DOS commands. If not, 
then not checking filenames in short name FAT for invalid names seems to 
be an omission in CHKDSK for me.

Even when fsck.{ext3,whatever}/xfs_check/xfs_repair doesn't moan about "/" 
in filenames I would still consider them invalid and think any software 
which creates those actively corrupts an Ext3, XFS, whatever unix 
filesystem.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Pavel Machek]

On Thu 2009-07-09 14:13:04, tridge@samba.org wrote:
> Hi Martin,
> 
>  > The question before that would be whether anyone has a comprehensive list 
>  > of those tools, cause I think there are quite many. Well at least those 
>  > from bigger vendors should be tested I think. Paragon, Symantec, ...
> 
> Do you happen to have any of those handy to test with?
> 
>  > And has it been tested with Linux tools such as fsck.msdos, fsck.vfat, 
>  > parted and partimage? I think it probably has not much effect on parted and 
>  > partimage, but what about the fscks?
> 
> I tested it with dosfstools (which provides the fsck.vfat on Linux
> distros) and with mtools. Both required patches to work correctly. I
> have submitted both patches to the maintainers of those packages.

Yes... so we know that your patch breaks at least

98 (completely)
XP (randomly)
mp3 players (completely)
cameras (workaround possible)
linux (patches available).

I believe my suggested help text was very fair.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

Pavel Machek wrote:
> > I tested it with dosfstools (which provides the fsck.vfat on Linux
> > distros) and with mtools. Both required patches to work correctly. I
> > have submitted both patches to the maintainers of those packages.
> 
> Yes... so we know that your patch breaks at least
> 
...
> linux (patches available).

Ew.

Old Linux boxes don't get updates, and they don't have udev, hotplug
or auto-mounting desktops either.  They're the ones where mtools is
used on floppies.

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

tridge@samba.org wrote:
> I haven't tested against w2k yet. I'll need to dig through my old MSDN
> CD stack and see if I can find a w2k CD to test with. It's no longer
> offered on current MSDN subscriptions.

Windows NT-derivatives and Windows 95-derivatives use a significantly
different code base for their filesystems, even to the extent of
having different VFAT behaviour.  Hence shortnames=win95, shortname=winnt.

The last Windows 95-derivative is Windows ME, which is was released
about the same time as Windows 2000.  Windows 2000 is an NT-derivative.

So it might be worth testing against Windows ME.

I don't know what code base is used for Windows CE.  CE is still used,
on phones, PDAs and media players among other things.  Has there been
any testing against CE - current versions and old versions?

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

Jamie Lokier wrote:
> I don't know what code base is used for Windows CE.  CE is still used,
> on phones, PDAs and media players among other things.  Has there been
> any testing against CE - current versions and old versions?

Found something about Windows CE:

    On WinCE, long filenames do not need to be generated if the filename
    is 8.3 compatible: it must contain all uppercase characters, no
    unicode characters, and be no longer than 8 characters long with no
    more than a 3 character extension. For example, FILENAME.TXT would not
    generate a long filename and uses only one directory
    entry. FileName.Txt, filename.txt, and FILENAME0.TXT would all
    genenerate a long filename.

    From: http://blogs.msdn.com/medmedia/archive/2007/01/04/fat-filesystem-performance-issues.aspx

In other words, Windows CE uses the same algorithm as Windows 9x when
deciding whether to _create_ a long filename, corresponding to
shortname=mixed.

That post is from 2007, so likely to be what goes on recent CE devices.

I don't know if that means it _reads_ FAT the same way as Windows 9x,
or if they decided to do the same as shortname=mixed for maximum
compatibility with everything.  Has anyone ever checked if
shortname=winnt is what NT-derivatives like XP and Vista still do?

Obviously we care about how CE reads FAT, not when it creates long
names.  This is just a little clue that it might be the same as Windows 9x.

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Pavel Machek]

On Mon 2009-07-06 23:05:19, tridge@samba.org wrote:
> Hi Jan,
> 
>  > With this extra patch:
>  > 
>  >  * "cp dscf4160.jpg dscf3010.jpg" works
> 
> good
> 
>  >  * "cp dscf4160.jpg dscF3011.jpg" does not work - i.e. cam ignores the
>  >    file without error.
>  >    Reason for that is that dscF3011 has an illegal 8.3 name
> 
> yep, that's expected. I suggested a possible way of handling this to
> Hirofumi-san (ie. change the case of the name), but he prefers to keep
> things simple for now.
> 
> It is probably not unreasonable to say that you should not be using
> mixed case names when you are putting files onto a device that doesn't
> understand mixed case. The reason it worked before my patch is that
> the kernel forced the case of the 8.3 name, and only retained the
> mixed case for the long name.

It worked before. You claim that devices not understanding long
filenames are now extinct, but that camera is the
counterexample. Perhaps camera vendors fear patents, too.

So the original description is bad -- it breaks also new stuff -- and
it certainly should not be default=y.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Pavel,

 > It worked before. You claim that devices not understanding long
 > filenames are now extinct, but that camera is the counterexample.

I certainly never claimed that devices that don't understand long
filenames are extinct! I own more than one digital camera that doesn't
understand long filenames.

What I said in the patch help text is:

	  That means that long filenames created with this option
	  disabled will not be accessible at all to operating systems
	  that do not understand the VFAT extensions.

that means you have to choose 8.3 names if you want to copy files onto
a SD card and have those files visible to digital cameras. That is not
actually much of a change from how things behave without my patch for
current 8.3 only cameras. As Jan discovered, cameras tend to be pretty
fussy about the format of the filenames for files. His particular
camera wants the filenames to be dscfNNNN.jpg, which means that he
needed to be careful about choosing file names with current kernels as
well.

 > Perhaps camera vendors fear patents, too.

quite likely they want to minimise their costs, and not paying for a
patent license they don't need is one way to do that. I also expect
that not many of their customers put files onto the camera from a
computer, as opposed to taking them off. 

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Wednesday 2009-07-08 09:42, tridge@samba.org wrote:
>What I said in the patch help text is:
>
>	  That means that long filenames created with this option
>	  disabled will not be accessible at all to operating systems
>	  that do not understand the VFAT extensions.
>
>that means you have to choose 8.3 names if you want to copy files onto
>a SD card and have those files visible to digital cameras. That is not
>actually much of a change from how things behave without my patch for
>current 8.3 only cameras. As Jan discovered, cameras tend to be pretty
>fussy about the format of the filenames for files. His particular
>camera wants the filenames to be dscfNNNN.jpg, which means that he
>needed to be careful about choosing file names with current kernels as
>well.

In the large picture, the problem is not so much about devices that
restrict themselves to 8.3, such as DCIM-compatible cameras. They
"only" accept 8.3, so there is no point in trying to use a long
name because the software won't look for it in the first place.

It is much more with devices that are commonly operated with long
names - multimedia players come to mind - and these "always"
want a valid 8.3.

>> Perhaps camera vendors fear patents, too.
>
>quite likely they want to minimise their costs, and not paying for a
>patent license they don't need is one way to do that.

Well and there's always some vendors (the more so the one-product
vendors) not paying the royalities they are supposed to.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jan,

 > It is much more with devices that are commonly operated with long
 > names - multimedia players come to mind - and these "always"
 > want a valid 8.3.

Would you mind running an experiment for me with your IOneIt MP3
player?

Change vfat_build_dummy_83_buffer() to just do this:

  memset(msdos_name, ' ', 11);

and nothing else. Then delete and re-create the files that you had
problems with and see if your MP3 player is then happy to play
them. That helps a lot with Win98, and I wonder if it also helps with
your IOneIt player.

As I mentioned previously, since your report I'm trying to test on a
bunch of different MP3/media players. So far the tests I've done have
worked fine with the dualnames patch, but I'm still waiting on
delivery of more cheap MP3 players from eBay.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jan Engelhardt]

On Thursday 2009-07-09 04:23, tridge@samba.org wrote:
>
> > It is much more with devices that are commonly operated with long
> > names - multimedia players come to mind - and these "always"
> > want a valid 8.3.
>
>Would you mind running an experiment for me with your IOneIt MP3
>player?
>
>Change vfat_build_dummy_83_buffer() to just do this:
>
>  memset(msdos_name, ' ', 11);
>
>and nothing else. Then delete and re-create the files that you had
>problems with and see if your MP3 player is then happy to play
>them. That helps a lot with Win98, and I wonder if it also helps with
>your IOneIt player.

Nope. It must not be blank.

>As I mentioned previously, since your report I'm trying to test on a
>bunch of different MP3/media players. So far the tests I've done have
>worked fine with the dualnames patch, but I'm still waiting on
>delivery of more cheap MP3 players from eBay.

If I read a review right, the device has also been
distributed under the rebranded name Grundig MPAXX MP 650.
(Other MPxyz numbers are rebrands of other devices.)
The availability of MP650 is similarly low.

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Pavel Machek]

On Wed 2009-07-08 17:42:09, tridge@samba.org wrote:
> Hi Pavel,
> 
>  > It worked before. You claim that devices not understanding long
>  > filenames are now extinct, but that camera is the counterexample.
> 
> I certainly never claimed that devices that don't understand long
> filenames are extinct! I own more than one digital camera that doesn't
> understand long filenames.

Ok, so it really should not be called vfat, and really should not be
"default y".
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

tridge@samba.org wrote:
> When CONFIG_VFAT_FS_DUALNAMES is not set and a 8.3 name is created,
> force the use of the shortname=winnt option to allow for lowercase or
> uppercase prefix and extensions without using a long name.
> 
> +	unsigned opts_shortname = opts->shortname;
> +
> +#ifndef CONFIG_VFAT_FS_DUALNAMES
> +	opts_shortname = VFAT_SFN_CREATE_WINNT;
> +#endif

I'd prefer if it changes the shortname= default instead.

When mounting explicitly with shortname=mixed, this patch changes it
to shortname=winnt, which breaks compatibility when creating files on
Linux and reading on Windows 95/98/ME and devices with equivalent FAT
implementation.

I think it's ok to break that compatibility if dualnames is off,
because that's unfortunately the best available compromise.

But if shortname=mixed is explicitly requested, it can mean (and does
in some cases I've seen) that there's a reason long names are required
for lower-case 8.3 names.

-- Jamie

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[tridge]

Hi Jamie,

 > I think it's ok to break that compatibility if dualnames is off,
 > because that's unfortunately the best available compromise.

You probably noticed this, but the patch that has been put in only
changes the shortname setting when dualnames is off.

Cheers, Tridge

Re: CONFIG_VFAT_FS_DUALNAMES regressions

[Jamie Lokier]

tridge@samba.org wrote:
> Hi Jamie,
> 
>  > I think it's ok to break that compatibility if dualnames is off,
>  > because that's unfortunately the best available compromise.
> 
> You probably noticed this, but the patch that has been put in only
> changes the shortname setting when dualnames is off.

Yes, I noticed.

What I mean is if I do this, with your patch active:

    mount /dev/sdb -t vfat -o shortname=mixed

(or shortname=lower, or shortname=win95), then it's wrong to override
my shortname request "shortname=mixed", because my request means "I
need lower-case names to _appear_ lower-case on something with Windows
95-like behaviour.

When dual names cannot be created, if I specify any shortname= option
except shortname=winnt, I'd rather create a long name entry for
lower-case 8.3 names than a short name.

That's why dualnames off should change the *default* to
shortname=winnt, but not override an explicit shortname= option.

With dualnames on, the default should be shortname=mixed because that
has wider compatibility, and it's like the old Linux default when
creating names.

-- Jamie

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Rusty Russell]

On Wed, 1 Jul 2009 08:55:58 pm Alan Cox wrote:
> > I'd actually prefer to see the code ripped out and no config option
> > available; it would the clearest avoidance case possible.
>
> What about the free world - why should we suffer because Americans have
> a broken patent system ? That just leads to a Farenheit 451 model where
> the kernel sources end up containing no code, text or image that can
> offend, harm or be found wanting in any possible legal jurisdiction.

We put in workarounds for broken hardware, even though we rightfully dislike 
it.  Similarly, in this case a minor change prevents a real problem 
experienced by our users (yes, I know we only have one public bug report).

It's bad.  Yes, let's bitch about it.  And workaround it anyway.
Rusty.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

On Wed 2009-07-01 20:19:51, Rusty Russell wrote:
> On Tue, 30 Jun 2009 04:01:03 pm Pavel Machek wrote:
> > > +config VFAT_FS_DUALNAMES
> > > +	bool "VFAT dual names support"
> > > +	depends on VFAT_FS
> >
> > Defaults should be back-compatible.
> 
> Hi Pavel,
> 
> I disagree with this: given there's been testing with no known issues, it's 
> not a back compat question.

The testing uncovered some pretty serious issues. ('Causes XP to
bluescreen').
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Pavel,

 > > I disagree with this: given there's been testing with no known issues, it's 
 > > not a back compat question.
 > 
 > The testing uncovered some pretty serious issues. ('Causes XP to
 > bluescreen').

no, that's not quite correct. I have never been able to produce a
bluescreen with the patch I have posted.

I did manage to produce a bluescreen with a different patch that
deliberately lowers the number of bits of randomness used (ie. just
change the mask to random32() in vfat_build_dummy_83_buffer()). Even
then, producing a bluescreen is not trivial, though it is definately
possible.

As I explained in a previous posting, the WindowsXP bug doesn't happen
just because two 8.3 entries have the same 11 bytes. There are
additional strong constraints on the access patterns by the WindowsXP
kernel before the bluescreen happens. I reproduced it by running 4
processes in parallel doing massive numbers of random file operations,
with a controlling script transferring control of the USB key between
the WindowsXP box and a Linux host every minute (so both are doing
random file operations).

Even with that, if I had even a few bits of randomness then it took a
lot of testing (often hours) to produce the bluescreen. I wouldn't
have even known the problem was possible if I hadn't been
experimenting with patch varients where all the bytes are always the
same (such as the 11 spaces varient I mentioned). With those varients
(where there is no randomness at all), you can produce a bluescreen
with a bit of effort on the DOS command line.

So I hope you can see why I don't think this is a particularly large
problem, especially given that the WindowsXP FAT driver is notoriously
prone to crashing, as a quick google search will show you.

If someone can actually demonstrate the bluescreen with the patch I've
posted in some access pattern that is at all likely to be made by a
WindowsXP user then I would be more concerned. As it is, I don't think
we should hold off on this patch for what amounts to a theoretical
issue.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

On Fri 2009-07-03 08:06:54, tridge@samba.org wrote:
> Hi Pavel,
> 
>  > > I disagree with this: given there's been testing with no known issues, it's 
>  > > not a back compat question.
>  > 
>  > The testing uncovered some pretty serious issues. ('Causes XP to
>  > bluescreen').
> 
> no, that's not quite correct. I have never been able to produce a
> bluescreen with the patch I have posted.

So you know it causes XP to bluescreen, but can not reproduce that. So
what? Someone, somewhere _will_ reproduce it.
										Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Pavel,

 > So you know it causes XP to bluescreen, but can not reproduce that. So
 > what? Someone, somewhere _will_ reproduce it.

yes, like someone, somewhere will get data corruption in a TCP
connection because the checksum is quite weak and networking hardware
ain't perfect. 

Once the probabilties become small enough then it is normal to accept
imperfection in operating systems. If I could make it perfect I would,
but I haven't thought of a way to do that while being absolutely sure
of maintaining the full strength of the legal defence.

If you find a way to actually produce this problem in a way that is
realistic for Linux users then let me know. Otherwise I think it is
the best option we have, imperfect though it is. 

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

On Fri 2009-07-03 08:41:13, tridge@samba.org wrote:
> Hi Pavel,
> 
>  > So you know it causes XP to bluescreen, but can not reproduce that. So
>  > what? Someone, somewhere _will_ reproduce it.
> 
> yes, like someone, somewhere will get data corruption in a TCP
> connection because the checksum is quite weak and networking hardware
> ain't perfect.

That's why we have ethernet checksums. 

> Once the probabilties become small enough then it is normal to accept
> imperfection in operating systems. If I could make it perfect I
> but I haven't thought of a way to do that while being absolutely sure
> of maintaining the full strength of the legal defence.

It already _was_ perfect before you started patching it.

> If you find a way to actually produce this problem in a way that is
> realistic for Linux users then let me know. Otherwise I think it is
> the best option we have, imperfect though it is. 

So, you know that particular option will cause someone's XP to
bluescreen, and  you still want it to default to "Y"?
											Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Pavel,

 > That's why we have ethernet checksums. 

which of course just changes the number of bits. The argument remains
the same. 

 > It already _was_ perfect before you started patching it.

wow, I really didn't expect the objection to my patch to be that VFAT
is perfect!

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

On Fri 2009-07-03 09:59:35, tridge@samba.org wrote:
> Hi Pavel,
> 
>  > That's why we have ethernet checksums. 
> 
> which of course just changes the number of bits. The argument remains
> the same. 

Ok, so what is your estimate of XP crash probability?

>  > It already _was_ perfect before you started patching it.
> 
> wow, I really didn't expect the objection to my patch to be that VFAT
> is perfect!

But ... it was, and there is still possibility of just using position
in directory to avoid collisions completely.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Paul E. McKenney]

On Wed, Jul 08, 2009 at 11:21:33AM +0200, Pavel Machek wrote:
> On Fri 2009-07-03 09:59:35, tridge@samba.org wrote:
> > Hi Pavel,
> > 
> >  > That's why we have ethernet checksums. 
> > 
> > which of course just changes the number of bits. The argument remains
> > the same. 
> 
> Ok, so what is your estimate of XP crash probability?

Assuming the worst case where the user opens each file in the directory
of interest, the probability depends on the number of long-named files
in the given directory as follows (derivation at EOM):

Files	Probability		Odds
-----	--------------------	-------
32767	3.934446601778345b-1	(39.3%)
16384	1.174969255061134b-1	(11.7%)
 8192	3.076314519945223b-2	( 3.1%)
 4096	7.780179085651447b-3	( 0.8%)
 2048	1.950268317016874b-3	( 0.2%)
 1024	4.87685610530225b-4	(1 in 2,000)
  512	1.218244920604881b-4	(1 in 8,000)
  256	3.039790922077076b-5	(1 in 30,000)
  128	7.569761535306629b-6	(1 in 100,000)
   64	1.877544584847826b-6	(1 in 500,000)
   32	4.61935894834079b-7	(1 in 2,000,000)
   16	1.117587032466174b-7	(1 in 9,000,000)
    8	2.607703180994292b-8	(1 in 38,000,000)
    4	5.587935438151892b-9	(1 in 180,000,000)
    2	9.313225746154785b-10	(1 in 1,000,000,000)
    1	0.0b0

The number of files that people keep in a given directory will vary,
but in my admittedly limited experience, most people tend towards the
low end of this range.  What number of files per directory do you see
on embedded-device memory sticks?

Furthermore, this assumes that the user actually opens each and every file
in the directory.  If the user keeps (say) 1024 files in the directory,
but only opens two of them on any given memory-stick insertion, then
the odds are one in a billion rather than one in two thousand.

[Tridge -- isn't this only for subdirectories?  Or can collisions in
the root directory also result in WinXP bluescreens?]

> >  > It already _was_ perfect before you started patching it.
> > 
> > wow, I really didn't expect the objection to my patch to be that VFAT
> > is perfect!
> 
> But ... it was, and there is still possibility of just using position
> in directory to avoid collisions completely.

Almost.  The imperfection comes from the fact that storage media are not
totally reliable, so that given enough uses of large enough numbers of
memory sticks, there will eventually be a WinXP bluescreen.  Please note
that people really have reported this WinXP bug.

							Thanx, Paul

------------------------------------------------------------------------

Derivation:

o	The patch uses 6 random bytes, with 6 bits each, for
	32^6 = 1,073,741,824 possible combinations.  (Based on code
	in vfat_build_dummy_83_buffer() in the patch Tridge posted on
	June 27th.)

o	There are a maximum of 32,767 entries in a VFAT directory.

o	In the worst case, the user application will open each and
	every file in the directory.  I don't have any information on
	whether WinXP has a limited memory for recently open files.
	I therefore assume the worst case, namely that WinXP remembers
	every file that it has opened.

o	As noted in this thread, the probability of bluescreen is
	given by the infamous Birthday Paradox:

		P(n, m) = (n-1)! / ((n-m)! * n^(m-1))

	Here "n" is the number of possible birthdays (1,073,741,824 in
	this case) and "m" is the number of people (32,767 in this case).
	P(n, m) is the probability of -no- common birthday, so the
	probability of WinXP bluescreen is 1-P(n,m).

	Because I don't want to worry about round-off error, I use the
	arbitrary-precision arithmetic in the "maxima" symbolic-math
	package, which is related to the fabled Macsyma project.  However,
	computing the factorials without cancellation takes too long,
	so we transform the factorials into the binomial function, which
	has a highly optimized maxima implementation.  The equivalent
	but more efficient representation is as follows:

		b(n,m):=binomial(n,m)*m!/n^m;

	Then 1-b(32^6,32767) gives the exact probability of bluescreen
	on a maximal-sized directory, which is the ratio of a pair of
	extremely large integers.  More usefully, bfloat(1-b(32^6,32767))
	gives an approximation in scientific notation.  (Don't forget
	the trailing semicolon that maxima requires.)

	Computing bfloat(1-b(32^6,32767)) takes about 70 seconds on my
	2GHz x86 laptop.

	See below for the actual maxima output, typos and all.

------------------------------------------------------------------------

maxima

Maxima 5.12.0 http://maxima.sourceforge.net
Using Lisp GNU Common Lisp (GCL) GCL 2.6.7 (aka GCL)
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
This is a development version of Maxima. The function bug_report()
provides bug reporting information.
(%i1) b(n,m):=binomial(n,m)*m!/n^m;
                                    binomial(n, m) m!
(%o1)                    b(n, m) := -----------------
                                            m
                                           n
(%i2) bfloat(1-b(32^6,32767));
(%o2)                        3.934446601778345b-1
(%i3) bfloat(1-b(32^6,2^14));
(%o3)                        1.174969255061134b-1
(%i4) bfloat(1-b(32^6,2^13));
(%o4)                        3.076314519945223b-2
(%i5) bfloat(1-b(32^6,2^12));
(%o5)                        7.780179085651447b-3
(%i6) bfloat(1-b(32^6,2^11));
(%o6)                        1.950268317016874b-3
(%i7) bfloat(1-b(32^6,2^10));
(%o7)                         4.87685610530225b-4
(%i8) bfloat(1-b(32^6,2^9));
(%o8)                        1.218244920604881b-4
(%i9) bfloat(1-b(32^6,2^8));
(%o9)                        3.039790922077076b-5
(%i10) bfloat(1-b(32^6,2^7));
(%o10)                       7.569761535306629b-6
(%i11) bfloat(1-b(32^6,2^6));
(%o11)                       1.877544584847826b-6
(%i12) bfloat(1-b(32^6,2^));
Incorrect syntax: Illegal use of delimiter )
bfloat(1-b(32^6,2^)
                 ^
(%i12) bfloat(1-b(32^6,2^5));
(%o12)                        4.61935894834079b-7
(%i13) bfloat(1-b(32^6,2^4));
(%o13)                       1.117587032466174b-7
(%i14) bfloat(1-b(32^6,2^3));
(%o14)                       2.607703180994292b-8
(%i15) bfloat(1-b(32^6,2^2));
(%o15)                       5.587935438151892b-9
(%i16) bfloat(1-b(32^6,2^1));
(%o16)                       9.313225746154785b-10
(%i17) bfloat(1-b(32^6,2^0));
(%o17)                               0.0b0
(%i18)

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Paul,

These probabilities are way off. They assume that whatever interaction
happens in XP has infinite memory. The testing I've done indicates
that the memory for this interaction is very small (maybe 3 or 4? it's
hard to know precisely).

I've also confirmed this with lots of testing. If the probability was
39% for any directory size then I would have found it.

In all my testing I did not once produce a XP crash with the full
patch. To produce the XP crash I needed to have a reduced version of
the patch with less randomness.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Paul E. McKenney]

On Thu, Jul 09, 2009 at 07:42:27AM +1000, tridge@samba.org wrote:
> Hi Paul,
> 
> These probabilities are way off. They assume that whatever interaction
> happens in XP has infinite memory. The testing I've done indicates
> that the memory for this interaction is very small (maybe 3 or 4? it's
> hard to know precisely).

Good to know!  I will rework assuming that the memory is 4, let me
know if you learn more.

> I've also confirmed this with lots of testing. If the probability was
> 39% for any directory size then I would have found it.

This new information will likely reduce the predicted probability of
bluescreen by several orders of magnitude for large directories.  Not
that much effect for small directories, but not a real issue given
how low the probabilities are to begin with.

> In all my testing I did not once produce a XP crash with the full
> patch. To produce the XP crash I needed to have a reduced version of
> the patch with less randomness.

Well, let's see if I can produce a reasonably realistic model.  :-)
(I knew I should have gotten more sleep last night!!!)

							Thanx, Paul

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Paul E. McKenney]

[-- Attachment #1: Type: text/plain, Size: 7283 bytes --]

On Wed, Jul 08, 2009 at 03:14:13PM -0700, Paul E. McKenney wrote:
> On Thu, Jul 09, 2009 at 07:42:27AM +1000, tridge@samba.org wrote:
> > Hi Paul,
> > 
> > These probabilities are way off. They assume that whatever interaction
> > happens in XP has infinite memory. The testing I've done indicates
> > that the memory for this interaction is very small (maybe 3 or 4? it's
> > hard to know precisely).
> 
> Good to know!  I will rework assuming that the memory is 4, let me
> know if you learn more.
> 
> > I've also confirmed this with lots of testing. If the probability was
> > 39% for any directory size then I would have found it.
> 
> This new information will likely reduce the predicted probability of
> bluescreen by several orders of magnitude for large directories.  Not
> that much effect for small directories, but not a real issue given
> how low the probabilities are to begin with.
> 
> > In all my testing I did not once produce a XP crash with the full
> > patch. To produce the XP crash I needed to have a reduced version of
> > the patch with less randomness.
> 
> Well, let's see if I can produce a reasonably realistic model.  :-)
> (I knew I should have gotten more sleep last night!!!)

This model assumes a finite memory, so that at least two files out of
a group of "l" recently opened files have to collide to result in a
bluescreen.  I don't trust it yet, but thought I should give people a
chance to poke holes in it.

							Thanx, Paul

------------------------------------------------------------------------

Results

Assuming the worst case where the user opens each file in the directory
of interest, the probability depends on the number of long-named files
in the given directory as follows (derivation at EOM):

Files	Probability		Odds
-----	--------------------	-------
32767	9.15401625433259b-5	(1 in 11,000)
16384	4.576973184985319b-5	(1 in 22,000)
 8192	2.288233388567135b-5	(1 in 44,000)
 4096	1.143843845796893b-5	(1 in 87,000)
 2048	5.716441632059139b-6	(1 in 170,000)
 1024	2.855430941007629b-6	(1 in 350,000)
  512	1.424922525947476b-6	(1 in 700,000)
  256	7.096675510325187b-7	(1 in 1,400,000)
  128	3.520398717286601b-7	(1 in 2,800,000)
   64	1.732259841151157b-7	(1 in 5,800,000)
   32	8.381902831793723b-8	(1 in 12,000,000)
   16	3.911554742174612b-8	(1 in 26,000,000)
    8	1.676380622425006b-8	(1 in 60,000,000)
    4	5.587935438151892b-9	(1 in 179,000,000)
    2	9.313225746154785b-10	(1 in 1,000,000,000)
    1	0.0b0

This is for 2^32 random values per entry and a WinXP "memory" of four
entries.

------------------------------------------------------------------------

Derivation:

o	The patch uses 6 random bytes, with 6 bits each, for
	32^6 = 1,073,741,824 possible combinations.  (Based on code
	in vfat_build_dummy_83_buffer() in the patch Tridge posted on
	June 27th.)

o	There are a maximum of 32,767 entries in a VFAT directory.

o	In the worst case, the user application will open each and
	every file in the directory.  WinXP seems to have a limited
	memory for recently opened files, and we assume that once this
	memory is full, opening a new file causes one of the recently
	opened files to be forgotten.  The size of its memory appears
	to be in the range of 3-4 based on Tridge's testing, so we
	will assume the worst case (4) and parameterize with l=4.

o	As noted earlier, the probability the infamous Birthday Paradox
	is given by:

		P(n, m) = (n-1)! / ((n-m)! * n^(m-1))

	Here "n" is the number of possible birthdays (1,073,741,824 in
	this case) and "m" is the number of people (32,767 in this case).
	P(n, m) is the probability of -no- common birthday.  This is not
	the probability of no bluescreen, but we will use this result
	to calculate this probability while initially filling WinXP's
	memory.  I again use maxima, and again use the optimized form
	based on the binomial function:

		b(n,m):=binomial(n,m)*m!/n^m;

o	See the attached .eps...

o	The probability of no bluescreen while opening the first "l"
	files is given by b(n,l), just as before.

o	Given no bluescreen while opening the first "l" files, the
	probability of no bluescreen while opening the "l+1"st file
	is the probability of missing the "l-1" files that remain
	after ejecting one of them to make room is "(n-l+1)/n".

	The cumulative probability of no bluescreen is thus:

		P(n,m,l) = b(n,l)*(n-l+1)/n

o	We have the same situation when opening the "l+2"nd file,
	the probability of no bluescreen is again "(n-l+1)/n".

o	This situation will repeat "m-l" times, so that we have:

		P(n,m,l) = b(n,l)*((n-l+1)/n)^(m-l)

	In maxima-speak:

		b(n,m):=binomial(n,m)*m!/n^m;
		nbs(n,m,l):=b(n,l)*((n-l+1)/n)^(m-l);

	The probability of bluescreening when faced with 32767 files
	in a directory, 32^6 possible random values, and the capacity
	to remember four files is then:

		1-nbs(32^6,32767,4);

	For floating-point results instead of a massive fraction:

		bfloat(1-nbs(32^6,32767,4));

	See below for the actual maxima output, typos and all.

------------------------------------------------------------------------

maxima

paulmck@paulmck-laptop:~$ maxima

Maxima 5.12.0 http://maxima.sourceforge.net
Using Lisp GNU Common Lisp (GCL) GCL 2.6.7 (aka GCL)
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
This is a development version of Maxima. The function bug_report()
provides bug reporting information.
(%i1) b(n,m):=binomial(n,m)*m!/n^m;
                                    binomial(n, m) m!
(%o1)                    b(n, m) := -----------------
                                            m
                                           n
(%i2) nbs(n,m,l):=b(n,l)*((n-l+1)/n)^(m-l);
                                            n - l + 1 m - l
(%o2)              nbs(n, m, l) := b(n, l) (---------)
                                                n
(%i3) bfloat(1-nbs(32^6,32767,4));
(%o3)                         9.15401625433259b-5
(%i4) bfloat(1-nbs(32^6,2^14,4));
(%o4)                        4.576973184985319b-5
(%i5) bfloat(1-nbs(32^6,2^13,4));
(%o5)                        2.288233388567135b-5
(%i6) bfloat(1-nbs(32^6,2^12,4));
(%o6)                        1.143843845796893b-5
(%i7) bfloat(1-nbs(32^6,2^11,4));
(%o7)                        5.716441632059139b-6
(%i8) bfloat(1-nbs(32^6,2^10,4));
(%o8)                        2.855430941007629b-6
(%i9) bfloat(1-nbs(32^6,2^9,4));
(%o9)                        1.424922525947476b-6
(%i10) bfloat(1-nbs(32^6,2^8,4));
(%o10)                       7.096675510325187b-7
(%i11) bfloat(1-nbs(32^6,2^7,4));
(%o11)                       3.520398717286601b-7
(%i12) bfloat(1-nbs(32^6,2^6,4));
(%o12)                       1.732259841151157b-7
(%i13) bfloat(1-nbs(32^6,2^5,4));
(%o13)                       8.381902831793723b-8
(%i14) bfloat(1-nbs(32^6,2^4,4));
(%o14)                       3.911554742174612b-8
(%i15) bfloat(1-nbs(32^6,2^3,4));
(%o15)                       1.676380622425006b-8
(%i16) bfloat(1-nbs(32^6,2^2,4));
(%o16)                       5.587935438151892b-9
(%i17) bfloat(1-nbs(32^6,2^1,4));
(%o17)                      - 1.734723480823569b-18
(%i18) bfloat(1-b(32^6,2));
(%o18)                       9.313225746154785b-10
(%i19) bfloat(1-b(32^6,1));
(%o19)                               0.0b0

[-- Attachment #2: WinXPmodel.eps --]
[-- Type: application/postscript, Size: 8861 bytes --]

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[H. Peter Anvin]

Pavel Machek wrote:
> 
> But ... it was, and there is still possibility of just using position
> in directory to avoid collisions completely.

I was just going to ask this.  Rather than using a random number, we
should use something based on the position in the directory, and I
suspect we should still verify that it is unique (since some FAT
implementation may garbage-collect the directory thereby moving the offset.)

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Rusty Russell]

On Fri, 3 Jul 2009 08:14:46 am Pavel Machek wrote:
> On Fri 2009-07-03 08:41:13, tridge@samba.org wrote:
> > If you find a way to actually produce this problem in a way that is
> > realistic for Linux users then let me know. Otherwise I think it is
> > the best option we have, imperfect though it is.
>
> So, you know that particular option will cause someone's XP to
> bluescreen, and  you still want it to default to "Y"?

No, his point is the opposite: the rate of incidence of bluescreens on XP will 
not measurably increase due to this patch.

Cheers,
Rusty.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Rusty Russell]

On Fri, 3 Jul 2009 07:16:46 am Pavel Machek wrote:
> On Wed 2009-07-01 20:19:51, Rusty Russell wrote:
> > On Tue, 30 Jun 2009 04:01:03 pm Pavel Machek wrote:
> > > > +config VFAT_FS_DUALNAMES
> > > > +	bool "VFAT dual names support"
> > > > +	depends on VFAT_FS
> > >
> > > Defaults should be back-compatible.
> >
> > Hi Pavel,
> >
> > I disagree with this: given there's been testing with no known issues,
> > it's not a back compat question.
>
> The testing uncovered some pretty serious issues. ('Causes XP to
> bluescreen').

No, read more carefully: testing revealed "couldn't cause XP to bluescreen".  
It took a deliberately hobbled version of the patch to tickle the XP bug, 
under a day of stress testing.

Cheers,
Rusty.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Pavel,

 > Defaults should be back-compatible.

I would usually agree, but I think we have an unusual situation here,
in some ways similar to a demonstrated security hole. The previous
behaviour exposed a lot of Linux vendors to the possibility of an
expensive legal fight.

If we had some way to be completely backwards compatible while
avoiding the legal issues then of course we should take that. I tried
hard to find a varient that achieved that, but the best I could manage
was what I have posted. It may be that someone else can find a better
varient that avoids the legal issues while remaining fully backward
compatible.

 > Users considering disabling this should understand that filesystem
 > they write to will not be valid vfat filesystems, and may trigger bugs
 > in some devices.

If we find any devices that exhibit any problems with this patch while
it is in linux-next (and maybe linux-mm?) then this warning would
indeed be appropriate. It no such devices are known then I think the
warning is going a bit far. 

Do you have a specific device in mind with regard to this warning?

 > What happens on collision? With 60000 entries in directory, there will
 > be 50% chance of collision. BAD.

Why do you say it is bad? You need to remember that those 11 bytes
cannot be used as a filename, so it isn't a filename collision. As I
mentioned in the reply to Eric's question, a quite good choice is to
use 11 spaces for every file, and that only falls down because
WindowsXP has a bug that causes that varient to bluescreen.

The only risks with collisions that I am aware of are:

  - there is a very small chance (much, _much_ less than 50% with 60k
    files) of WindowsXP bluescreening. I've never in fact managed to
    reproduce this with the current patch, despite many days of
    continuous randomised testing. The reason the probability is much
    smaller than 50% at 60k is that it doesn't bluescreen just because
    there is a duplicate 8.3 entry - it bluescreens with a particular
    access pattern that involves 2 entries with the same 11 bytes,
    that access pattern is very hard to produce.

  - chkdsk.exe will complain about duplicates, and will rename one of
    the two files. That is a 50% chance of 1 file being renamed for a
    single directory containing 60k files. Given it isn't all that
    common to run chkdsk on removable media that is shared between
    Linux and Windows, I thought that this is not a terribly large
    concern.

 > Why not use something like position in directory instead of random
 > number?

We did of course consider that, and the changes to the patch to
implement collision avoidance are relatively simple. We didn't do it
as it would weaken the legal basis behind the patch. I'll leave it to
John Lanza (the LF patent attorney) to expand on that if you want more
information.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> I would usually agree, but I think we have an unusual situation here,
> in some ways similar to a demonstrated security hole. The previous
> behaviour exposed a lot of Linux vendors to the possibility of an
> expensive legal fight.

They don't have to ship the code. They can rip it out. They deal with
video players the same way for the USSA market and have done for years.

>   - chkdsk.exe will complain about duplicates, and will rename one of
>     the two files. That is a 50% chance of 1 file being renamed for a
>     single directory containing 60k files. Given it isn't all that
>     common to run chkdsk on removable media that is shared between

Its a standard usage pattern for some people. Think about Linux based
commodity devices such as the N770 and plugging it into the users general
purpose PC box. Whenever it got pulled out wrongly it *will* get a chkdsk
in Windows.

>     Linux and Windows, I thought that this is not a terribly large
>     concern.

Disagree. Its a rapidly growing market segment.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Alan,

 > They don't have to ship the code. They can rip it out. They deal with
 > video players the same way for the USSA market and have done for years.

yes, vendors can make the patch unconditional of course. I thought
that we were not trying to encourage divergance between the kernel.org
tree and the vendor trees though? I know some divergance is always
going to happen, but it seems counter productive to be encouraging it.

 > Its a standard usage pattern for some people. Think about Linux based
 > commodity devices such as the N770 and plugging it into the users general
 > purpose PC box. Whenever it got pulled out wrongly it *will* get a chkdsk
 > in Windows.

really? I haven't noticed that behaviour for removable devices in
windows. You can manually set a drive to be checked on reboot, but I
wasn't aware of any automatic chkdsk mechanism for VFAT removable
media in windows. Have you seen this yourself?

In testing this patch I've pulled USB keys in and out of WinXP, Vista
and Windows7 hundreds of time, and done it programatically millions of
times via scripting on virtual machines, but I never noticed this
feature. Perhaps it happens only under some specific circumstances I
haven't seen?

 > >     Linux and Windows, I thought that this is not a terribly large
 > >     concern.
 > 
 > Disagree. Its a rapidly growing market segment.

you chopped off a word or two in the quote :-)

Of course I care about Linux/Windows interop, as I'm sure you know!
What I'm saying is that running chkdsk on a shared removable media
which contains 30k files in a single directory is not a common event,
and even when it does happen the consequences are that approximately
one in 3 times you do this, one of those 30k files gets renamed.

As evidence for this I'd like to point out that windows chkdsk has
complained about the Linux VFAT implementation for a long time, even
without my patch. If you create a filesystem with mformat and then
chkdsk it on windows you get:

  Bad links in lost chains at cluster 2 corrected
  Convert lost chains to files (Y/N)

yet I can't find a single instance of anyone complaining about this
with a google search. If a chkdsk was automated for removable media
then a lot of peoples machines would be asking them this question a
lot of the time. If people manually ran windows chkdsk on removable
VFAT media created on Linux then they also would have hit this and I
would have expected at least someone to have mentioned it.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> yes, vendors can make the patch unconditional of course. I thought
> that we were not trying to encourage divergance between the kernel.org
> tree and the vendor trees though? I know some divergance is always
> going to happen, but it seems counter productive to be encouraging it.

What about all the other damage vendors do to the tree and the junk they
stuff in their kernels - we don't accept that upstream either ?

> 
>  > Its a standard usage pattern for some people. Think about Linux based
>  > commodity devices such as the N770 and plugging it into the users general
>  > purpose PC box. Whenever it got pulled out wrongly it *will* get a chkdsk
>  > in Windows.
> 
> really? I haven't noticed that behaviour for removable devices in
> windows. You can manually set a drive to be checked on reboot, but I
> wasn't aware of any automatic chkdsk mechanism for VFAT removable
> media in windows. Have you seen this yourself?

Its manual unless the device is visibly corrupted but it should still
work. The point I was making is that the world of "Windows PC & Linux
handheld device" is an important one.

> lot of the time. If people manually ran windows chkdsk on removable
> VFAT media created on Linux then they also would have hit this and I
> would have expected at least someone to have mentioned it.

Devices come formatted for VFAT already - and I can't duplicate your bug
report on a USB key here so presumably some specific size/defaults are
needed ? (Would be good to fix that anyway)

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Alan,

 > What about all the other damage vendors do to the tree and the junk they
 > stuff in their kernels - we don't accept that upstream either ?

sure, but there is no need to encourage bad behaviour by building it
into the upstream :-)

 > Its manual unless the device is visibly corrupted but it should still
 > work.

ok, with the patch I've posted applied, can you reproduce this?
Perhaps by deliberately corrupting the device in some other way (using
dd?) then running a chkdsk?

If you try to reproduce it then make sure you completely fill the VFAT
directory. If you have (for example) only 1000 files in a directory
then you'd have to redo the deliberate corruption test with new files
about 2000 times before you get a single error and rename from
chkdsk. If you only have 100 files then you'll have to do it nearly a
million times per rename.

 > The point I was making is that the world of "Windows PC & Linux
 > handheld device" is an important one.

yes, it's an extremely important use for Linux. That is why I've spent
the last 4 months working on ensuring that it continues to be viable,
by trying to create the most legally robust, most compatible patch I
can that allows these devices to continue to exist without running
significant legal risks.

If there is another approach that achieves this goal in a better way
then we should look at it. Can you suggest an alternative that will
work better for Linux handheld device makers?

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> If there is another approach that achieves this goal in a better way
> then we should look at it. Can you suggest an alternative that will
> work better for Linux handheld device makers?

They can type 

patch -p1 < linux-foundation-recommended-ussa-fat.patch

which removes all the subversive terrorist code that Americans are scared
to posess.

As has already been said from their point of view thats lowest risk so
they will go that path anyway because the question is always "how do I
cover my backside maximally"

Vendors already do this for other stuff. Why is the kernel special or
different ? Take a look at Red Hat RPMS of music players, video players,
even things like netpbm for a while where JBIG had patent issues and was
simply ripped out of the package.

I think you are in real danger of creating a compromise that helps nobody

Putting the stuff in kernel upsets everyone who isn't under US rule,
creates situations where things cannot be discussed and doesn't make one
iota of difference to the vendors because they will remove the code from
the source tree options and all anyway - because as has already been said
it reduces risk.

It also sets a very dangerous precedent. What do you want to do about say
our Kconfig help for 'Taiwanese' if it doesn't meet Chinese regulations.
What about the Burmese code page. What about code submissions from Iran ?

The kernel really cannot and should not get involved in these. Its a
vendor problem. They deal with it every day of the year already from MP3
to Taiwanese flags to US crypto export regulations to patents to the
openssh option for using the DVD crypto algorithm to making sure their
language definitions don't get their product banned in some jurisdiction
or another. They have to remove games that glorify war in any way from
German product, and so on..

None of this other mass of red tape, bogocracy and the like gets dumped
on users all over the world. It's handled by specialists in vendor
companies supported by qualified legal personnel with whom they can have
full and frank discussion.

Another problem with this is that the foundation lawyers goals are almost
certainly influenced by the goal to maximise arse coverage for large
businesses with US connections. That is who pays them, that is who asks
their advice. That puts them partly at odds with the general good.

Alan

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Alan,

 > They can type 
 > 
 > patch -p1 < linux-foundation-recommended-ussa-fat.patch
 > 
 > which removes all the subversive terrorist code that Americans are scared
 > to posess.

I think you'll find that most distro makers and most vendors of Linux
based devices will want to have this patch applied. So if we took the
approach you suggest, then the testing done via kernel.org trees will
not match what the vast majority of Linux users will actually be
running. That would seem to be counter to good QA practices.

If the patch had significant negative consequences for normal use then
I'd be proposing the default be for dualnames to be enabled, which is
why the first patch I posted in May that disabled creation of long
filenames defaulted to off.

So please do as much testing as you have time for, and I'll continue
to ask people to let me try their mobile phones with a microSD card
and see how many different embedded OSes I can test that way. So far
the count is zero devices affected in a negative way by the proposed
patch.

If we get some real examples of devices that are badly affected then
we could revisit the question of what the default should be. With the
count at zero it seems counter-productive to disable this change in
the git trees that will allow it to get some wider testing.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> I think you'll find that most distro makers and most vendors of Linux
> based devices will want to have this patch applied. So if we took the

Good for them

> approach you suggest, then the testing done via kernel.org trees will
> not match what the vast majority of Linux users will actually be
> running. That would seem to be counter to good QA practices.

The vendor trees all contain patches and have done for years, often lots
of them. They will apply the patch if you put VFAT_DUALNAMES into the
kernel as well so your argument is totally bogus. It always was totally
bogus, it always will be. Vendors do not use the base kernel as is in
normal product. They ship

	Base kernel
	+ Patches for packaging systems
	+ .1/.2/.3 bits
	+ Cherry pick of stuff that deals with bugs they think have to be
	  stomped by some means
	+ drivers they add which are not yet fit for kernel
	+ exciting stuff they think is cool and makes their distro
	  special (eg KMS patches)


Its starting to sound like the Foundation and someone have signed some
kind of settlement to get this change into the Linux kernel regardless of
the technical issues, practicality or community and this is all for show
anyway.

> If the patch had significant negative consequences for normal use then

You said yourself it can crash XP.

> If we get some real examples of devices that are badly affected

XP crashing by suprise if you are really really unlucky strikes me as a
good example and one you provided.

You've also ignored entirely the fact that this isn't a VFAT file system
so irrespective of whether this goes in it should not be used for mount
-o vfat.

There is a clear end user expectation that vfat means "microsoft fat with
extended names". By all means add support for "not vfat" but don't call
it "vfat" as that is misleading an interface breakage.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Alan,

 > The vendor trees all contain patches and have done for years, often lots
 > of them. They will apply the patch if you put VFAT_DUALNAMES into the
 > kernel as well so your argument is totally bogus. It always was totally
 > bogus, it always will be.

well, I certainly don't have your experience with kernel development,
but it seems fairly basic to me that if a patch is likely to be in use
by the majority of Linux users (via distros/vendors) that having it in
the upstream kernel makes sense. 

If I'm wrong and a large proportion of distros/vendors won't want this
patch then of course the argument falls down, and in that case it
would seem perfectly reasonable for the upstream kernel not to have
it, or for it to be inactive by default.

Meanwhile it has gone into fatfs-2.6, and linux-next where I hope it
will get some testing. Whether it then goes into Linus's tree for
2.6.32 is a decision for Hirofumi (and presumably Linus if he decides
to weigh in on this). Similarly what the default should be is their
decision. I'm happy that it is getting some testing now, and I hope
that if that testing doesn't show any issues that it will be accepted
for 2.6.32.

 > Its starting to sound like the Foundation and someone have signed some
 > kind of settlement to get this change into the Linux kernel regardless of
 > the technical issues, practicality or community and this is all for show
 > anyway.

err, no. The idea for this patch came when I was reading the first lwn
article on the TomTom suit, and because I happen to have a TomTom at
home and like it. I then started investigating and found some ways to
avoid the patent. I do patent avoidance work quite frequently,
although usually for Samba and in that case I just put the patches
into Samba along with the rest of my code.

There is no "settlement" here (who exactly are you saying is settling
with whom??). There is just a patent avoidance patch that I think is a
very good idea, and that I have gone to the effort of getting very
carefully checked by the appropriate experts. I was helped in this by
some extremely good people (such as Paul McKenney, who has a lot of
experience in this area), and I think the result is a good one.

This discussion is not for show. If Hirofumi/Linus or whoever else is
responsible reject the patch then we don't have any secret backdoor to
get it in. All we can do is present the patch and hope to persuade the
kernel community that it is a good idea.

 > > If the patch had significant negative consequences for normal use then
 > 
 > You said yourself it can crash XP.

well, the WindowsXP fastfat driver isn't exactly known as being
stable! Do a search for fastfat bluescreen and you'll find a huge
number of hits on people complaining about it bluescreening when it
opens FAT filesystems created by all sorts of systems (eg. iPods).

 > You've also ignored entirely the fact that this isn't a VFAT file system
 > so irrespective of whether this goes in it should not be used for mount
 > -o vfat.

Other OSes (eg. Windows) don't tend to distinguish between FAT and
VFAT. It is just a FAT filesystem with a varying range of features. On
Linux we've chosen one particular set of features and labelled that
VFAT, then we've chosen a different set of features and labelled it
'MSDOS'. Within what we have labelled as VFAT we have some existing
runtime options that select different varients. I would have liked to
do the 'dualnames' patch as a runtime option, but that doesn't satisfy
the legal requirements.

So I think you're coming at this very much from a Linux centric point
of view (not surprising!), and even there I very much doubt that many
end users think about the intricacies of what FAT filesystem options
they want. Most systems just auto-mount removable media these days
with whatever options the distro vendors choose (or via HAL rules).

Thus I don't think the "end user expectation" is a strong argument for
making this an entirely new filesystem.

It really is up to Hirofumi-san though. If he asks me to redo the
patch so that it creates a new FAT filesystem varient then I'd be more
than happy to do that. I personally think it will create more
maintainence overhead than it is worth, but as the FAT filesystem
maintainer it is certainly his choice.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> but it seems fairly basic to me that if a patch is likely to be in use
> by the majority of Linux users (via distros/vendors) that having it in
> the upstream kernel makes sense. 
> 
> If I'm wrong and a large proportion of distros/vendors won't want this
> patch then of course the argument falls down, and in that case it

It's an open list so they can speak their minds here....

> would seem perfectly reasonable for the upstream kernel not to have
> it, or for it to be inactive by default.

You said yourself that the vendors would prefer to simply remove the code
which means they will carry the "unpatch" anyway.

> Other OSes (eg. Windows) don't tend to distinguish between FAT and
> VFAT. It is just a FAT filesystem with a varying range of features. On
> Linux we've chosen one particular set of features and labelled that
> VFAT, then we've chosen a different set of features and labelled it
> 'MSDOS'. Within what we have labelled as VFAT we have some existing
> runtime options that select different varients. I would have liked to
> do the 'dualnames' patch as a runtime option, but that doesn't satisfy
> the legal requirements.

That I don't believe for one minute providing the dualnames option (or -t
vfat) does not work in the non VFAT case.

That is

CONFIG_VFAT_DUALNAMES = y
	mount -t vfat		works
	mount -o dualnames=1    works
	mount -o tridgefat	works (gives you dualnames = 0)

CONFIG_VFAT_DUALNAMES = n
	mount -t vfat		fails
	mount -t vfat,-o dualnames=0 maybe works
	mount-o tridgefat	works

I can believe that shipping code supporting vfat dualnames and not using
the option at runtime would be a problem IFF the patent is shown to be
valid and software patents are valid in the USA. Thus for the problem
space you are worried about it needs to be compiled out. Given the GPL
says I can ask for the source to match the binaries I'm very very sure
any worried distro would go further and never ship that source either -
its lower risk/lower hassle.

Question for the lawyers: If the option is runtime but the case of
concern cannot be selected (viz support is not even compiled in is that a
problem. If not then -o dualnames= makes life much easier)

I have no problem with the idea of a compile time option for not doing
true VFAT so certain US people can provide it. I don't howeve believe
anyone will use it because they either

a) think it is a problem and have US issues in which case they will
remove the code entirely as that is the lowest risk option and with
little other internal inconvenience

b) they don't think it is a problem (the vast array of non-US
distributions from Ubuntu to Red Flag) and don't select it.

> end users think about the intricacies of what FAT filesystem options
> they want. Most systems just auto-mount removable media these days

That is precisely why it needs to not mount as vfat

> with whatever options the distro vendors choose (or via HAL rules).
> 
> Thus I don't think the "end user expectation" is a strong argument for
> making this an entirely new filesystem.

Its a name not a file system. As simple as

#ifdef CONFIG_FOO
	register vfat
#else
	register lfat
#endif


To my mind we need to be careful of three things

- Harming the kernel to work around a potentially country specific
  unproven problem for the benefit of a few big corporations only
- Getting into situations where big companies behind closed doors make
  unaccountable decisions about a project they do not own
- Setting trends for country specific fixups. There are a lot of
  countries and if we keep the US happy we have to keep China happy and
  so it goes on. Big corporations employ armies of specialists for these
  purposes and make the gain from it. The community doesn't so it should
  no more carry the pain of it than of long term stable releases and
  supporting five year old vendor kernels

So IMHO

	Put dualnames in by all means - but don't pretend it makes any
	difference on the QA front or concerned vendors would set the
	option - they won't, they'll go further.

	Don't make the non dualnames mode "vfat", it's not, and the very
	fact users are casual about assuming vfat/fat/windows compatible
	is why it is so important. (As an analogy most end users don't
	know that a 'CD' is specified by a pile of specs, they don't
	care. But they most definitely care if their CD won't play on
	everything).

And a question: Is there a trivial way to make tridge fat differentiable
from vfat/fat without Windows noticing. That way HAL can preserve the
type settings given a kernel that supports it.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[OGAWA Hirofumi]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> To my mind we need to be careful of three things
>
> - Harming the kernel to work around a potentially country specific
>   unproven problem for the benefit of a few big corporations only

My choice is to provide the option of those to users. I'm not saying to
ripping out the code for some users.

> - Getting into situations where big companies behind closed doors make
>   unaccountable decisions about a project they do not own

Now, I'm not working for any companies. I can say there is no closed
doors for me.

> - Setting trends for country specific fixups. There are a lot of
>   countries and if we keep the US happy we have to keep China happy and
>   so it goes on. Big corporations employ armies of specialists for these
>   purposes and make the gain from it. The community doesn't so it should
>   no more carry the pain of it than of long term stable releases and
>   supporting five year old vendor kernels

I'm just thinking about users. If vendors shipped the buggy code like
first buggy patch, I just thought it's more bad for our users. That's
all.

If there is community decision, I'll be really glad to follow it.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[James Bottomley]

On Thu, 2009-07-02 at 11:32 +0100, Alan Cox wrote:
> > I think you'll find that most distro makers and most vendors of Linux
> > based devices will want to have this patch applied. So if we took the
> 
> Good for them
> 
> > approach you suggest, then the testing done via kernel.org trees will
> > not match what the vast majority of Linux users will actually be
> > running. That would seem to be counter to good QA practices.
> 
> The vendor trees all contain patches and have done for years, often lots
> of them. They will apply the patch if you put VFAT_DUALNAMES into the
> kernel as well so your argument is totally bogus. It always was totally
> bogus, it always will be. Vendors do not use the base kernel as is in
> normal product. They ship

As a linux maintainer, you know we try very hard to encourage vendors
not to carry long lived out of tree patches because of the complexity it
causes for everyone.

> 	Base kernel
> 	+ Patches for packaging systems
> 	+ .1/.2/.3 bits
> 	+ Cherry pick of stuff that deals with bugs they think have to be
> 	  stomped by some means
> 	+ drivers they add which are not yet fit for kernel
> 	+ exciting stuff they think is cool and makes their distro
> 	  special (eg KMS patches)

Right but all of the above are short lived except the packaging patches
(which are distro kernel specific), meaning they eventually go upstream.
You're proposing a permanent out of tree patch, which is counter to
this.

> Its starting to sound like the Foundation and someone have signed some
> kind of settlement to get this change into the Linux kernel regardless of
> the technical issues, practicality or community and this is all for show
> anyway.

This is getting a bit far into conspiracy theories even for you, but if
I must I can categorically deny this.  The involvement of the Linux
foundation in this area is threefold:

     1. Pulling together OIN, OSAPA and a few other patent projects into
        a workable patent shield around Linux.
     2. Specifically for TomTom, assisting with their patent defence by
        explaining open source to their lawyers and also putting them in
        touch with OIN.
     3. For this patch, since there are legal issues which could harm
        the defence of any company to be discussed openly, the LF is
        paying the retainer for John Lanza to be available to any member
        of the community on a one on one basis, so that the discussions
        they engage in would be protected by client privilege (i.e. not
        subject to discovery).

In my book that amounts to trying to mitigate some of the impacts of the
US patent system on open source and linux rather than conspiracy.  You
could also see the "undisclosed settlement" in the TomTom case, with
virtually no repercussions for Tom Tom shipping Linux, as a reasonable
success of the above strategy.

> > If the patch had significant negative consequences for normal use then
> 
> You said yourself it can crash XP.
> 
> > If we get some real examples of devices that are badly affected
> 
> XP crashing by suprise if you are really really unlucky strikes me as a
> good example and one you provided.

OK, so this is a legitimate technical concern ... somewhat mitigated by
the fact that a lot of other FAT32 implementations also crash XP.

> You've also ignored entirely the fact that this isn't a VFAT file system
> so irrespective of whether this goes in it should not be used for mount
> -o vfat.
> 
> There is a clear end user expectation that vfat means "microsoft fat with
> extended names". By all means add support for "not vfat" but don't call
> it "vfat" as that is misleading an interface breakage.

The Microsoft FAT32 standard only says files with long names *may* be
visible on 8.3 FAT systems, it doesn't say *shall*.  Therefore, my
reading is that this patch is fully compliant with the FAT32 standard,
and thus *is* definitely what we call vfat in Linux.  Or are you
claiming that we've somehow extended the definition of vfat to mean
complies with FAT32 + makes 8.3 names available as well?

James

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Theodore Tso]

On Thu, Jul 02, 2009 at 02:56:07PM +0000, James Bottomley wrote:
>      3. For this patch, since there are legal issues which could harm
>         the defence of any company to be discussed openly, the LF is
>         paying the retainer for John Lanza to be available to any member
>         of the community on a one on one basis, so that the discussions
>         they engage in would be protected by client privilege (i.e. not
>         subject to discovery).

I believe what has been made available is that people who have some
detailed legal questions can contact John Lanza who can either respond
off-line via e-mail, schedule time for a phone call, and if really
necessary, it _can_ be possible to establish a formal client/attorney
privilege with John.  These different options are listed in order of
increasing cost to the Linux Foundation.  

Which option gets chosen will depend on the specific question and who
is asking the question; which is to say, setting up a formal
client/attorney privilege might be more appropriate for Hirofumi or
Linus, but obviously the LF can't afford to set up relationships where
client privilege would apply for any random LKML reader who is
randomly curious about the way things work.

So just to make it clear, with respect to expectations, simply sending
e-mail to John doesn't automatically set up a client relationship that
which protects the discussion under client/attorney privilege.
Setting that up *does* cost LF real money, so I would ask folks to be
considerate about the LF's resources!

BTW, Tridge has offered to be available for phone conversations, since
he's probably asked the various lawyers that have been consulted most
of the questions that people might have; so that's another option for
people who have legal questions that would be kinder and gentler to
the LF's budget.

And of course, keep in mind that this offer is really only for people
who want to clarification of any background legal issues, not to
debate the issue of whether or not the patch should go in.  That
policy discussion should happen openly, like any other proposed patch,
and ultimately it's Hirofumi-san's and/or Linus's call.

						- Ted

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Alan Cox]

> As a linux maintainer, you know we try very hard to encourage vendors
> not to carry long lived out of tree patches because of the complexity it
> causes for everyone.

For the non kernel cases with patent questions this isn't the pattern you
see. And for good reasons. A US patent compliant version of mplayer is
probably a jpeg viewer. It would be very harmful to the rest of the world
to gut the main package in that case.

For the kernel case the reality is this

If we include no patch the people concerned will remove the code from
their distributed source tarball

If we include a config option, the people concerned will *STILL* remove
the code from their distributed source tarball.

so the debate about out of tree patches is not as relevant as might at
first seem obvious. The simple problem is that the GPL says if you give
me the binaries I can ask for the sources which produced them. If those
sources contain the source to an algorithm where there are claims of
possible patent problems then the lawyers will prefer to play as safe as
possible and pull the sources that concern them before they even permit
anyone to type "make". A quick look at some packages from big US
based vendors will show you that is happening all the time to pull out
stuff as varied as mp3 playing and the openssh decss crypto mode.

[Other bits cut, conspiracy case point noted and its why you should not
post to linux-kernel with a fever temperature ;) ]

> > There is a clear end user expectation that vfat means "microsoft fat with
> > extended names". By all means add support for "not vfat" but don't call
> > it "vfat" as that is misleading an interface breakage.
> 
> The Microsoft FAT32 standard only says files with long names *may* be
> visible on 8.3 FAT systems, it doesn't say *shall*.  Therefore, my
> reading is that this patch is fully compliant with the FAT32 standard,
> and thus *is* definitely what we call vfat in Linux.  Or are you
> claiming that we've somehow extended the definition of vfat to mean
> complies with FAT32 + makes 8.3 names available as well?

I'm simply of the viewpoint that users expect certain behaviours and
compatibilities and that therefore it would be best if the new filesystem
variant had a new name. I'm not saying cp -r fs/foo fs/bar, just a new
mount name.

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[James Bottomley]

On Thu, 2009-07-02 at 19:12 +0100, Alan Cox wrote:
> > As a linux maintainer, you know we try very hard to encourage vendors
> > not to carry long lived out of tree patches because of the complexity it
> > causes for everyone.
> 
> For the non kernel cases with patent questions this isn't the pattern you
> see. And for good reasons. A US patent compliant version of mplayer is
> probably a jpeg viewer. It would be very harmful to the rest of the world
> to gut the main package in that case.

I'd agree *if* we were gutting the package ... and for that reason I
believe not accepting the first incarnation of this patch, which
basically lost the ability to create long filenames was correct.

I think the loss of functionality with this patch is pretty minor so it
is worth carrying it in the kernel.  Obviously, this type of thing has
to be decided on a case by case basis, so there's no precedent (at least
as I see it) being set here.

> For the kernel case the reality is this
> 
> If we include no patch the people concerned will remove the code from
> their distributed source tarball
> 
> If we include a config option, the people concerned will *STILL* remove
> the code from their distributed source tarball.
> 
> so the debate about out of tree patches is not as relevant as might at
> first seem obvious. The simple problem is that the GPL says if you give
> me the binaries I can ask for the sources which produced them. If those
> sources contain the source to an algorithm where there are claims of
> possible patent problems then the lawyers will prefer to play as safe as
> possible and pull the sources that concern them before they even permit
> anyone to type "make". A quick look at some packages from big US
> based vendors will show you that is happening all the time to pull out
> stuff as varied as mp3 playing and the openssh decss crypto mode.

So the advice of our lawyers is that in the US infringement only comes
from practising the patent ... i.e. having a working implementation.
Just giving someone source code for a method to practise it isn't
sufficient ... after all, that's really what a patent specification is
supposed to be: source code in english sufficient to reduce the patent
to practice.

> [Other bits cut, conspiracy case point noted and its why you should not
> post to linux-kernel with a fever temperature ;) ]

Gosh, sorry to hear that ... I find hot honey lemon and brandy useful
for this ... in extreme cases, I just skip the honey and lemon.

> > > There is a clear end user expectation that vfat means "microsoft fat with
> > > extended names". By all means add support for "not vfat" but don't call
> > > it "vfat" as that is misleading an interface breakage.
> > 
> > The Microsoft FAT32 standard only says files with long names *may* be
> > visible on 8.3 FAT systems, it doesn't say *shall*.  Therefore, my
> > reading is that this patch is fully compliant with the FAT32 standard,
> > and thus *is* definitely what we call vfat in Linux.  Or are you
> > claiming that we've somehow extended the definition of vfat to mean
> > complies with FAT32 + makes 8.3 names available as well?
> 
> I'm simply of the viewpoint that users expect certain behaviours and
> compatibilities and that therefore it would be best if the new filesystem
> variant had a new name. I'm not saying cp -r fs/foo fs/bar, just a new
> mount name.

It's a plausible argument.  My counter argument is that in the US with
this patch applied and your fs name change, it would cause all vfat
mounts to have to change mount type ... this is really a massive
surprise to all USB key users here and smoothing it over in the distros
would end up being quite a bit of work.  So, I'd favour, on the
principle of least surprise, keeping the mount name vfat for what is a
compliant implementation of the FAT32 filesystem, albeit missing one
small piece of functionality we previously had.

James

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Boaz Harrosh]

On 07/01/2009 01:50 PM, tridge@samba.org wrote:
> Hi Pavel,
> 
> We did of course consider that, and the changes to the patch to
> implement collision avoidance are relatively simple. We didn't do it
> as it would weaken the legal basis behind the patch. I'll leave it to
> John Lanza (the LF patent attorney) to expand on that if you want more
> information.
> 

You completely lost me here. And I thought I did understand the patent
and the fix.

what is the difference between.

short_name = rand(sid);
and
short_name = sid++;

Now if you would do
short_name = MD5(long_name);

That I understand since short_name is some function of long_name
but if I'm just inventing the short_name out of my hat. In what legal
system does it matter what is my random function I use?

> Cheers, Tridge

Boaz

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Boaz,

 > That I understand since short_name is some function of long_name
 > but if I'm just inventing the short_name out of my hat. In what legal
 > system does it matter what is my random function I use?

I like to defer to John Lanza for a more complete answer to this, but
I can tell you that it is not related to the choice of random
function.

I'd also suggest that if anyone wants to go into this in any depth
then private email with John Lanza is the best option.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[James Bottomley]

On Wed, 2009-07-01 at 14:48 +0300, Boaz Harrosh wrote:
> On 07/01/2009 01:50 PM, tridge@samba.org wrote:
> > Hi Pavel,
> > 
> > We did of course consider that, and the changes to the patch to
> > implement collision avoidance are relatively simple. We didn't do it
> > as it would weaken the legal basis behind the patch. I'll leave it to
> > John Lanza (the LF patent attorney) to expand on that if you want more
> > information.
> > 
> 
> You completely lost me here. And I thought I did understand the patent
> and the fix.
> 
> what is the difference between.
> 
> short_name = rand(sid);
> and
> short_name = sid++;
> 
> Now if you would do
> short_name = MD5(long_name);
> 
> That I understand since short_name is some function of long_name
> but if I'm just inventing the short_name out of my hat. In what legal
> system does it matter what is my random function I use?

We're sort of arguing moot technicalities here.  If you look at the way
the filename is constructed, given the constraints of a leading space
and a NULL, the need for a NULL padded leading slash extension and the
need to put control characters in the remaining bytes, we've only got 30
bits to play with, we're never going to avoid collisions in a space of
up to 31 bits.  Technically, a random function is at least as good at
collision avoidance as any deterministic solution ... and it's a lot
easier to code.

James

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

On Wed 2009-07-01 10:44:47, James Bottomley wrote:
> On Wed, 2009-07-01 at 14:48 +0300, Boaz Harrosh wrote:
> > On 07/01/2009 01:50 PM, tridge@samba.org wrote:
> > > Hi Pavel,
> > > 
> > > We did of course consider that, and the changes to the patch to
> > > implement collision avoidance are relatively simple. We didn't do it
> > > as it would weaken the legal basis behind the patch. I'll leave it to
> > > John Lanza (the LF patent attorney) to expand on that if you want more
> > > information.
> > > 
> > 
> > You completely lost me here. And I thought I did understand the patent
> > and the fix.
> > 
> > what is the difference between.
> > 
> > short_name = rand(sid);
> > and
> > short_name = sid++;
> > 
> > Now if you would do
> > short_name = MD5(long_name);
> > 
> > That I understand since short_name is some function of long_name
> > but if I'm just inventing the short_name out of my hat. In what legal
> > system does it matter what is my random function I use?
> 
> We're sort of arguing moot technicalities here.  If you look at the way
> the filename is constructed, given the constraints of a leading space
> and a NULL, the need for a NULL padded leading slash extension and the
> need to put control characters in the remaining bytes, we've only got 30
> bits to play with, we're never going to avoid collisions in a space of
> up to 31 bits.  Technically, a random function is at least as good at
> collision avoidance as any deterministic solution ... and it's a lot
> easier to code.

You could be deterministic and restrict maximum number of entries in
directory?

You could do random function but check if duplicate exists, and return
-EPERM if it does?

								Pavel 

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Jamie Lokier]

James Bottomley wrote:
> On Wed, 2009-07-01 at 14:48 +0300, Boaz Harrosh wrote:
> > On 07/01/2009 01:50 PM, tridge@samba.org wrote:
> > > Hi Pavel,
> > > 
> > > We did of course consider that, and the changes to the patch to
> > > implement collision avoidance are relatively simple. We didn't do it
> > > as it would weaken the legal basis behind the patch. I'll leave it to
> > > John Lanza (the LF patent attorney) to expand on that if you want more
> > > information.
> > > 
> > 
> > You completely lost me here. And I thought I did understand the patent
> > and the fix.
> > 
> > what is the difference between.
> > 
> > short_name = rand(sid);
> > and
> > short_name = sid++;
> > 
> > Now if you would do
> > short_name = MD5(long_name);
> > 
> > That I understand since short_name is some function of long_name
> > but if I'm just inventing the short_name out of my hat. In what legal
> > system does it matter what is my random function I use?
> 
> We're sort of arguing moot technicalities here.  If you look at the way
> the filename is constructed, given the constraints of a leading space
> and a NULL, the need for a NULL padded leading slash extension and the
> need to put control characters in the remaining bytes, we've only got 30
> bits to play with, we're never going to avoid collisions in a space of
> up to 31 bits.

> Technically, a random function is at least as good at
> collision avoidance as any deterministic solution ...

No, it isn't.

A deterministic value based on position in the directory, or by
checking for collisions and avoiding them, will _never_ collide,
provided you limit directories to no more than 2^30 entries, which is
reasonable for FAT.

Whereas a random value can collide.
That's a fundamental technical difference.

A quick read of the Birthday Problem page on Wikipedia leads to:

    With a directory of 1000 files, not especially rare with a camera
    or MP3 players, and 30-bit random numbers:

    The probably of a collision is 0.04% [1]

    If 10000 people each have a directory of 1000 files (not
    unreasonable given the huge number of people who use FAT media),
    the probability that any of them have a collision is approximately
    100%.

    
[1] perl -e '$d = 2.0**30; $n = 1000; $x = 1; for $k (1..$n-1) { $x *= (1 - $k/$d); } printf "Probability = %f%%\n", 100*(1-$x);'

In other words, using random values you are _guaranteeing_ collisions
for a few users.

So the argument comes down to: Does it matter if there are collisions?

Tridge's testing didn't blue screen Windows XP.
Tridge's testing did run a lot of operaitons.

But Tridge isn't 10000 people doing crazy diverse things with
different devices in all sorts of systematic but different patterns
over a period of years.

Given it's technically trivial to avoid collisions completely, and
there is some risk of breakage, even though it would be rare, there
had better be a good reason for not doing it.

-- Jamie

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Boaz Harrosh]

On 07/06/2009 11:41 PM, Jamie Lokier wrote:
> James Bottomley wrote:
>> On Wed, 2009-07-01 at 14:48 +0300, Boaz Harrosh wrote:
>>> On 07/01/2009 01:50 PM, tridge@samba.org wrote:
>>>> Hi Pavel,
>>>>
>>>> We did of course consider that, and the changes to the patch to
>>>> implement collision avoidance are relatively simple. We didn't do it
>>>> as it would weaken the legal basis behind the patch. I'll leave it to
>>>> John Lanza (the LF patent attorney) to expand on that if you want more
>>>> information.
>>>>
>>> You completely lost me here. And I thought I did understand the patent
>>> and the fix.
>>>
>>> what is the difference between.
>>>
>>> short_name = rand(sid);
>>> and
>>> short_name = sid++;
>>>
>>> Now if you would do
>>> short_name = MD5(long_name);
>>>
>>> That I understand since short_name is some function of long_name
>>> but if I'm just inventing the short_name out of my hat. In what legal
>>> system does it matter what is my random function I use?
>> We're sort of arguing moot technicalities here.  If you look at the way
>> the filename is constructed, given the constraints of a leading space
>> and a NULL, the need for a NULL padded leading slash extension and the
>> need to put control characters in the remaining bytes, we've only got 30
>> bits to play with, we're never going to avoid collisions in a space of
>> up to 31 bits.
> 
>> Technically, a random function is at least as good at
>> collision avoidance as any deterministic solution ...
> 
> No, it isn't.
> 
> A deterministic value based on position in the directory, or by
> checking for collisions and avoiding them, will _never_ collide,
> provided you limit directories to no more than 2^30 entries, which is

Exactly this is the key, find the real limit and enforce it.

> reasonable for FAT.
> 
> Whereas a random value can collide.
> That's a fundamental technical difference.
> 
> A quick read of the Birthday Problem page on Wikipedia leads to:
> 
>     With a directory of 1000 files, not especially rare with a camera
>     or MP3 players, and 30-bit random numbers:
> 
>     The probably of a collision is 0.04% [1]
> 
>     If 10000 people each have a directory of 1000 files (not
>     unreasonable given the huge number of people who use FAT media),
>     the probability that any of them have a collision is approximately
>     100%.
> 
>     
> [1] perl -e '$d = 2.0**30; $n = 1000; $x = 1; for $k (1..$n-1) { $x *= (1 - $k/$d); } printf "Probability = %f%%\n", 100*(1-$x);'
> 
> In other words, using random values you are _guaranteeing_ collisions
> for a few users.
> 

Thanks, I thought it was just me.

> So the argument comes down to: Does it matter if there are collisions?
> 
> Tridge's testing didn't blue screen Windows XP.
> Tridge's testing did run a lot of operaitons.
> 
> But Tridge isn't 10000 people doing crazy diverse things with
> different devices in all sorts of systematic but different patterns
> over a period of years.
> 

What? you say there are 10,000 people with cameras that are using Linux
in the world ;-)

> Given it's technically trivial to avoid collisions completely, and
> there is some risk of breakage, even though it would be rare, there
> had better be a good reason for not doing it.
> 

I wish the lawyers people would come forward, as promised,,and explain what
are the constraints on the short_name, given a long_name is present.
I'm still waiting for that private mail in my e-box. If the names do not
correspond at all but are both valid, why is that a problem?

> -- Jamie

Thanks
Boaz

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Jamie Lokier]

Boaz Harrosh wrote:
> I wish the lawyers people would come forward, as promised,,and explain what
> are the constraints on the short_name, given a long_name is present.
> I'm still waiting for that private mail in my e-box. If the names do not
> correspond at all but are both valid, why is that a problem?

To be fair Tridge said there is a reason.

The only problem is we don't know what it is...

-- Jamie

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Boaz Harrosh]

On 07/07/2009 02:25 PM, Jamie Lokier wrote:
> Boaz Harrosh wrote:
>> I wish the lawyers people would come forward, as promised,,and explain what
>> are the constraints on the short_name, given a long_name is present.
>> I'm still waiting for that private mail in my e-box. If the names do not
>> correspond at all but are both valid, why is that a problem?
> 
> To be fair Tridge said there is a reason.
> 
> The only problem is we don't know what it is...
> 

He also said someone will answer this in a private e-mail, I'm still
waiting.

> -- Jamie

Boaz

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Boaz,

 > He also said someone will answer this in a private e-mail, I'm still
 > waiting.

My apologies, I assumed John had sent you something.

John, can you please look at this?

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

Hi!

>  > Defaults should be back-compatible.
> 
> I would usually agree, but I think we have an unusual situation here,
> in some ways similar to a demonstrated security hole. The previous
> behaviour exposed a lot of Linux vendors to the possibility of an
> expensive legal fight.

I'd actually like to see all-out software patent war in the U.S. It
would make sure that software patents do not spread to the rest of
world. Bad for U.S.? Yes. Good for world? Yes!

>  > Users considering disabling this should understand that filesystem
>  > they write to will not be valid vfat filesystems, and may trigger bugs
>  > in some devices.
> 
> If we find any devices that exhibit any problems with this patch while
> it is in linux-next (and maybe linux-mm?) then this warning would
> indeed be appropriate. It no such devices are known then I think the
> warning is going a bit far. 

You already know that it breaks XP and older linuxes. So... what are
you arguing about?! Chkdsk marks it as invalid filesystem... not a
vfat.

If mickey$oft intentionally modified windows 8 to intentionally corrupt
ext3 filesystems so that linux oopses on accessing them, how'd you
like that?

>  > Why not use something like position in directory instead of random
>  > number?
> 
> We did of course consider that, and the changes to the patch to
> implement collision avoidance are relatively simple. We didn't do it
> as it would weaken the legal basis behind the patch. I'll leave it to

Consider it again. Or put fair 'this makes XP crash' warning in
kconfig.

								Pavel
PS: I find it bad that original patch description did not contain XP
crashing / chkdsk explanation.
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[tridge]

Hi Pavel,

 > You already know that it breaks XP and older linuxes. So... what are
 > you arguing about?! Chkdsk marks it as invalid filesystem... not a
 > vfat.

no, wrong on all 3 counts.

For the "breaks XP", see my previous message where I point out that I
have never been able to crash XP with this patch, only with other
varients on the patch that reduce the number of random bits we use.

The "breaks older linuxes" claim is also wrong. The patch I have
posted works fine with all older versions of the Linux kernel that I
have tested. If you know of a old version of the Linux kernel that
doesn't work with a filesystem written when this patch is enabled then
please let me know. I think you may be getting confused by the
discussion I had with Eric where I explained about the reasons for the
change in fat/dir.c. I won't repeat that long discussion again here,
but please re-read the reply I sent to Eric and if you still have
questions about it then please ask.

Finally, saying that "chkdsk marks it as an invalid filesystem" is not
correct. It only complains if there happens (with a very low
probability!) to be two files with the same 11 bytes in their
respective 8.3 entries. When it complains it renames one of the two
files.

That rename would not in itself be a problem at all, as changing the
8.3 entry would not affect the long filename, except that chkdsk has a
bug where it doesn't follow the Microsoft FAT specification. When you
rename a file on a FAT filesystem you are supposed to also update the
8 bit checksum field in the corresponding long filename entries. If
you don't do that update then you effectively strip off the long
filename. Current versions of Microsoft chkdsk neglect to update the
long filename checksum when renaming after detecting a duplicate 8.3
entry.

So it is Microsoft's chkdsk, not the patched Linux kernel, that is in
violation of the FAT spec. Luckily the probability of this bug
cropping up is quite small in practice.

Cheers, Tridge

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[Pavel Machek]

On Fri 2009-07-03 08:23:34, tridge@samba.org wrote:
> Hi Pavel,
> 
>  > You already know that it breaks XP and older linuxes. So... what are
>  > you arguing about?! Chkdsk marks it as invalid filesystem... not a
>  > vfat.
> 
> no, wrong on all 3 counts.
> 
> For the "breaks XP", see my previous message where I point out that I
> have never been able to crash XP with this patch, only with other
> varients on the patch that reduce the number of random bits we use.

...where you point out that probability of breakage is not too
high. But ammount of XP machines out there _is_ quite high, so someone
will hit it. You are deliberately breaking XP.

> The "breaks older linuxes" claim is also wrong. The patch I have
> posted works fine with all older versions of the Linux kernel that I
> have tested. If you know of a old version of the Linux kernel that
> doesn't work with a filesystem written when this patch is enabled then
> please let me know. I think you may be getting confused by the
> discussion I had with Eric where I explained about the reasons for the
> change in fat/dir.c. I won't repeat that long discussion again here,
> but please re-read the reply I sent to Eric and if you still have
> questions about it then please ask.

Ok, I guess I misunderstood. So what is the reason for pushing stable patch?

> Finally, saying that "chkdsk marks it as an invalid filesystem" is not
> correct. It only complains if there happens (with a very low
> probability!) to be two files with the same 11 bytes in their
> respective 8.3 entries. When it complains it renames one of the two
> files.

The "very low probability" is 50% for 30000 entries in directory,
AFAICT.

> That rename would not in itself be a problem at all, as changing the
> 8.3 entry would not affect the long filename, except that chkdsk has a
> bug where it doesn't follow the Microsoft FAT specification. When

Yeah, when map and reality disagree, trust the map. Not.

I don't care what M$ spec says, and you should not care, either. It is
M$ chkdsk that matters, and M$ VFAT implementation that matters.

> So it is Microsoft's chkdsk, not the patched Linux kernel, that is in
> violation of the FAT spec. Luckily the probability of this bug
> cropping up is quite small in practice.

Yes, explain that to the people. Actually, write it to the Kconfig.
"Turn this option ON to show bugs in M$ chkdsk". 

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[John Lanza]

I apologize to those of you who have already gotten this email.  I am
resending because I just learned the mailing list has been rejecting
my emails beacuse they are in HTML format.

johnl

On Wed, Jul 8, 2009 at 11:14 PM, John Lanza<jdlanza@gmail.com> wrote:
> All:
>
> I apologize for the lack of responsiveness on my part over the past couple
> of days.  Although I checked in dutifully over the weekend, most of Monday
> and Tuesday were consumed with a project.
>
> At this point, though, I have reached out to certain members of this
> community and provided at least an initial explanation.  I am willing to do
> the same for anyone interested -- simply reply back to this email and let me
> know of your interest.
>
> johnl
>
> On Tue, Jul 7, 2009 at 7:25 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>
>> Boaz Harrosh wrote:
>> > I wish the lawyers people would come forward, as promised,,and explain
>> > what
>> > are the constraints on the short_name, given a long_name is present.
>> > I'm still waiting for that private mail in my e-box. If the names do not
>> > correspond at all but are both valid, why is that a problem?
>>
>> To be fair Tridge said there is a reason.
>>
>> The only problem is we don't know what it is...
>>
>> -- Jamie
>
>

Re: [PATCH] Added CONFIG_VFAT_FS_DUALNAMES option

[John Lanza]

I apologize to those of you who have already gotten this email.  I am
resending because I just learned the mailing list has been rejecting
my emails beacuse they are in HTML format.

johnl

On Wed, Jul 8, 2009 at 11:14 PM, John Lanza<jdlanza@gmail.com> wrote:
> All:
>
> I apologize for the lack of responsiveness on my part over the past couple
> of days.  Although I checked in dutifully over the weekend, most of Monday
> and Tuesday were consumed with a project.
>
> At this point, though, I have reached out to certain members of this
> community and provided at least an initial explanation.  I am willing to do
> the same for anyone interested -- simply reply back to this email and let me
> know of your interest.
>
> johnl
>
> On Tue, Jul 7, 2009 at 7:25 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>
>> Boaz Harrosh wrote:
>> > I wish the lawyers people would come forward, as promised,,and explain
>> > what
>> > are the constraints on the short_name, given a long_name is present.
>> > I'm still waiting for that private mail in my e-box. If the names do not
>> > correspond at all but are both valid, why is that a problem?
>>
>> To be fair Tridge said there is a reason.
>>
>> The only problem is we don't know what it is...
>>
>> -- Jamie
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html