Re: [PATCH v2 1/8] tests: Make sure LANG is set properly for iso9660_test

["Thomas Schmitt" <scdbackup@gmx.net>]

Hi,

now i know that i should have asked for the test-suite.log file.

The problem turned out to be in Joliet, not in ISO 9660.
Because Joliet gets its file names and other texts encoded as UCS-2
16-bit characters it is indeed inavoidable to define the meaning of the
bytes in a Unix file name.
The UTF-8 sequence \303\251 has no meaning as ANSI_X3.4-1968 characters
(which are what we learned to love as 7-bit ASCII). So iconv fails.

My remedy proposal

  xorriso ... -as mkisofs -input-charset UTF-8 -output-charset UTF-8 ...

turned out to be a valid alternative to

  LANG=en_US.UTF-8

Using above charset options unconditionally would make obsolete the
warning in Glenn Washburn's patch at least for the xorriso runs:

  +elif [ -n "${LANG##*UTF*}" ]; then
  +   echo "WARNING: LANG=$LANG appears to not be unicode, international file test may fail."

It would still be beneficial to default LANG="" to LANG=en_US.UTF-8
because 7-bit ASCII is really an odd character interpretation for Unix
filenames. (HP-UX from 1986 maybe ...)

--------------------------------------------------------------------------

Noob question:
How can i force "make check" to do something at its second run ?

--------------------------------------------------------------------------
Long story:

On a Debian 10 with a freshly made GRUB git clone i see indeed a failure
of iso9660_test with mutilated volume id and a mutilated exotic filename
after

  make SUBDIRS= LANG= TESTS=iso9660_test check

On the same machine with the same xorriso, the same file name from the
log and the volume id taken from the script, i get with

  LANG= xorriso -as mkisofs -o test.iso -V "...exotic.characters..." test_tree

a somewhat confused xterm (or libreadline) but

  xorriso -indev test.iso -find / --

reports the exotic volume id and the exotic filename.

The "make ... check" run says on the second try
  make[3]: 'iso9660_test' is up to date.
but i can reproduce the problem by

  LANG= ./grub-fs-tester joliet

The trigger for the file name mutilation seems to be the mkisofs
emulation option for creating a Joliet tree
  -J
With default settings of mount(8) or xorriso's file name display the
problem becomes visible only if no Rock Ridge info was added to the
ISO 9660 tree. This is caused by xorriso command
  --rockridge off

Nevertheless file(1) and xorriso still see the exotic volume id.
Binary inspection shows that the ISO 9660 volume id (at offset 32768 + 40)
is originally exotic, but the Joliet volume id (at offset 34816 + 40) was
mutilated.

So obviously the problem sits in the Joliet name conversion to UCS-2.
It is inavoidable if the file names do not match the input character set
(defaulted from locale) because the Unix file names cannot be just copied
to Joliet directory records.

Now that i can reproduce the problem i can also confirm that my remedy
proposal would work as expected. It preservies the exotic names in the
Joliet tree and the exotic volume id in the Joliet supplementary volume
descriptor.


Have a nice day :)

Thomas