From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1mJLnw-0006el-Gg for mharc-grub-devel@gnu.org; Thu, 26 Aug 2021 16:16:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53782) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mJLnu-0006ec-Bj for grub-devel@gnu.org; Thu, 26 Aug 2021 16:16:50 -0400 Received: from mail-qt1-x82d.google.com ([2607:f8b0:4864:20::82d]:45882) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mJLnr-00037l-Ut for grub-devel@gnu.org; Thu, 26 Aug 2021 16:16:49 -0400 Received: by mail-qt1-x82d.google.com with SMTP id s32so3540427qtc.12 for ; Thu, 26 Aug 2021 13:16:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficientek-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=ugUeR4S0yrpg+cRZCiKyTmayccAFxf1QwEasa+ih3Wg=; b=IRyrjlK8qsFdZt/Y2ETUKM02K8Z7WJ+lIo6bP3y/YgX2zMBpPmJQ+c/YgSx+RFLhmi O5g5PbpXhT62tEfsf2sr2x6DXS4OUiHQJcLJnt24IT4yFjKcv+OQ3dDS9GDQAw32Ei+B PZIEBJrzTW+Kzvunj9lEFp0CqWcdOvrjeQ4O6C+EwhSUgg5BkNuy5fzTDy7ugd/m0u/P Et2/9KY5o5+2cOh3zVvmuS0aXh2ttLFV4Z5WwFNBpSgKWBxmDECPolJ+gAXRlTkF1onC IeuVscDI0RI2Qi34YQzj9owiY86dvmxEkJP9+d4KzSspJQdAIBKiZHTRMBhBZYl1wASI eB8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=ugUeR4S0yrpg+cRZCiKyTmayccAFxf1QwEasa+ih3Wg=; b=dyLzkK7zLzOm3wIHf0B5PIgue//7VKLGQhIQBGUVdkBHDI2xX9zCbDp03jOhg2CYDs zfx+YpSMy26EG9RI/daObkRXHP8Aifku6Ikaq9HS6l4KU8w/TLfT1R/5+3Leie7eiqmu +I8ZgQQ7jZa97I0E9X6NOrukj3ow60sRuG+4Dwj2AaCCSBL/R8u3x+NqSvf45qfXR/I9 WpQHtbV/YkOjM9RT9Dm1Hp/+kPMKFD+HxZxD2pUKOtVyVWb8cwLmUe8KhAdZu92k/2v0 70keugibqxJiJffncJiF6CZdvuynQb7LOlDuClQSUyouENqOlVNdiUFNuiWD2IRqsMf5 VCPg== X-Gm-Message-State: AOAM530YGZqLeEi0fKDtjvyTmoG0ODxzwR+fI6ViXs/iP+alMxdcCuej b2gSGRN0jgK4s2G9AuK3Ah9HmswOR9scMw== X-Google-Smtp-Source: ABdhPJzrVERsolTLt+AtanQ1ILWktof6i93BKozDuE1tMi62SyojenbQJsYSDM9DWAHxQvm2x3apRA== X-Received: by 2002:ac8:7090:: with SMTP id y16mr5200224qto.19.1630009006813; Thu, 26 Aug 2021 13:16:46 -0700 (PDT) Received: from crass-HP-ZBook-15-G2 ([199.58.83.11]) by smtp.gmail.com with ESMTPSA id b13sm237354qtp.26.2021.08.26.13.16.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 13:16:46 -0700 (PDT) Date: Thu, 26 Aug 2021 15:16:40 -0500 From: Glenn Washburn To: "Thomas Schmitt" Cc: The development of GNU GRUB Subject: Re: [PATCH v2 1/8] tests: Make sure LANG is set properly for iso9660_test Message-ID: <20210826151640.45f9fe81@crass-HP-ZBook-15-G2> In-Reply-To: <11704360906509513569@scdbackup.webframe.org> References: <25689360845655887275@scdbackup.webframe.org> <11704360906509513569@scdbackup.webframe.org> Reply-To: development@efficientek.com X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=2607:f8b0:4864:20::82d; envelope-from=development@efficientek.com; helo=mail-qt1-x82d.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: The development of GNU GRUB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Aug 2021 20:16:50 -0000 On Thu, 26 Aug 2021 19:49:37 +0200 "Thomas Schmitt" wrote: > Hi, > > now i know that i should have asked for the test-suite.log file. > > The problem turned out to be in Joliet, not in ISO 9660. > Because Joliet gets its file names and other texts encoded as UCS-2 > 16-bit characters it is indeed inavoidable to define the meaning of > the bytes in a Unix file name. > The UTF-8 sequence \303\251 has no meaning as ANSI_X3.4-1968 > characters (which are what we learned to love as 7-bit ASCII). So > iconv fails. > > My remedy proposal > > xorriso ... -as mkisofs -input-charset UTF-8 -output-charset UTF-8 > ... Do there need to be any UTF-8 locales installed (or any locales for that matter) for this to work? My guess is no. > turned out to be a valid alternative to > > LANG=en_US.UTF-8 > > Using above charset options unconditionally would make obsolete the > warning in Glenn Washburn's patch at least for the xorriso runs: > > +elif [ -n "${LANG##*UTF*}" ]; then > + echo "WARNING: LANG=$LANG appears to not be unicode, > international file test may fail." > > It would still be beneficial to default LANG="" to LANG=en_US.UTF-8 > because 7-bit ASCII is really an odd character interpretation for Unix > filenames. (HP-UX from 1986 maybe ...) Correct me if I'm wrong, but this logic about defaulting LANG is more general than the iso9660_test. Would you agree that this should be applied to the tests a a whole? If yes, then I should probably remove this patch altogether and put this code elsewhere. -------------------------------------------------------------------------- > > Noob question: > How can i force "make check" to do something at its second run ? I might be answering this question below. -------------------------------------------------------------------------- > Long story: > > On a Debian 10 with a freshly made GRUB git clone i see indeed a > failure of iso9660_test with mutilated volume id and a mutilated > exotic filename after > > make SUBDIRS= LANG= TESTS=iso9660_test check > > On the same machine with the same xorriso, the same file name from the > log and the volume id taken from the script, i get with > > LANG= xorriso -as mkisofs -o test.iso -V "...exotic.characters..." > test_tree > > a somewhat confused xterm (or libreadline) but > > xorriso -indev test.iso -find / -- > > reports the exotic volume id and the exotic filename. > > The "make ... check" run says on the second try > make[3]: 'iso9660_test' is up to date. > but i can reproduce the problem by > > LANG= ./grub-fs-tester joliet Yes the iso9660_test is up to date in the sense that it does not need to be generated from iso9660_test.in. However, "make check" should still be running the test. IOW, the tests are always rerun, there is no make rule that does not run tests based on them having been perviously run (that I know of). > The trigger for the file name mutilation seems to be the mkisofs > emulation option for creating a Joliet tree > -J > With default settings of mount(8) or xorriso's file name display the > problem becomes visible only if no Rock Ridge info was added to the > ISO 9660 tree. This is caused by xorriso command > --rockridge off > > Nevertheless file(1) and xorriso still see the exotic volume id. > Binary inspection shows that the ISO 9660 volume id (at offset 32768 > + 40) is originally exotic, but the Joliet volume id (at offset 34816 > + 40) was mutilated. > > So obviously the problem sits in the Joliet name conversion to UCS-2. > It is inavoidable if the file names do not match the input character > set (defaulted from locale) because the Unix file names cannot be > just copied to Joliet directory records. > > Now that i can reproduce the problem i can also confirm that my remedy > proposal would work as expected. It preservies the exotic names in the > Joliet tree and the exotic volume id in the Joliet supplementary > volume descriptor. As i understand it, your proposed change is entirely with in grub-fs-tester. Would you submit a patch with the changes you've outlined that would fix this issue for when LANG is empty? Based on the work you've done, I think it makes most sense for you to author the patch. And I'll test/review it. Glenn