RFH: fincore/count lsfd/column-xmode failing on sparc64

RFH: fincore/count lsfd/column-xmode failing on sparc64

[John Paul Adrian Glaubitz]

Hello!

On sparc64 running Linux 6.5.0, the following two tests are failing:

  2 tests of 283 FAILED

      fincore/count
      lsfd/column-xmode

The details are:

--- /dev/null   2023-11-08 19:15:02.955999947 +0000
+++ /home/glaubitz/util-linux/tests/output/fincore/count.err    2023-11-11 18:19:47.242570274 +0000
@@ -0,0 +1,4 @@
+[ NO EXCITING FILE ]
+fincore: failed to open: no_such_file: No such file or directory
+[ MULTIPLE FILES ]
+fincore: failed to open: no_such_file: No such file or directory
}}}-diff

      fincore: count file contents in core                   ... FAILED (fincore/count)
     libmount: options string: [11] set-empty                ... OK
     libfdisk: GPT: [02] all-defaults-with-typo              ... OK
          cal: Year 2147483646: [22] 1sw-year                ... OK
        fdisk: MBR - sort                                    ... OK (all 4 sub-tests PASSED)
          cal: September 1752: [21] 3sj-month                ... OK
        fdisk: sunlabel tests: [03] set-partition-sysid      ... OK
     libmount: utils: [10] ends-with                         ... OK
         lsfd: XMODE.m for classical system calls for multiplexing: [02] select ... OK
        lscpu: lscpu: [03] ppc-qemu                          ... OK
         lsfd: NAME and KNAME column                         ... OK (all 4 sub-tests PASSED)
         lsfd: TYPE and STTYPE column                        ... OK (all 4 sub-tests PASSED)
      findmnt: filter                                        ... OK (all 11 sub-tests PASSED)
        fdisk: MBR - dos mode: [03] set-partition-type       ... OK
     libmount: tab files: [10] find-target                   ... OK
         lsfd: timerfd associating alarm                     ... SKIPPED (no capability: WAKE_ALARM)
      hexdump: highlighting: [17] 2b_dec-3                   ... OK
       getopt: options: [14] quiet_option_short              ... OK
       column: table: [18] empty-column-at-eol2              ... OK
     libmount: options string: [12] set-new                  ... OK
 libsmartcols: fromfile: [10] padding-minout                 ... OK
     libmount: options string: [12] get-flg-linux            ... OK

diff-{{{
--- /home/glaubitz/util-linux/tests/expected/lsfd/column-xmode-MODE-x-bit       2023-11-11 18:12:22.478579711 +0000
+++ /home/glaubitz/util-linux/tests/output/lsfd/column-xmode-MODE-x-bit 2023-11-11 18:19:47.434570270 +0000
@@ -1,2 +1,3 @@
 r-x
+rwx
 MODE(x-bit):  0
}}}-diff

         lsfd: TCP sockets                                   ... OK
         lsfd: TCP6 sockets                                  ... OK
          cal: September 1752: [22] 1m-year                  ... OK
        lsblk: lsblk: [04] simple-lvm-state                  ... OK
       logger: errors: [07] tcp                              ... OK
          cal: Year 2147483646: [23] 1mjw-year               ... OK
     libmount: utils: [11] mountpoint                        ... OK
     libfdisk: GPT: [03] getattr                             ... OK
         lsfd: MODE and XMODE columns: [05] MODE-x-bit       ... FAILED (lsfd/column-xmode-MODE-x-bit)
      hexdump: highlighting: [18] 2b_dec-4                   ... OK
         lsfd: UNIX sockets made in a differenct net namespace ... SKIPPED (no root permissions)
     libmount: tab files: [11] find-target2                  ... OK
         lsfd: --pid option                                  ... SKIPPED (no root permissions)
     libmount: options string: [13] set-new-empty            ... OK
     libmount: options string: [13] get-flg-user             ... OK
          cal: September 1752: [23] 1s-year                  ... OK
        blkid: superblocks probing: [07] bcachefs            ... OK
       getopt: options: [15] quiet_output_option_long        ... OK
     libfdisk: mkpart-full: [04] mbr-err-nospace             ... OK
 libsmartcols: fromfile: [11] strictwidth                    ... OK
       column: table: [19] neg-1                             ... OK
     libmount: utils: [12] mountpoint-subdir                 ... OK
         lsfd: XMODE.m for classical system calls for multiplexing: [03] poll ... OK
      hexdump: highlighting: [19] 2b_dec-5                   ... OK
     libmount: options string                                ... OK (all 13 sub-tests PASSED)
          cal: Year 2147483646: [24] 1sjw-year               ... OK
       cramfs: fsck bad header: [05] pad-64K-be              ... OK
     libmount: tab files: [12] find-target3                  ... OK
          cal: September 1752: [24] 1mj-year                 ... OK
         lsfd: --summary option                              ... SKIPPED (missing in PATH: ps)
     libmount: options string: [14] set-new-end              ... OK
       logger: errors: [08] multi-line                       ... OK
     libmount: utils: [13] mountpoint-root                   ... OK
        lsblk: lsblk: [05] simple-lvm-topo                   ... OK
          cal: Year 2147483646                               ... OK (all 24 sub-tests PASSED)

diff-{{{
--- /home/glaubitz/util-linux/tests/expected/lsfd/column-xmode-XMODE-x-bit      2023-11-11 18:12:22.478579711 +0000
+++ /home/glaubitz/util-linux/tests/output/lsfd/column-xmode-XMODE-x-bit        2023-11-11 18:19:47.826570261 +0000
@@ -1,2 +1,3 @@
 r-x---
+rwx---
 XMODE(x-bit):  0
}}}-diff

Does anyone have a clue how to address these issues? I'm not quite sure what these
tests actually test.

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[Thomas Weißschuh]

Hi,

Nov 11, 2023 19:23:20 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>:

> Hello!
>
> On sparc64 running Linux 6.5.0, the following two tests are failing:

Which version of util-linux is this?

For the fincore test:
Can you check with strace if it is using the mincore or cachestat syscall?

>   2 tests of 283 FAILED
>
>       fincore/count
>       lsfd/column-xmode
>

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[John Paul Adrian Glaubitz]

Hi Thomas,

On Sat, 2023-11-11 at 19:37 +0100, Thomas Weißschuh wrote:
> Hi,
> 
> Nov 11, 2023 19:23:20 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>:
> 
> > Hello!
> > 
> > On sparc64 running Linux 6.5.0, the following two tests are failing:
> 
> Which version of util-linux is this?

This is on git master, so e0f3b02d5.

> For the fincore test:
> Can you check with strace if it is using the mincore or cachestat syscall?

I can't seem to spot neither in the strace log:

> https://people.debian.org/~glaubitz/fincore-strace-sparce64.log

Log created with "strace -o ~/fincore-strace-sparce64.log ./tests/ts/fincore/count".

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[Thomas Weißschuh]

Nov 11, 2023 19:46:08 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>:

> Hi Thomas,
>
> On Sat, 2023-11-11 at 19:37 +0100, Thomas Weißschuh wrote:
>> Hi,
>>
>> Nov 11, 2023 19:23:20 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>:
>>
>>> Hello!
>>>
>>> On sparc64 running Linux 6.5.0, the following two tests are failing:
>>
>> Which version of util-linux is this?
>
> This is on git master, so e0f3b02d5.
>
>> For the fincore test:
>> Can you check with strace if it is using the mincore or cachestat syscall?
>
> I can't seem to spot neither in the strace log:
>
>> https://people.debian.org/~glaubitz/fincore-strace-sparce64.log
>
> Log created with "strace -o ~/fincore-strace-sparce64.log ./tests/ts/fincore/count".

The log is missing the calls to the fincore binary itself.
Can you add a "-f"?

Did it work with an older Kernel?
Since 6.5 it should use cachestat. Before it was mincore().
Or disable cachestat via "enosys" from util-linux.

I can also take a look on Monday.

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[John Paul Adrian Glaubitz]

On Sat, 2023-11-11 at 20:10 +0100, Thomas Weißschuh wrote:
> > Log created with "strace -o ~/fincore-strace-sparce64.log ./tests/ts/fincore/count".
> 
> The log is missing the calls to the fincore binary itself.
> Can you add a "-f"?

Ah, sorry. Here you go:

> https://people.debian.org/~glaubitz/fincore-strace-sparce64-fork.log.gz

Compressed the file because it's considerably larger now.

> Did it work with an older Kernel?
> Since 6.5 it should use cachestat. Before it was mincore().
> Or disable cachestat via "enosys" from util-linux.

Haven't tested with an older kernel. Will check.

> I can also take a look on Monday.

Great, thank you!

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[Masatake YAMATO]

Thank you for reporint the issue about lsfd.

    diff-{{{
    --- /home/glaubitz/util-linux/tests/expected/lsfd/column-xmode-MODE-x-bit       2023-11-11 18:12:22.478579711 +0000
    +++ /home/glaubitz/util-linux/tests/output/lsfd/column-xmode-MODE-x-bit 2023-11-11 18:19:47.434570270 +0000
    @@ -1,2 +1,3 @@
     r-x
    +rwx
     MODE(x-bit):  0
    }}}-diff

Could you show the outout of "pmap $$" on the target platform?

On my PC (Fedora 38 running on x86_64):

    929547:   zsh
    000055e86f2d7000     92K r---- zsh
    000055e86f2ee000    716K r-x-- zsh
    000055e86f3a1000     88K r---- zsh
    ...

The failed test case is highly depends on "r-x--" iin
the line "000055e86f2ee000    716K r-x-- zsh".

I guess, on your environment, the field is "rwx--".

If, my guessing is correct, I must fix the test case.

Masatake YAMATO

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[John Paul Adrian Glaubitz]

Hi Masatake!

On Mon, 2023-11-13 at 02:54 +0900, Masatake YAMATO wrote:
> Could you show the outout of "pmap $$" on the target platform?

Here you go:

glaubitz@stadler:~$ pmap $$
1490335:   -bash
0000010000000000   1248K r-x-- bash
00000100002fa000     24K r---- bash
0000010000300000     48K rwx-- bash
000001000030c000     40K rwx--   [ anon ]
0000010000316000    520K rwx--   [ anon ]
000007feffa22000    136K rw---   [ stack ]
fff8000100000000    160K r-x-- ld-linux.so.2
fff8000100028000     16K rw---   [ anon ]
fff800010002c000     32K r--s- gconv-modules.cache
fff80001001fe000      8K r---- ld-linux.so.2
fff8000100200000      8K rw--- ld-linux.so.2
fff8000100426000      8K r----   [ anon ]
fff8000100428000      8K r-x--   [ anon ]
fff8000100500000    184K r-x-- libtinfo.so.6.4
fff800010052e000   1848K ----- libtinfo.so.6.4
fff80001006fc000     16K r---- libtinfo.so.6.4
fff8000100700000      8K rwx-- libtinfo.so.6.4
fff8000100800000   1584K r-x-- libc.so.6
fff800010098c000   1472K ----- libc.so.6
fff8000100afc000     16K r---- libc.so.6
fff8000100b00000     16K rwx-- libc.so.6
fff8000100b04000     48K rwx--   [ anon ]
fff8000100b10000   2984K r---- locale-archive
fff8000100e00000     32K r-x-- libnss_compat.so.2
fff8000100e08000   2008K ----- libnss_compat.so.2
fff8000100ffe000      8K r---- libnss_compat.so.2
fff8000101000000      8K rwx-- libnss_compat.so.2
fff8000101100000     24K r-x-- libnss_db-2.2.3.so
fff8000101106000   2016K ----- libnss_db-2.2.3.so
fff80001012fe000      8K r---- libnss_db-2.2.3.so
fff8000101300000      8K rwx-- libnss_db-2.2.3.so
fff8000101400000   1624K r-x-- libdb-5.3.so
fff8000101596000   1424K ----- libdb-5.3.so
fff80001016fa000     24K r---- libdb-5.3.so
fff8000101700000     56K rwx-- libdb-5.3.so
 total            17672K
glaubitz@stadler:~$

and for zsh:

stadler% pmap $$
1490470:   zsh
0000010000000000    792K r-x-- zsh
00000100001fe000      8K r---- zsh
0000010000200000     32K rwx-- zsh
0000010000208000     80K rwx--   [ anon ]
000001000021c000    408K rwx--   [ anon ]
000007feff92a000    136K rw---   [ stack ]
fff8000100000000    160K r-x-- ld-linux.so.2
fff8000100028000     32K rw---   [ anon ]
fff8000100032000     16K rw---   [ anon ]
fff8000100038000     32K r--s- gconv-modules.cache
fff8000100066000      8K r--s- Newuser.zwc
fff8000100082000      8K r--s- Completion.zwc
fff80001001fe000      8K r---- ld-linux.so.2
fff8000100200000      8K rw--- ld-linux.so.2
fff800010032c000      8K r----   [ anon ]
fff800010032e000      8K r-x--   [ anon ]
fff8000100400000    392K r-x-- libpcre2-8.so.0.11.2
fff8000100462000   1648K ----- libpcre2-8.so.0.11.2
fff80001005fe000      8K r---- libpcre2-8.so.0.11.2
fff8000100600000      8K rwx-- libpcre2-8.so.0.11.2
fff8000100700000     56K r-x-- libgdbm.so.6.0.0
fff800010070e000   1984K ----- libgdbm.so.6.0.0
fff80001008fe000      8K r---- libgdbm.so.6.0.0
fff8000100900000      8K rwx-- libgdbm.so.6.0.0
fff8000100a00000     40K r-x-- libcap.so.2.66
fff8000100a0a000   2000K ----- libcap.so.2.66
fff8000100bfe000      8K r---- libcap.so.2.66
fff8000100c00000      8K rwx-- libcap.so.2.66
fff8000100d00000    208K r-x-- libncursesw.so.6.4
fff8000100d34000   1832K ----- libncursesw.so.6.4
fff8000100efe000      8K r---- libncursesw.so.6.4
fff8000100f00000     16K rwx-- libncursesw.so.6.4
fff8000101000000    184K r-x-- libtinfo.so.6.4
fff800010102e000   1848K ----- libtinfo.so.6.4
fff80001011fc000     16K r---- libtinfo.so.6.4
fff8000101200000      8K rwx-- libtinfo.so.6.4
fff8000101300000    736K r-x-- libm.so.6
fff80001013b8000   1304K ----- libm.so.6
fff80001014fe000      8K r---- libm.so.6
fff8000101500000      8K rwx-- libm.so.6
fff8000101600000   1584K r-x-- libc.so.6
fff800010178c000   1472K ----- libc.so.6
fff80001018fc000     16K r---- libc.so.6
fff8000101900000     16K rwx-- libc.so.6
fff8000101904000     48K rwx--   [ anon ]
fff8000101910000   2984K r---- locale-archive
fff8000101c00000     32K r-x-- libnss_compat.so.2
fff8000101c08000   2008K ----- libnss_compat.so.2
fff8000101dfe000      8K r---- libnss_compat.so.2
fff8000101e00000      8K rwx-- libnss_compat.so.2
fff8000101f00000     24K r-x-- libnss_db-2.2.3.so
fff8000101f06000   2016K ----- libnss_db-2.2.3.so
fff80001020fe000      8K r---- libnss_db-2.2.3.so
fff8000102100000      8K rwx-- libnss_db-2.2.3.so
fff8000102200000   1624K r-x-- libdb-5.3.so
fff8000102396000   1424K ----- libdb-5.3.so
fff80001024fa000     24K r---- libdb-5.3.so
fff8000102500000     56K rwx-- libdb-5.3.so
fff8000102600000      8K r-x-- terminfo.so
fff8000102602000   2032K ----- terminfo.so
fff80001027fe000      8K r---- terminfo.so
fff8000102800000      8K rwx-- terminfo.so
fff8000102900000    296K r-x-- zle.so
fff800010294a000   1744K ----- zle.so
fff8000102afe000      8K r---- zle.so
fff8000102b00000     40K rwx-- zle.so
fff8000102b0a000      8K rwx--   [ anon ]
fff8000102c00000    144K r-x-- complete.so
fff8000102c24000   1896K ----- complete.so
fff8000102dfe000      8K r---- complete.so
fff8000102e00000     16K rwx-- complete.so
fff8000102f00000     32K r-x-- zutil.so
fff8000102f08000   2008K ----- zutil.so
fff80001030fe000      8K r---- zutil.so
fff8000103100000      8K rwx-- zutil.so
fff8000103200000     16K r-x-- files.so
fff8000103204000   2024K ----- files.so
fff80001033fe000      8K r---- files.so
fff8000103400000      8K rwx-- files.so
fff8000103500000     40K r-x-- parameter.so
fff800010350a000   2000K ----- parameter.so
fff80001036fe000      8K r---- parameter.so
fff8000103700000      8K rwx-- parameter.so
 total            39832K
stadler%

> On my PC (Fedora 38 running on x86_64):
> 
>     929547:   zsh
>     000055e86f2d7000     92K r---- zsh
>     000055e86f2ee000    716K r-x-- zsh
>     000055e86f3a1000     88K r---- zsh
>     ...
> 
> The failed test case is highly depends on "r-x--" iin
> the line "000055e86f2ee000    716K r-x-- zsh".
> 
> I guess, on your environment, the field is "rwx--".

It currently looks like this for bash:

1490335:   -bash
0000010000000000   1248K r-x-- bash
00000100002fa000     24K r---- bash
0000010000300000     48K rwx-- bash

and for zsh:

1490470:   zsh
0000010000000000    792K r-x-- zsh
00000100001fe000      8K r---- zsh
0000010000200000     32K rwx-- zsh

> If, my guessing is correct, I must fix the test case.

That would be great. Thank you!

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[Thomas Weißschuh]

On 2023-11-12 08:53:28+0100, John Paul Adrian Glaubitz wrote:
> On Sat, 2023-11-11 at 20:10 +0100, Thomas Weißschuh wrote:
> [..]
> > I can also take a look on Monday.
> 
> Great, thank you!

What's your system's page size?

./test_sysinfo


Do you have a VM image available I could use?
I was not able to make the available Debian images work and could not
reproduce the issue on the Ubuntu images.


Thomas

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[John Paul Adrian Glaubitz]

Hi Thomas!

On Mon, 2023-11-13 at 18:50 +0100, Thomas Weißschuh wrote:
> What's your system's page size?
> 
> ./test_sysinfo

glaubitz@stadler:~/util-linux$ ./test_sysinfo 
       WORDSIZE: 64
       pagesize: 8192
        INT_MAX: 2147483647
       UINT_MAX: 4294967295
       LONG_MAX: 9223372036854775807
      ULONG_MAX: 18446744073709551615
    ULONG_MAX32: 4294967295
     UINT64_MAX: 18446744073709551615
     byte-order: BE
      wcsspn-ok: 1
      enotty-ok: 1
      fsopen-ok: 1
glaubitz@stadler:~/util-linux$

> Do you have a VM image available I could use?
> I was not able to make the available Debian images work and could not
> reproduce the issue on the Ubuntu images.

This is on sparc64, not amd64.

I installed with this image:

> https://cdimage.debian.org/cdimage/ports/snapshots/2022-03-24/debian-11.0.0-sparc64-NETINST-1.iso

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[Thomas Weißschuh]

On 2023-11-13 19:06:19+0100, John Paul Adrian Glaubitz wrote:
> On Mon, 2023-11-13 at 18:50 +0100, Thomas Weißschuh wrote:
> > What's your system's page size?
> > 
> > ./test_sysinfo
> 
> glaubitz@stadler:~/util-linux$ ./test_sysinfo 
>        WORDSIZE: 64
>        pagesize: 8192
>         INT_MAX: 2147483647
>        UINT_MAX: 4294967295
>        LONG_MAX: 9223372036854775807
>       ULONG_MAX: 18446744073709551615
>     ULONG_MAX32: 4294967295
>      UINT64_MAX: 18446744073709551615
>      byte-order: BE
>       wcsspn-ok: 1
>       enotty-ok: 1
>       fsopen-ok: 1
> glaubitz@stadler:~/util-linux$

8k page size should explain it.

The test output depends on the page size and has known test outputs for
a few pagesizes and a fallback output.

The fallback output was broken.

Can you test this PR?
https://github.com/util-linux/util-linux/pull/2581


Thomas

Re: RFH: fincore/count lsfd/column-xmode failing on sparc64

[John Paul Adrian Glaubitz]

Hi Thomas!

On Mon, 2023-11-13 at 21:10 +0100, Thomas Weißschuh wrote:
> On 2023-11-13 19:06:19+0100, John Paul Adrian Glaubitz wrote:
> > On Mon, 2023-11-13 at 18:50 +0100, Thomas Weißschuh wrote:
> > > What's your system's page size?
> > > 
> > > ./test_sysinfo
> > 
> > glaubitz@stadler:~/util-linux$ ./test_sysinfo 
> >        WORDSIZE: 64
> >        pagesize: 8192
> >         INT_MAX: 2147483647
> >        UINT_MAX: 4294967295
> >        LONG_MAX: 9223372036854775807
> >       ULONG_MAX: 18446744073709551615
> >     ULONG_MAX32: 4294967295
> >      UINT64_MAX: 18446744073709551615
> >      byte-order: BE
> >       wcsspn-ok: 1
> >       enotty-ok: 1
> >       fsopen-ok: 1
> > glaubitz@stadler:~/util-linux$
> 
> 8k page size should explain it.
> 
> The test output depends on the page size and has known test outputs for
> a few pagesizes and a fallback output.
> 
> The fallback output was broken.
> 
> Can you test this PR?
> https://github.com/util-linux/util-linux/pull/2581

I can confirm that the test fincore/count no longer fails on sparc64 with
your suggested patch applied.

Tested-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913