[PATCH 0/1] Possible bug in zram on ppc64le on vfat

[PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Petr Vorel]

Hi all,

following bug is trying to workaround an error on ppc64le, where
zram01.sh LTP test (there is also kernel selftest
tools/testing/selftests/zram/zram01.sh, but LTP test got further
updates) has often mem_used_total 0 although zram is already filled.

Patch tries to repeatedly read /sys/block/zram*/mm_stat for 1 sec,
waiting for mem_used_total > 0. The question if this is expected and
should be workarounded or a bug which should be fixed.

REPRODUCE THE ISSUE
Quickest way to install only zram tests and their dependencies:
make autotools && ./configure && for i in testcases/lib/ testcases/kernel/device-drivers/zram/; do cd $i && make -j$(getconf _NPROCESSORS_ONLN) && make install && cd -; done

Run the test (only on vfat)
PATH="/opt/ltp/testcases/bin:$PATH" LTP_SINGLE_FS_TYPE=vfat zram01.sh

Petr Vorel (1):
  zram01.sh: Workaround division by 0 on vfat on ppc64le

 .../kernel/device-drivers/zram/zram01.sh      | 27 +++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

-- 
2.38.0

[PATCH 1/1] zram01.sh: Workaround division by 0 on vfat on ppc64le

[Petr Vorel]

Repeatedly read /sys/block/zram*/mm_stat for 1 sec. This should fix bug
on ppc64le on stable kernels, where mem_used_total is often 0.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 .../kernel/device-drivers/zram/zram01.sh      | 27 +++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/testcases/kernel/device-drivers/zram/zram01.sh b/testcases/kernel/device-drivers/zram/zram01.sh
index 58d233f91..76a8ccab4 100755
--- a/testcases/kernel/device-drivers/zram/zram01.sh
+++ b/testcases/kernel/device-drivers/zram/zram01.sh
@@ -105,6 +105,26 @@ zram_mount()
 	tst_res TPASS "mount of zram device(s) succeeded"
 }
 
+read_mem_used_total()
+{
+	echo $(awk '{print $3}' $1)
+}
+
+# Reads /sys/block/zram*/mm_stat until mem_used_total is not 0.
+loop_read_mem_used_total()
+{
+	local file="$1"
+	local mem_used_total
+
+	tst_res TINFO "$file"
+	cat $file >&2
+
+	mem_used_total=$(read_mem_used_total $file)
+	[ "$mem_used_total" -eq 0 ] && return 1
+
+	return 0
+}
+
 zram_fill_fs()
 {
 	local mem_used_total
@@ -133,9 +153,12 @@ zram_fill_fs()
 			continue
 		fi
 
-		mem_used_total=`awk '{print $3}' "/sys/block/zram$i/mm_stat"`
+		TST_RETRY_FUNC "loop_read_mem_used_total /sys/block/zram$i/mm_stat" 0
+		mem_used_total=$(read_mem_used_total /sys/block/zram$i/mm_stat)
+		tst_res TINFO "mem_used_total: $mem_used_total"
+
 		v=$((100 * 1024 * $b / $mem_used_total))
-		r=`echo "scale=2; $v / 100 " | bc`
+		r=$(echo "scale=2; $v / 100 " | bc)
 
 		if [ "$v" -lt 100 ]; then
 			tst_res TFAIL "compression ratio: $r:1"
-- 
2.38.0

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Minchan Kim]

On Mon, Nov 07, 2022 at 08:11:35PM +0100, Petr Vorel wrote:
> Hi all,
> 
> following bug is trying to workaround an error on ppc64le, where
> zram01.sh LTP test (there is also kernel selftest
> tools/testing/selftests/zram/zram01.sh, but LTP test got further
> updates) has often mem_used_total 0 although zram is already filled.

Hi, Petr,

Is it happening on only ppc64le?

Is it a new regression? What kernel version did you use?

Actually, mem_used_total indicates how many *physical memory* were
currently used to keep original data size.

However, if the test data is repeated pattern of unsigned long
(https://github.com/torvalds/linux/blob/master/drivers/block/zram/zram_drv.c#L210)
zram doesn't allocate the physical memory but just mark the unsigned long's value
in meta area for decompression later.

Not sure you hit the this case.

> 
> Patch tries to repeatedly read /sys/block/zram*/mm_stat for 1 sec,
> waiting for mem_used_total > 0. The question if this is expected and
> should be workarounded or a bug which should be fixed.
> 
> REPRODUCE THE ISSUE
> Quickest way to install only zram tests and their dependencies:
> make autotools && ./configure && for i in testcases/lib/ testcases/kernel/device-drivers/zram/; do cd $i && make -j$(getconf _NPROCESSORS_ONLN) && make install && cd -; done
> 
> Run the test (only on vfat)
> PATH="/opt/ltp/testcases/bin:$PATH" LTP_SINGLE_FS_TYPE=vfat zram01.sh
> 
> Petr Vorel (1):
>   zram01.sh: Workaround division by 0 on vfat on ppc64le
> 
>  .../kernel/device-drivers/zram/zram01.sh      | 27 +++++++++++++++++--
>  1 file changed, 25 insertions(+), 2 deletions(-)
> 
> -- 
> 2.38.0
> 

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Petr Vorel]

Hi Minchan,

> On Mon, Nov 07, 2022 at 08:11:35PM +0100, Petr Vorel wrote:
> > Hi all,

> > following bug is trying to workaround an error on ppc64le, where
> > zram01.sh LTP test (there is also kernel selftest
> > tools/testing/selftests/zram/zram01.sh, but LTP test got further
> > updates) has often mem_used_total 0 although zram is already filled.

> Hi, Petr,

> Is it happening on only ppc64le?
I haven't seen it on other archs (x86_64, aarch64).

> Is it a new regression? What kernel version did you use?
Found on openSUSE kernel, which uses stable kernel releases 6.0.x.
It's probably much older, first I've seen it some years ago (I'm not able to find kernel version), but it was random. Now it's much more common.

Test runs on VM (I can give qemu command or whatever you need to know about it)
I'll try to verify it on some bare metal ppc64le.

> Actually, mem_used_total indicates how many *physical memory* were
> currently used to keep original data size.

> However, if the test data is repeated pattern of unsigned long
> (https://github.com/torvalds/linux/blob/master/drivers/block/zram/zram_drv.c#L210)
> zram doesn't allocate the physical memory but just mark the unsigned long's value
> in meta area for decompression later.

> Not sure you hit the this case.
Thanks for a hint, I'll try to debug it.

Kind regards,
Petr

> > Patch tries to repeatedly read /sys/block/zram*/mm_stat for 1 sec,
> > waiting for mem_used_total > 0. The question if this is expected and
> > should be workarounded or a bug which should be fixed.

> > REPRODUCE THE ISSUE
> > Quickest way to install only zram tests and their dependencies:
> > make autotools && ./configure && for i in testcases/lib/ testcases/kernel/device-drivers/zram/; do cd $i && make -j$(getconf _NPROCESSORS_ONLN) && make install && cd -; done

> > Run the test (only on vfat)
> > PATH="/opt/ltp/testcases/bin:$PATH" LTP_SINGLE_FS_TYPE=vfat zram01.sh

> > Petr Vorel (1):
> >   zram01.sh: Workaround division by 0 on vfat on ppc64le

> >  .../kernel/device-drivers/zram/zram01.sh      | 27 +++++++++++++++++--
> >  1 file changed, 25 insertions(+), 2 deletions(-)

> > -- 
> > 2.38.0

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Petr Vorel]

> Hi Minchan,

> > On Mon, Nov 07, 2022 at 08:11:35PM +0100, Petr Vorel wrote:
> > > Hi all,

> > > following bug is trying to workaround an error on ppc64le, where
> > > zram01.sh LTP test (there is also kernel selftest
> > > tools/testing/selftests/zram/zram01.sh, but LTP test got further
> > > updates) has often mem_used_total 0 although zram is already filled.

> > Hi, Petr,

> > Is it happening on only ppc64le?
> I haven't seen it on other archs (x86_64, aarch64).

> > Is it a new regression? What kernel version did you use?
> Found on openSUSE kernel, which uses stable kernel releases 6.0.x.
> It's probably much older, first I've seen it some years ago (I'm not able to find kernel version), but it was random. Now it's much more common.

I tested it on bare metal machine with some older SLES kernel (based on 4.12.14,
with thousands of patches) and it fails:

# PATH="/opt/ltp/testcases/bin:$PATH" LTP_SINGLE_FS_TYPE=vfat zram01.sh
...
zram01 5 TINFO: make vfat filesystem on /dev/zram0
zram01 5 TPASS: zram_makefs succeeded
zram01 6 TINFO: mount /dev/zram0
zram01 6 TPASS: mount of zram device(s) succeeded
zram01 7 TINFO: filling zram0 (it can take long time)
zram01 7 TPASS: zram0 was filled with '25568' KB
/opt/ltp/testcases/bin/zram01.sh: line 137: 100 * 1024 * 25568 / 0: division by 0 (error token is "0")
...

My patch does not help, obviously the value does not change.

zram01 5 TINFO: make vfat filesystem on /dev/zram1
zram01 5 TPASS: zram_makefs succeeded
zram01 6 TINFO: mount /dev/zram1
zram01 6 TPASS: mount of zram device(s) succeeded
zram01 7 TINFO: filling zram1 (it can take long time)
zram01 7 TPASS: zram1 was filled with '25568' KB
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TINFO: /sys/block/zram1/mm_stat
15859712        0        0 26214400   196608      242        0
zram01 7 TBROK: "loop_read_mem_used_total /sys/block/zram1/mm_stat" timed out

This is just to demonstrate, that the problem has been here long time,
and it's not QEMU related (although it could be theoretically related to
openSUSE/SLES user space, but it's very unlikely).

I'll debug if page_same_filled() is being called on mainline kernel.

Kind regards,
Petr

> Test runs on VM (I can give qemu command or whatever you need to know about it)
> I'll try to verify it on some bare metal ppc64le.

> > Actually, mem_used_total indicates how many *physical memory* were
> > currently used to keep original data size.

> > However, if the test data is repeated pattern of unsigned long
> > (https://github.com/torvalds/linux/blob/master/drivers/block/zram/zram_drv.c#L210)
> > zram doesn't allocate the physical memory but just mark the unsigned long's value
> > in meta area for decompression later.

> > Not sure you hit the this case.
> Thanks for a hint, I'll try to debug it.

> Kind regards,
> Petr

> > > Patch tries to repeatedly read /sys/block/zram*/mm_stat for 1 sec,
> > > waiting for mem_used_total > 0. The question if this is expected and
> > > should be workarounded or a bug which should be fixed.

> > > REPRODUCE THE ISSUE
> > > Quickest way to install only zram tests and their dependencies:
> > > make autotools && ./configure && for i in testcases/lib/ testcases/kernel/device-drivers/zram/; do cd $i && make -j$(getconf _NPROCESSORS_ONLN) && make install && cd -; done

> > > Run the test (only on vfat)
> > > PATH="/opt/ltp/testcases/bin:$PATH" LTP_SINGLE_FS_TYPE=vfat zram01.sh

> > > Petr Vorel (1):
> > >   zram01.sh: Workaround division by 0 on vfat on ppc64le

> > >  .../kernel/device-drivers/zram/zram01.sh      | 27 +++++++++++++++++--
> > >  1 file changed, 25 insertions(+), 2 deletions(-)

> > > -- 
> > > 2.38.0

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Sergey Senozhatsky]

On (22/11/07 23:42), Petr Vorel wrote:
[..]
> 15859712        0        0 26214400   196608      242        0
> zram01 7 TINFO: /sys/block/zram1/mm_stat
> 15859712        0        0 26214400   196608      242        0
> zram01 7 TINFO: /sys/block/zram1/mm_stat
> 15859712        0        0 26214400   196608      242        0
> zram01 7 TINFO: /sys/block/zram1/mm_stat
> 15859712        0        0 26214400   196608      242        0
> zram01 7 TINFO: /sys/block/zram1/mm_stat
> 15859712        0        0 26214400   196608      242        0
> zram01 7 TINFO: /sys/block/zram1/mm_stat
> 15859712        0        0 26214400   196608      242        0
> zram01 7 TINFO: /sys/block/zram1/mm_stat
> 15859712        0        0 26214400   196608      242        0
> zram01 7 TBROK: "loop_read_mem_used_total /sys/block/zram1/mm_stat" timed out

Looking at mm_stat_show(), mem_used can be 0 when mm_stat_show() is
called on un-initialized device

---
...
        u64 orig_size, mem_used = 0;
        long max_used;
        ssize_t ret;

        memset(&pool_stats, 0x00, sizeof(struct zs_pool_stats));

        down_read(&zram->init_lock);
        if (init_done(zram)) {
                mem_used = zs_get_total_pages(zram->mem_pool);
                zs_pool_stats(zram->mem_pool, &pool_stats);
        }
...
---

Can you check if init_done() returns true in your tests?

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Petr Vorel]

Hi Sergey, Minchan, all,


> On (22/11/07 23:42), Petr Vorel wrote:
> [..]
> > 15859712        0        0 26214400   196608      242        0
> > zram01 7 TINFO: /sys/block/zram1/mm_stat
> > 15859712        0        0 26214400   196608      242        0
> > zram01 7 TINFO: /sys/block/zram1/mm_stat
> > 15859712        0        0 26214400   196608      242        0
> > zram01 7 TINFO: /sys/block/zram1/mm_stat
> > 15859712        0        0 26214400   196608      242        0
> > zram01 7 TINFO: /sys/block/zram1/mm_stat
> > 15859712        0        0 26214400   196608      242        0
> > zram01 7 TINFO: /sys/block/zram1/mm_stat
> > 15859712        0        0 26214400   196608      242        0
> > zram01 7 TINFO: /sys/block/zram1/mm_stat
> > 15859712        0        0 26214400   196608      242        0
> > zram01 7 TBROK: "loop_read_mem_used_total /sys/block/zram1/mm_stat" timed out

> Looking at mm_stat_show(), mem_used can be 0 when mm_stat_show() is
> called on un-initialized device

> ---
> ...
>         u64 orig_size, mem_used = 0;
>         long max_used;
>         ssize_t ret;

>         memset(&pool_stats, 0x00, sizeof(struct zs_pool_stats));

>         down_read(&zram->init_lock);
>         if (init_done(zram)) {
>                 mem_used = zs_get_total_pages(zram->mem_pool);
>                 zs_pool_stats(zram->mem_pool, &pool_stats);
>         }
> ...
> ---

> Can you check if init_done() returns true in your tests?

I'm sorry, it took me some time to find machine where I'd be compile kernel.
Yes, init_done() returns non-zero (1), code goes into if clause and sets
mem_used. I'll check also for mem_used value.

Kind regards,
Petr

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Martin Doucha]

On 07. 11. 22 22:25, Minchan Kim wrote:
> On Mon, Nov 07, 2022 at 08:11:35PM +0100, Petr Vorel wrote:
>> Hi all,
>>
>> following bug is trying to workaround an error on ppc64le, where
>> zram01.sh LTP test (there is also kernel selftest
>> tools/testing/selftests/zram/zram01.sh, but LTP test got further
>> updates) has often mem_used_total 0 although zram is already filled.
> 
> Hi, Petr,
> 
> Is it happening on only ppc64le?
> 
> Is it a new regression? What kernel version did you use?

Hi,
I've reported the same issue on kernels 4.12.14 and 5.3.18 internally to 
our kernel developers at SUSE. The bugreport is not public but I'll copy 
the bug description here:

New version of LTP test zram01 found a sysfile issue with zram devices 
mounted using VFAT filesystem. When when all available space is filled, 
e.g. by `dd if=/dev/zero of=/mnt/zram0/file`, the corresponding sysfile 
/sys/block/zram0/mm_stat will report that the compressed data size on 
the device is 0 and total memory usage is also 0. LTP test zram01 uses 
these values to calculate compression ratio, which results in division 
by zero.

The issue is specific to PPC64LE architecture and the VFAT filesystem. 
No other tested filesystem has this issue and I could not reproduce it 
on other archs (s390 not tested). The issue appears randomly about every 
3 test runs on SLE-15SP2 and 15SP3 (kernel 5.3). It appears less 
frequently on SLE-12SP5 (kernel 4.12). Other SLE version were not tested 
with the new test version yet. The previous version of the test did not 
check the VFAT filesystem on zram devices.

I've tried to debug the issue and collected some interesting data (all 
values come from zram device with 25M size limit and zstd compression 
algorithm):
- mm_stat values are correct after mkfs.vfat:
65536      220    65536 26214400    65536        0        0        0

- mm_stat values stay correct after mount:
65536      220    65536 26214400    65536        0        0        0

- the bug is triggered by filling the filesystem to capacity (using dd):
4194304        0        0 26214400   327680       64        0        0

- adding `sleep 1` between `dd` and reading mm_stat does not help
- adding sync between `dd` and reading mm_stat appears to fix the issue:
26214400     2404   262144 26214400   327680      399        0        0

-- 
Martin Doucha   mdoucha@suse.cz
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Minchan Kim]

On Mon, Nov 07, 2022 at 10:47:33PM +0100, Petr Vorel wrote:
> Hi Minchan,
> 
> > On Mon, Nov 07, 2022 at 08:11:35PM +0100, Petr Vorel wrote:
> > > Hi all,
> 
> > > following bug is trying to workaround an error on ppc64le, where
> > > zram01.sh LTP test (there is also kernel selftest
> > > tools/testing/selftests/zram/zram01.sh, but LTP test got further
> > > updates) has often mem_used_total 0 although zram is already filled.
> 
> > Hi, Petr,
> 
> > Is it happening on only ppc64le?
> I haven't seen it on other archs (x86_64, aarch64).
> 
> > Is it a new regression? What kernel version did you use?
> Found on openSUSE kernel, which uses stable kernel releases 6.0.x.
> It's probably much older, first I've seen it some years ago (I'm not able to find kernel version), but it was random. Now it's much more common.
> 
> Test runs on VM (I can give qemu command or whatever you need to know about it)
> I'll try to verify it on some bare metal ppc64le.

Hi Petr and Martin,

Thanks for testing and meaning information.

Could you tell how I could create VM to run ppc64le and run the test?
I'd like to reproduce in my local to debug it.

Thanks!

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Sergey Senozhatsky]

On (22/11/10 15:29), Martin Doucha wrote:
> New version of LTP test zram01 found a sysfile issue with zram devices
> mounted using VFAT filesystem. When when all available space is filled, e.g.
> by `dd if=/dev/zero of=/mnt/zram0/file`, the corresponding sysfile
> /sys/block/zram0/mm_stat will report that the compressed data size on the
> device is 0 and total memory usage is also 0. LTP test zram01 uses these
> values to calculate compression ratio, which results in division by zero.
> 
> The issue is specific to PPC64LE architecture and the VFAT filesystem. No
> other tested filesystem has this issue and I could not reproduce it on other
> archs (s390 not tested). The issue appears randomly about every 3 test runs
> on SLE-15SP2 and 15SP3 (kernel 5.3). It appears less frequently on SLE-12SP5
> (kernel 4.12). Other SLE version were not tested with the new test version
> yet. The previous version of the test did not check the VFAT filesystem on
> zram devices.

Whoooaa...

> I've tried to debug the issue and collected some interesting data (all
> values come from zram device with 25M size limit and zstd compression
> algorithm):
> - mm_stat values are correct after mkfs.vfat:
> 65536      220    65536 26214400    65536        0        0        0
> 
> - mm_stat values stay correct after mount:
> 65536      220    65536 26214400    65536        0        0        0
> 
> - the bug is triggered by filling the filesystem to capacity (using dd):
> 4194304        0        0 26214400   327680       64        0        0

Can you try using /dev/urandom for dd, not /dev/zero?
Do you still see zeroes in sysfs output or some random values?

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Petr Vorel]

Hi Martin,

thanks a lot for providing more complete description!

Kind regards,
Petr

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Petr Vorel]

> On Mon, Nov 07, 2022 at 10:47:33PM +0100, Petr Vorel wrote:
> > Hi Minchan,

> > > On Mon, Nov 07, 2022 at 08:11:35PM +0100, Petr Vorel wrote:
> > > > Hi all,

> > > > following bug is trying to workaround an error on ppc64le, where
> > > > zram01.sh LTP test (there is also kernel selftest
> > > > tools/testing/selftests/zram/zram01.sh, but LTP test got further
> > > > updates) has often mem_used_total 0 although zram is already filled.

> > > Hi, Petr,

> > > Is it happening on only ppc64le?
> > I haven't seen it on other archs (x86_64, aarch64).

> > > Is it a new regression? What kernel version did you use?
> > Found on openSUSE kernel, which uses stable kernel releases 6.0.x.
> > It's probably much older, first I've seen it some years ago (I'm not able to find kernel version), but it was random. Now it's much more common.

> > Test runs on VM (I can give qemu command or whatever you need to know about it)
> > I'll try to verify it on some bare metal ppc64le.

> Hi Petr and Martin,

> Thanks for testing and meaning information.

> Could you tell how I could create VM to run ppc64le and run the test?
> I'd like to reproduce in my local to debug it.
I suppose you don't have ppc64le bare metal machine, thus you run on x86_64.

One way would be to install on host qemu-system-ppc64, download iso image of any
distro which supports ppc64le and install it with virt-manager (which would fill
necessary qemu params).

Other way, which I often use is to compile system with Buildroot distribution.
You can clone my Buildroot distro fork, branch debug/zram [1].
I put there in 3 commits my configuration.
I added 0001-zram-Debug-mm_stat_show.patch [2] on the top of 6.0.7 with little debugging.

What is now only needed is to 1) install on host qemu-system-ppc64
(speedup build + Buildroot is configured not to compile qemu-system-ppc64),
then:
$ make # takes time
$ ./output/images/start-qemu.sh serial-only

When I have ppc64le host with enough space, I often use rapido [3],
but that crashed stable kernel (another story which I'll report soon).

Hope that helps.

Kind regards,
Petr

[1] https://github.com/pevik/buildroot/commits/debug/zram
[2] https://github.com/pevik/buildroot/blob/debug/zram/0001-zram-Debug-mm_stat_show.patch
[3] https://github.com/rapido-linux/rapido

Re: [LTP] [PATCH 1/1] zram01.sh: Workaround division by 0 on vfat on ppc64le

[Petr Vorel]

Hi Li,

> Hi Petr,

> On Tue, Nov 8, 2022 at 3:12 AM Petr Vorel <pvorel@suse.cz> wrote:

> > Repeatedly read /sys/block/zram*/mm_stat for 1 sec. This should fix bug
> > on ppc64le on stable kernels, where mem_used_total is often 0.

> > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > ---
> >  .../kernel/device-drivers/zram/zram01.sh      | 27 +++++++++++++++++--
> >  1 file changed, 25 insertions(+), 2 deletions(-)

> > diff --git a/testcases/kernel/device-drivers/zram/zram01.sh
> > b/testcases/kernel/device-drivers/zram/zram01.sh
> > index 58d233f91..76a8ccab4 100755
> > --- a/testcases/kernel/device-drivers/zram/zram01.sh
> > +++ b/testcases/kernel/device-drivers/zram/zram01.sh
> > @@ -105,6 +105,26 @@ zram_mount()
> >         tst_res TPASS "mount of zram device(s) succeeded"
> >  }

> > +read_mem_used_total()
> > +{
> > +       echo $(awk '{print $3}' $1)
> > +}
> > +
> > +# Reads /sys/block/zram*/mm_stat until mem_used_total is not 0.
> > +loop_read_mem_used_total()


> This is not a looping function to check if mem_used_total is equal to zero,
> the loop part is by means of the TST_RETRY_FUNC macro.
Thanks for your review!

> So, I'd suggest renaming it to check_read_mem_used_total().
Agree. Unfortunately even this didn't help on ppc64le system where I was able to
reproduce it, thus probably not worth to merge.

Unfortunately later I was not able to reproduce the problem any more, I'll try
it more this week.

Kind regards,
Petr

> Reviewed-by: Li Wang <liwang@redhat.com>



> > +{
> > +       local file="$1"
> > +       local mem_used_total
> > +
> > +       tst_res TINFO "$file"
> > +       cat $file >&2
> > +
> > +       mem_used_total=$(read_mem_used_total $file)
> > +       [ "$mem_used_total" -eq 0 ] && return 1
> > +
> > +       return 0
> > +}
> > +
> >  zram_fill_fs()
> >  {
> >         local mem_used_total
> > @@ -133,9 +153,12 @@ zram_fill_fs()
> >                         continue
> >                 fi

> > -               mem_used_total=`awk '{print $3}'
> > "/sys/block/zram$i/mm_stat"`
> > +               TST_RETRY_FUNC "loop_read_mem_used_total
> > /sys/block/zram$i/mm_stat" 0
> > +               mem_used_total=$(read_mem_used_total
> > /sys/block/zram$i/mm_stat)
> > +               tst_res TINFO "mem_used_total: $mem_used_total"
> > +
> >                 v=$((100 * 1024 * $b / $mem_used_total))
> > -               r=`echo "scale=2; $v / 100 " | bc`
> > +               r=$(echo "scale=2; $v / 100 " | bc)

> >                 if [ "$v" -lt 100 ]; then
> >                         tst_res TFAIL "compression ratio: $r:1"
> > --
> > 2.38.0


> > --
> > Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Petr Vorel]

Hi Sergey,

> On (22/11/10 15:29), Martin Doucha wrote:
> > New version of LTP test zram01 found a sysfile issue with zram devices
> > mounted using VFAT filesystem. When when all available space is filled, e.g.
> > by `dd if=/dev/zero of=/mnt/zram0/file`, the corresponding sysfile
> > /sys/block/zram0/mm_stat will report that the compressed data size on the
> > device is 0 and total memory usage is also 0. LTP test zram01 uses these
> > values to calculate compression ratio, which results in division by zero.

> > The issue is specific to PPC64LE architecture and the VFAT filesystem. No
> > other tested filesystem has this issue and I could not reproduce it on other
> > archs (s390 not tested). The issue appears randomly about every 3 test runs
> > on SLE-15SP2 and 15SP3 (kernel 5.3). It appears less frequently on SLE-12SP5
> > (kernel 4.12). Other SLE version were not tested with the new test version
> > yet. The previous version of the test did not check the VFAT filesystem on
> > zram devices.

> Whoooaa...

> > I've tried to debug the issue and collected some interesting data (all
> > values come from zram device with 25M size limit and zstd compression
> > algorithm):
> > - mm_stat values are correct after mkfs.vfat:
> > 65536      220    65536 26214400    65536        0        0        0

> > - mm_stat values stay correct after mount:
> > 65536      220    65536 26214400    65536        0        0        0

> > - the bug is triggered by filling the filesystem to capacity (using dd):
> > 4194304        0        0 26214400   327680       64        0        0

> Can you try using /dev/urandom for dd, not /dev/zero?
> Do you still see zeroes in sysfs output or some random values?

I'm not sure if Martin had time to rerun the test. I was not able to reproduce
the problem any more on machine where the test was failing. But I'll have look
into this during this week.

Kind regards,
Petr

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Martin Doucha]

On 11. 11. 22 1:48, Sergey Senozhatsky wrote:
> On (22/11/10 15:29), Martin Doucha wrote:
>> I've tried to debug the issue and collected some interesting data (all
>> values come from zram device with 25M size limit and zstd compression
>> algorithm):
>> - mm_stat values are correct after mkfs.vfat:
>> 65536      220    65536 26214400    65536        0        0        0
>>
>> - mm_stat values stay correct after mount:
>> 65536      220    65536 26214400    65536        0        0        0
>>
>> - the bug is triggered by filling the filesystem to capacity (using dd):
>> 4194304        0        0 26214400   327680       64        0        0
> 
> Can you try using /dev/urandom for dd, not /dev/zero?
> Do you still see zeroes in sysfs output or some random values?

After 50 test runs on a kernel where the issue is confirmed, I could not 
reproduce the failure while filling the device from /dev/urandom instead 
of /dev/zero. The test reported compression ratio around 1.8-2.5 which 
means the memory usage reported by mm_stat was 10-13MB.

Note that I had to disable the other filesystems in the test because 
some of them kept failing with compression ratio <1.

-- 
Martin Doucha   mdoucha@suse.cz
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Petr Vorel]

> On 11. 11. 22 1:48, Sergey Senozhatsky wrote:
> > On (22/11/10 15:29), Martin Doucha wrote:
> > > I've tried to debug the issue and collected some interesting data (all
> > > values come from zram device with 25M size limit and zstd compression
> > > algorithm):
> > > - mm_stat values are correct after mkfs.vfat:
> > > 65536      220    65536 26214400    65536        0        0        0

> > > - mm_stat values stay correct after mount:
> > > 65536      220    65536 26214400    65536        0        0        0

> > > - the bug is triggered by filling the filesystem to capacity (using dd):
> > > 4194304        0        0 26214400   327680       64        0        0

> > Can you try using /dev/urandom for dd, not /dev/zero?
> > Do you still see zeroes in sysfs output or some random values?

> After 50 test runs on a kernel where the issue is confirmed, I could not
> reproduce the failure while filling the device from /dev/urandom instead of
> /dev/zero. The test reported compression ratio around 1.8-2.5 which means
> the memory usage reported by mm_stat was 10-13MB.

Martin, thanks a lot for reruning tests. I wonder problems on /dev/zero is a
kernel bug or just problem which should be workarounded.

> Note that I had to disable the other filesystems in the test because some of
> them kept failing with compression ratio <1.

Yes, I noted that as well at least on exfat and btrfs (if I remember correctly).
It wouldn't be a problem to just use it for vfat if we agreed test should be
modified.

Kind regards,
Petr

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Sergey Senozhatsky]

On (22/11/22 16:07), Petr Vorel wrote:
> > On 11. 11. 22 1:48, Sergey Senozhatsky wrote:
> > > On (22/11/10 15:29), Martin Doucha wrote:

[..]

> > > Can you try using /dev/urandom for dd, not /dev/zero?
> > > Do you still see zeroes in sysfs output or some random values?
> 
> > After 50 test runs on a kernel where the issue is confirmed, I could not
> > reproduce the failure while filling the device from /dev/urandom instead of
> > /dev/zero. The test reported compression ratio around 1.8-2.5 which means
> > the memory usage reported by mm_stat was 10-13MB.
> 
> Martin, thanks a lot for reruning tests. I wonder problems on /dev/zero is a
> kernel bug or just problem which should be workarounded.

Hmm. Does CONFIG_KASAN show anything interesting?

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Martin Doucha]

On 29. 11. 22 5:38, Sergey Senozhatsky wrote:
> On (22/11/22 16:07), Petr Vorel wrote:
>> Martin, thanks a lot for reruning tests. I wonder problems on /dev/zero is a
>> kernel bug or just problem which should be workarounded.
> 
> Hmm. Does CONFIG_KASAN show anything interesting?

Sorry for the delay. We've tried to reproduce the bug with CONFIG_KASAN 
enabled but the only affected arch is PPC64LE and KASAN is not available 
there at all. Our kernel maintainers confirmed that if we need KASAN to 
debug this, we're out of luck.

-- 
Martin Doucha   mdoucha@suse.cz
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Ian Wienand]

On Mon, Nov 07, 2022 at 01:25:16PM -0800, Minchan Kim wrote:
> > following bug is trying to workaround an error on ppc64le, where
> > zram01.sh LTP test (there is also kernel selftest
> > tools/testing/selftests/zram/zram01.sh, but LTP test got further
> > updates) has often mem_used_total 0 although zram is already filled.

> Is it happening on only ppc64le?

I have managed to replicate this on an arm64 system.  I frankly don't
know what is so special about it -- it's a qemu guest and I'm not sure
what exactly it's running ontop of.

> Is it a new regression? What kernel version did you use?

I've replicated this on 4.18.0; obviously something more recent would
be useful but I'm hesitant to destroy too much state in case it is
something ...

> Actually, mem_used_total indicates how many *physical memory* were
> currently used to keep original data size.
> 
> However, if the test data is repeated pattern of unsigned long
> (https://github.com/torvalds/linux/blob/master/drivers/block/zram/zram_drv.c#L210)
> zram doesn't allocate the physical memory but just mark the unsigned long's value
> in meta area for decompression later.

To recap; this test [1] creates a zram device, makes a filesystem on
it, and fills it with sequential 1k writes from /dev/zero via dd.  The
problem is that it sees the mem_used_total for the zram device as zero
in the sysfs stats after the writes; this causes a divide by zero
error in the script calculation.

An annoted extract:

 zram01 3 TINFO: /sys/block/zram1/disksize = '26214400'
 zram01 3 TPASS: test succeeded
 zram01 4 TINFO: set memory limit to zram device(s)
 zram01 4 TINFO: /sys/block/zram1/mem_limit = '25M'
 zram01 4 TPASS: test succeeded
 zram01 5 TINFO: make vfat filesystem on /dev/zram1

 >> at this point a cat of /sys/block/zram1/mm_stat shows
 >>   65536      527    65536 26214400    65536        0        0        0

 zram01 5 TPASS: zram_makefs succeeded
 zram01 6 TINFO: mount /dev/zram1
 zram01 6 TPASS: mount of zram device(s) succeeded
 zram01 7 TINFO: filling zram1 (it can take long time)
 zram01 7 TPASS: zram1 was filled with '25568' KB

 >> at this point "ls -lh" shows the file
 >> total 25M
 >> -rwxr-xr-x. 1 root root 25M Aug  4 01:06 file

 >> however, /sys/block/zram1/mm_stat shows
 >>   9502720        0        0 26214400   196608      145        0        0
 >> the script reads this zero value and tries to calculate the
 >> compression ratio

 ./zram01.sh: line 145: 100 * 1024 * 25568 / 0: division by 0 (error token is "0")

 >> If we do a "sync" then redisply the mm_stat after, we get
 >>   26214400     2842    65536 26214400   196608      399        0        0

I have managed to instrument this, and in the following

 static ssize_t mm_stat_show(struct device *dev,
                struct device_attribute *attr, char *buf)
 {
  ...
        if (init_done(zram)) {
		mem_used = zs_get_total_pages(zram->mem_pool);
                pr_info("mm_stat_show: init done %p %lld\n", zram->mem_pool, mem_used);
                zs_pool_stats(zram->mem_pool, &pool_stats);

zs_get_total_pages(zram->mem_pool) is definitely zero, which is why
the mm_stat is returning zero.  i.e. zsmalloc really doesn't seem to
have any pages recorded for that mem_pool ...

This doesn't seem to make sense; how can a device that has a file
system on it not even have one page assigned to it in zram->mem_pool?

I *think* this has something to do with the de-deuplication as noted.
If I stub out page_same_filled() to return false always, we see instead

 zram01 7 TPASS: zram1 was filled with '25568' KB
 >>  < immediately after >
 >> 10223616    48516   131072 26214400   196608        0        0        0
 >>  < after sync >
 >> 26214400   126933   327680 26214400   327680        0        0        0

So I think this test still needs a sync to be sure that it's seeing
the right values?  It's probably expected that this takes some time to
write everything out?

But is it possible that mem_used_total being zero is a bug -- possibly
triggered by the de-dup path and the test writing the same thing in
every block?  Something like the first de-duped page also being thrown
out?

-i

[1] https://github.com/linux-test-project/ltp/blob/8c201e55f684965df2ae5a13ff439b28278dec0d/testcases/kernel/device-drivers/zram/zram01.sh

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Ian Wienand]

After thinking it through, I think I might have a explanation...

On Fri, Aug 04, 2023 at 04:37:11PM +1000, Ian Wienand wrote:
> To recap; this test [1] creates a zram device, makes a filesystem on
> it, and fills it with sequential 1k writes from /dev/zero via dd.  The
> problem is that it sees the mem_used_total for the zram device as zero
> in the sysfs stats after the writes; this causes a divide by zero
> error in the script calculation.
> 
> An annoted extract:
> 
>  zram01 3 TINFO: /sys/block/zram1/disksize = '26214400'
>  zram01 3 TPASS: test succeeded
>  zram01 4 TINFO: set memory limit to zram device(s)
>  zram01 4 TINFO: /sys/block/zram1/mem_limit = '25M'
>  zram01 4 TPASS: test succeeded
>  zram01 5 TINFO: make vfat filesystem on /dev/zram1
> 
>  >> at this point a cat of /sys/block/zram1/mm_stat shows
>  >>   65536      527    65536 26214400    65536        0        0        0
> 
>  zram01 5 TPASS: zram_makefs succeeded

So I think the thing to note is that mem_used_total is the current
number of pages (reported * PAGE_SIZE) used by the zsmalloc allocator
to store compressed data.

So we have made the file system, which is now quiescent and just has
basic vfat data; this is compressed and stored and there's one page
allocated for this (arm64, 64k pages).

>  zram01 6 TINFO: mount /dev/zram1
>  zram01 6 TPASS: mount of zram device(s) succeeded
>  zram01 7 TINFO: filling zram1 (it can take long time)
>  zram01 7 TPASS: zram1 was filled with '25568' KB
>
>  >> however, /sys/block/zram1/mm_stat shows
>  >>   9502720        0        0 26214400   196608      145        0        0
>  >> the script reads this zero value and tries to calculate the
>  >> compression ratio
> 
>  ./zram01.sh: line 145: 100 * 1024 * 25568 / 0: division by 0 (error token is "0")

At this point, because this test fills from /dev/zero, the zsmalloc
pool doesn't actually have anything in it.  The filesystem metadata is
in-use from the writes, and is not written out as compressed data.
The zram page de-duplication has kicked in, and instead of handles to
zsmalloc areas for data we just have "this is a page of zeros"
recorded.  So this is correctly reflecting that fact that we don't
actually have anything compressed stored at this time.

>  >> If we do a "sync" then redisply the mm_stat after, we get
>  >>   26214400     2842    65536 26214400   196608      399        0        0

Now we've finished writing all our zeros and have synced, we would
have finished updating vfat allocations, etc.  So this gets compressed
and written, and we're back to have some small FS metadata compressed
in our 1 page of zsmalloc allocations.

I think what is probably "special" about this reproducer system is
that it is slow enough to allow the zero allocation to persist between
the end of the test writes and examining the stats.

I'd be happy for any thoughts on the likelyhood of this!

If we think this is right; then the point of the end of this test [1]
is ensure a high reported compression ratio on the device, presumably
to ensure the compression is working.  Filling it with urandom would
be unreliable in this regard.  I think what we want to do is something
highly compressable like alternate lengths of 0x00 and 0xFF.  This
will avoid the same-page detection and ensure we actually have
compressed data, and we can continue to assert on the high compression
ratio reliably.  I'm happy to propose this if we generally agree.

Thanks,

-i

> [1] https://github.com/linux-test-project/ltp/blob/8c201e55f684965df2ae5a13ff439b28278dec0d/testcases/kernel/device-drivers/zram/zram01.sh

Re: [PATCH 0/1] Possible bug in zram on ppc64le on vfat

[Sergey Senozhatsky]

On (23/08/07 14:44), Ian Wienand wrote:
[..]
> 
> At this point, because this test fills from /dev/zero, the zsmalloc
> pool doesn't actually have anything in it.  The filesystem metadata is
> in-use from the writes, and is not written out as compressed data.
> The zram page de-duplication has kicked in, and instead of handles to
> zsmalloc areas for data we just have "this is a page of zeros"
> recorded.  So this is correctly reflecting that fact that we don't
> actually have anything compressed stored at this time.
> 
> >  >> If we do a "sync" then redisply the mm_stat after, we get
> >  >>   26214400     2842    65536 26214400   196608      399        0        0
> 
> Now we've finished writing all our zeros and have synced, we would
> have finished updating vfat allocations, etc.  So this gets compressed
> and written, and we're back to have some small FS metadata compressed
> in our 1 page of zsmalloc allocations.
> 
> I think what is probably "special" about this reproducer system is
> that it is slow enough to allow the zero allocation to persist between
> the end of the test writes and examining the stats.
> 
> I'd be happy for any thoughts on the likelyhood of this!

Thanks for looking into this.

Yes, the fact that /dev/urandom shows non-zero values in mm_stat means
that we don't have any fishy going on in zram but instead very likely
have ZRAM_SAME pages, which don't reach zsmalloc pool and don't use any
physical pages.

And this is what 145 is in mm_stat that was posted earlier. We have 145
pages that are filled with the same bytes pattern:

> >  >> however, /sys/block/zram1/mm_stat shows
> >  >>   9502720        0        0 26214400   196608      145        0        0

> If we think this is right; then the point of the end of this test [1]
> is ensure a high reported compression ratio on the device, presumably
> to ensure the compression is working.  Filling it with urandom would
> be unreliable in this regard.  I think what we want to do is something
> highly compressable like alternate lengths of 0x00 and 0xFF.

So var-lengths 0x00/0xff should work, just make sure that you don't have
a pattern of sizeof(unsigned long) length.

I think fio had an option to generate bin data with a certain level
of compress-ability. If that option works then maybe you can just use
fio with some static buffer_compress_percentage configuration.