[Cluster-devel] gfs2-utils 3.3.0 released

[Cluster-devel] gfs2-utils 3.3.0 released

[Andrew Price]

Hi all,

I am pleased to announce the 3.3.0 release of gfs2-utils.

This release brings further tooling support for the new log header and 
resource group fields, bug fixes across the board, testing improvements, 
and various other fixes, cleanups and code re-structuring.

Some user-visible changes:

  * mkfs.gfs2 now defaults to a 4K block size even when the device 
reports a smaller optimal i/o size (as reported for shared NVMe devices 
in cloud environments)
  * Extensive man page improvements
  * gfs2_edit restoremeta can now restore bzip2-compressed files
    + New build-dependency: bzip2
  * Improved gfs2_jadd error handling, particularly on a full filesystem
  * The gfs2_withdraw_helper script is now installed to $libexecdir

See below for a complete list of changes.

The source tarball is available from:

   https://releases.pagure.org/gfs2-utils/gfs2-utils-3.3.0.tar.gz

Please test, and report bugs against the gfs2-utils component of Fedora 
rawhide: 
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&component=gfs2-utils&version=rawhide

Thanks,
Andy


Changes since version 3.2.0:

Abhi Das (2):
       gfs2_jadd: Handle out-of-space issues
       gfs2_jadd: error handling overhaul

Andreas Gruenbacher (6):
       libgfs2: Update metadata definition of gfs2_log_header.lh_flags
       libgfs2: Allow to suppress symbol prefixes
       libgfs2: lgfs2_field_str: Separate field extraction from printing
       libgfs2: lgfs2_field_str: Allow to decode bitmasks
       libgfs2: Fix gfs[2]_log_header metadata description
       gfs2edit: journal dump: decode lh_flags

Andrew Price (86):
       scripts: Install gfs2_withdraw_helper into $libexecdir
       configure: Show $libexecdir in configure summary
       Add 82-gfs2-withdraw.rules to .gitignore
       libgfs2: Make the resource group plan system more flexible
       fsck.gfs2: Don't check fs formats we don't recognise
       libgfs2: Fix pointer cast byte order issue
       gfs2-utils: String handling fix-ups for gcc9
       Fix libuuid linking
       Fix more linking errors
       gfs2.5: General updates and layout improvements
       gfs2_grow.8: Add a reference to the lvm docs
       testsuite: Skip tests if unshipped helpers are not found
       mkfs.gfs2: Improve alignment of first resource group
       fsck.gfs2: Remove some unnecessary and slow gettext calls
       fsck.gfs2: Disambiguate 'check_data'
       libgfs2: Speed up bread()
       gfs2-utils: Accept a char* instead of a buffer head in 
gfs2_check_meta
       fsck.gfs2: Disambiguate check_metalist
       mkfs.gfs2: Use strncpy to copy lockproto/table into sb
       gfs2_edit: Don't move cursor when displaying indirect blocks
       fsck.gfs2(8): Manpage updates
       fsck.gfs2: Fix segfault in build_and_check_metalist
       fsck.gfs2: Retain context for indirect pointers in ->check_metalist
       fsck.gfs2: Clear bad indirect block pointers when bitmap meets 
expectations
       restoremeta: Use zlib by file descriptor
       restoremeta: Abstract out metadata file opening
       restoremeta: Use metafd instead of gzFile for parameters
       restoremeta: Abstract out decompression operations
       restoremeta: Combine restore_init() and open_metadata()
       restoremeta: Don't seek in restore_header()
       savemeta: Remove anthropomorphize()
       restoremeta: Remove find_highest_block()
       restoremeta: Metadata file reading overhaul
       restoremeta: Convert iseof function to a flag
       restoremeta: Combine parse_header() and check_header()
       restoremeta: Add bzip2 support
       restoremeta: Skip the right number of bytes for the superblock
       tests: Add fsck.gfs2 test script
       gfs2.5: Update some mentions of gfs2_tool
       libgfs2: Don't use bh's in gfs_rgrp_in/out
       libgfs2: Remove bi_bh from struct gfs2_bitmap
       savemeta: Save rgrps without using gfs2_buffer_heads
       savemeta: Check file format version properly
       savemeta: Use larger buffers for gzip compression
       mkfs.gfs2: Don't use i/o limits hints <4K for block size
       gfs2_jadd: Fix static analysis warnings
       mkf.gfs2: Clarify when extended options are invalid
       tests: Don't use fail_unless in unit tests
       savemeta: Allow saving to /dev/null
       mkfs.gfs2: Fix strncpy warnings
       libgfs2: Separate out gfs2l's language API
       glocktop: Improve mount info handling
       savemeta: Don't save bad xattr blocks twice
       libgfs2: Remove gfs2_buffer_head from gfs_dinode_in()
       libgfs2: Remove gfs2_buffer_head from lgfs2_gfs_inode_get()
       libgfs2: Remove gfs2_buffer_head from lgfs2_write_journal_data()
       libgfs2: Move get_file_buf() into structures.c
       gfs2l: Remove uses of gfs2_buffer_heads
       libgfs2: No need to use gfs2_buffer_head in metapointer()
       gfs2_edit: Don't use gfs2_buffer_head in do_dinode_extended() args
       libgfs2: Add a display name field to struct lgfs2_metadata
       gfs2_edit: get_block_type() improvements
       gfs2_edit: Don't use gfs2_buffer_head in display_block_type()
       gfs2_edit: Don't use gfs2_buffer_head in display_gfs2()
       gfs2_edit: restore_block() improvements
       savemeta: Simplify di_save_len()
       savemeta: Remove gfs2_buffer_head from get_gfs_struct_info()
       savemeta: Remove gfs2_buffer_head from save_bh() (and rename it)
       savemeta: Don't use gfs2_buffer_head in save_leaf_chain()
       savemeta: Remove gfs2_buffer_head from save_block()
       savemeta: Split block reading from saving
       savemeta: Call get_struct_info() in the read path
       savemeta: Introduce multi-block reads
       savemeta: Process indirect pointers in chunks
       savemeta: Don't trim off trailing zeroes when compressing
       savemeta: Leaf block processing improvements
       savemeta: Remove some unnecessary reads from save_inode_data()
       savemeta: Remove some unnecessary jindex reading code
       savemeta: Move block range queue ops into functions
       restoremeta: Fix up superblock processing
       mkfs.gfs2: Don't exceed IOV_MAX when zeroing first alignment gap
       mkfs.gfs2: Tighten minimum journal size checks
       Update copyright notices
       Convert README to markdown
       Update translation template
       Use the current version in AC_INIT()

Bob Peterson (1):
       gfs2-utils: Wrong hash value used to clean journals

Valentin Vidic (1):
       gfs2-utils: Fix bashism in test script

[Cluster-devel] [ClusterLabs] gfs2-utils 3.3.0 released

[Valentin Vidić]

On Tue, Sep 01, 2020 at 04:41:44PM +0100, Andrew Price wrote:
> I am pleased to announce the 3.3.0 release of gfs2-utils.
> 
> This release brings further tooling support for the new log header and
> resource group fields, bug fixes across the board, testing improvements, and
> various other fixes, cleanups and code re-structuring.

Hi, it seems some tests fail during ARM build:

gfs2_edit tests

 31: Save/restoremeta, defaults                      FAILED (edit.at:13)
 32: Save/restoremeta, no compression                FAILED (edit.at:23)
 33: Save/restoremeta, min. block size               FAILED (edit.at:33)
 34: Save/restoremeta, 4 journals                    FAILED (edit.at:43)
 35: Save/restoremeta, min. block size, 4 journals   FAILED (edit.at:53)
 36: Save metadata to /dev/null                      ok

More details here:

https://buildd.debian.org/status/fetch.php?pkg=gfs2-utils&arch=armhf&ver=3.3.0-1&stamp=1599171965&file=log

-- 
Valentin

[Cluster-devel] [ClusterLabs] gfs2-utils 3.3.0 released

[Andrew Price]

On 03/09/2020 23:39, Valentin Vidi? wrote:
> Hi, it seems some tests fail during ARM build:
> 
> gfs2_edit tests
> 
>   31: Save/restoremeta, defaults                      FAILED (edit.at:13)
>   32: Save/restoremeta, no compression                FAILED (edit.at:23)
>   33: Save/restoremeta, min. block size               FAILED (edit.at:33)
>   34: Save/restoremeta, 4 journals                    FAILED (edit.at:43)
>   35: Save/restoremeta, min. block size, 4 journals   FAILED (edit.at:53)
>   36: Save metadata to /dev/null                      ok
> 
> More details here:
> 
> https://buildd.debian.org/status/fetch.php?pkg=gfs2-utils&arch=armhf&ver=3.3.0-1&stamp=1599171965&file=log
> 

The "Bus error"s suggest it's failing on an unaligned access somewhere 
in restoremeta, so just armhf and sparc64 are affected. I'll look into it.

Thanks,
Andy

[Cluster-devel] [ClusterLabs] gfs2-utils 3.3.0 released

[Valentin Vidic]

On Fri, Sep 04, 2020 at 04:13:08AM +0100, Andrew Price wrote:
> The "Bus error"s suggest it's failing on an unaligned access somewhere in
> restoremeta, so just armhf and sparc64 are affected. I'll look into it.

Thanks, there is a lot of [-Wcast-align] warnings in the build log that
could be relevant and don't show up on amd64. Also I don't think the
previous 3.2.0 version had this problem with the tests:

https://buildd.debian.org/status/fetch.php?pkg=gfs2-utils&arch=armhf&ver=3.2.0-3&stamp=1581778584&raw=0
https://buildd.debian.org/status/fetch.php?pkg=gfs2-utils&arch=armhf&ver=3.3.0-1&stamp=1599171965&raw=0

-- 
Valentin

[Cluster-devel] [ClusterLabs] gfs2-utils 3.3.0 released

[Andrew Price]

On 04/09/2020 07:18, Valentin Vidic wrote:
> On Fri, Sep 04, 2020 at 04:13:08AM +0100, Andrew Price wrote:
>> The "Bus error"s suggest it's failing on an unaligned access somewhere in
>> restoremeta, so just armhf and sparc64 are affected. I'll look into it.
> 
> Thanks, there is a lot of [-Wcast-align] warnings in the build log that
> could be relevant and don't show up on amd64. Also I don't think the
> previous 3.2.0 version had this problem with the tests:
> 
> https://buildd.debian.org/status/fetch.php?pkg=gfs2-utils&arch=armhf&ver=3.2.0-3&stamp=1581778584&raw=0
> https://buildd.debian.org/status/fetch.php?pkg=gfs2-utils&arch=armhf&ver=3.3.0-1&stamp=1599171965&raw=0
> 

I'm pretty sure I know which change added the bug, but my Debian sparc64 
vm is hitting this problem too frequently to test:

https://lkml.org/lkml/2020/5/18/1455

So probably the best I can do is build with -Wcast-align=strict and make 
the relevant warning go away.

Andy

[Cluster-devel] [ClusterLabs] gfs2-utils 3.3.0 released

[Valentin Vidic]

On Fri, Sep 04, 2020 at 02:42:10PM +0100, Andrew Price wrote:
> I'm pretty sure I know which change added the bug, but my Debian sparc64 vm
> is hitting this problem too frequently to test:
> 
> https://lkml.org/lkml/2020/5/18/1455
> 
> So probably the best I can do is build with -Wcast-align=strict and make the
> relevant warning go away.

I tried to reproduce the problem on arm, but after a rebuild tests look
fine, so I'm not sure anymore if this is a new problem or something that
happens randomly.

For sparc64 it seems to be reproducible and I have access to a machine
in case you want me to test some patches.

-- 
Valentin

[Cluster-devel] [ClusterLabs] gfs2-utils 3.3.0 released

[Andrew Price]

On 04/09/2020 18:42, Valentin Vidic wrote:
> On Fri, Sep 04, 2020 at 02:42:10PM +0100, Andrew Price wrote:
>> I'm pretty sure I know which change added the bug, but my Debian sparc64 vm
>> is hitting this problem too frequently to test:
>>
>> https://lkml.org/lkml/2020/5/18/1455
>>
>> So probably the best I can do is build with -Wcast-align=strict and make the
>> relevant warning go away.
> 
> I tried to reproduce the problem on arm, but after a rebuild tests look
> fine, so I'm not sure anymore if this is a new problem or something that
> happens randomly.

It should be consistent if it's the same alignment issue, I think. 
Perhaps the build machines are different flavours of arm?

> For sparc64 it seems to be reproducible and I have access to a machine
> in case you want me to test some patches.

Could you give this one a try:

https://pagure.io/gfs2-utils/c/eb102019?branch=andyp-restore-init-align

Andy

[Cluster-devel] [ClusterLabs] gfs2-utils 3.3.0 released

[Valentin Vidic]

On Mon, Sep 07, 2020 at 02:00:41PM +0100, Andrew Price wrote:
> It should be consistent if it's the same alignment issue, I think. Perhaps
> the build machines are different flavours of arm?

That is possible, second build was scheduled on a different machine.

> Could you give this one a try:
> 
> https://pagure.io/gfs2-utils/c/eb102019?branch=andyp-restore-init-align

Thanks, tests look good again on sparc64 with this patch applied:

gfs2_edit tests

 31: Save/restoremeta, defaults                      ok
 32: Save/restoremeta, no compression                ok
 33: Save/restoremeta, min. block size               ok
 34: Save/restoremeta, 4 journals                    ok
 35: Save/restoremeta, min. block size, 4 journals   ok
 36: Save metadata to /dev/null                      ok

-- 
Valentin

[Cluster-devel] [ClusterLabs] gfs2-utils 3.3.0 released

[Andrew Price]

On 07/09/2020 22:22, Valentin Vidic wrote:
> On Mon, Sep 07, 2020 at 02:00:41PM +0100, Andrew Price wrote:
>> It should be consistent if it's the same alignment issue, I think. Perhaps
>> the build machines are different flavours of arm?
> 
> That is possible, second build was scheduled on a different machine.
> 
>> Could you give this one a try:
>>
>> https://pagure.io/gfs2-utils/c/eb102019?branch=andyp-restore-init-align
> 
> Thanks, tests look good again on sparc64 with this patch applied:
> 
> gfs2_edit tests
> 
>   31: Save/restoremeta, defaults                      ok
>   32: Save/restoremeta, no compression                ok
>   33: Save/restoremeta, min. block size               ok
>   34: Save/restoremeta, 4 journals                    ok
>   35: Save/restoremeta, min. block size, 4 journals   ok
>   36: Save metadata to /dev/null                      ok
> 

Thanks for testing. I've pushed the patch to master.

Thinking about all the other cast-align warnings, we might be able to 
replace many of our be*_to_cpu() calls with functions that take a char* 
instead of integer types, so that we never have to cast char* to 
uint{16,32,64}_t* and widen the alignment when reading the on-disk 
structures. That should tidy up a lot of code too as we wouldn't need 
intermediate pointers. I'll look into it.

Andy