* Btrfs progs release 4.1
@ 2015-06-22 15:00 David Sterba
2015-06-22 16:18 ` Goffredo Baroncelli
` (5 more replies)
0 siblings, 6 replies; 24+ messages in thread
From: David Sterba @ 2015-06-22 15:00 UTC (permalink / raw)
To: linux-btrfs; +Cc: clm
Hi,
btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual load of
changes.
Fixed since rc1:
- uuid rewrite prints the correct original UUID
- map-logical updated
- fi show size units
- typos
* bugfixes
- fsck.btrfs: no bash-isms
- bugzilla 97171: invalid memory access (with tests)
- receive:
- cloning works with --chroot
- capabilities not lost
- mkfs: do not try to register bare file images
- option --help accepted by the standalone utilities
* enhancements
- corrupt block: ability to remove csums
- mkfs:
- warn if metadata redundancy is lower than for data
- options to make the output quiet (only errors)
- mixed case names of raid profiles accepted
- rework the output:
- more comprehensive, 'key: value' format
- subvol:
- show:
- print received uuid
- update the output
- new options to specify size units
- sync:
- grab all deleted ids and print them as they're removed,
previous implementation only checked if there are any
to be deleted - change in command semantics
- scrub: print timestamps in days HMS format
- receive:
- can specify mount point, do not rely on /proc
- can work inside subvolumes
- send:
- new option to send stream without data (NO_FILE_DATA)
- convert:
- specify incompat features on the new fs
- qgroup:
- show: distinguish no limits and 0 limit value
- limit: ability to clear the limit
- help for 'btrfs' is shorter, 1st level command overview
- debug tree: print key names according to their C name
* new
- rescure zero-log
- btrfsune:
- rewrite uuid on a filesystem image
- new option to turn on NO_HOLES incompat feature
* deprecated
- standalone btrfs-zero-log
* other
- testing framework updates
- uuid rewrite test
- btrfstune feature setting test
- zero-log tests
- more testing image formats
- manual page updates
- ioctl.h synced with current kernel uapi version
- convert: preparatory works for more filesystems (reiserfs pending)
- use static buffers for path handling where possible
- add new helpers for send uilts that check memory allocations,
switch all users, deprecate old helpers
- Makefile: fix build dependency generation
- map-logical: make it work again
Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
Shortlog:
Anand Jain (2):
btrfs-progs: add info about list-all to the help
btrfs-progs: use function is_block_device() instead
Dimitri John Ledkov (1):
btrfs-progs: fsck.btrfs: Fix bashism and bad getopts processing
Dongsheng Yang (4):
btrfs-progs: qgroup: show 'none' when we did not limit it on this qgroup
btrfs-progs: qgroup: allow user to clear some limitation on qgroup.
btrfs-progs: qgroup limit: error out if input value is negative
btrfs-progs: qgroup limit: add a check for invalid input of 'T/G/M/K'
Emil Karlson (1):
btrfs-progs: use openat for process_clone in receive
Goffredo Baroncelli (4):
btrfs-progs: add strdup in btrfs_add_to_fsid() to track the device path
btrfs-progs: return the fsid from make_btrfs()
btrfs-progs: mkfs: track sizes of created block groups
btrfs-progs: mkfs: print the summary
Jeff Mahoney (8):
btrfs-progs: convert: clean up blk_iterate_data handling wrt record_file_blocks
btrfs-progs: convert: remove unused fs argument from block_iterate_proc
btrfs-progs: convert: remove unused inode_key in copy_single_inode
btrfs-progs: convert: rename ext2_root to image_root
btrfs-progs: compat: define DIV_ROUND_UP if not already defined
btrfs-progs: convert: fix typo in btrfs_insert_dir_item call
btrfs-progs: convert: factor out adding dirent into convert_insert_dirent
btrfs-progs: convert: factor out block iteration callback
Josef Bacik (3):
Btrfs-progs: corrupt-block: add the ability to remove csums
btrfs-progs: specify mountpoint for recieve
btrfs-progs: make receive work inside of subvolumes
Qu Wenruo (13):
btrfs-progs: Enhance read_tree_block to avoid memory corruption
btrfs-progs: btrfstune: rework change_uuid
btrfs-progs: btrfstune: add ability to restore unfinished fsid change
btrfs-progs: btrfstune: add '-U' and '-u' option to change fsid
btrfs-progs: Documentation: uuid change
btrfs-progs: btrfstune: fix a bug which makes unfinished fsid change unrecoverable
btrfs-progs: export read_extent_data function
btrfs-progs: map-logical: introduce map_one_extent function
Btrfs-progs: map-logical: introduce print_mapping_info function
Btrfs-progs: map-logical: introduce write_extent_content function
btrfs-progs: map-logical: Rework map-logical logics
btrfs-progs: Allow "filesystem show" command to handle different units
btrfs-progs: docs: new size options for fi show
Sam Tygier (1):
btrfs-progs: mkfs: check metadata redundancy
David Sterba (79):
btrfs-progs: tests: log the test name in results file
btrfs-progs: tests: support more formats of test images
btrfs-progs: tests: use readlink -f to resolve path
btrfs-progs: tests: sort image files
btrfs-progs: test: 015-check-bad-memory-access
btrfs-progs: tests: add script to clean intermediate images
btrfs-progs: receive: restore capabilities after chown
btrfs-progs: tests: add driver script for misc tests
btrfs-progs: tests: common: add helper run_check_stdout
btrfs-progs: tests: add misc test for fs features
btrfs-progs: update .gitignore
btrfs-progs: subvol show: print received uuid
btrfs-progs: subvol show: tweak output
btrfs-progs: btrfstune: rework option handling
btrfs-progs: btrfstune: add option to enable NO_HOLES
btrfs-progs: doc: update btrfstune manpage
btrfs-progs: tests: add misc tests for uuid rewrite
btrfs-progs: btrfstune: make uuid rewrite progress more verbose
btrfs-progs: ioctl.h: reorder to match kernel uapi header
btrfs-progs: ioctl.h: copy commented version of scrub_progress structure
btrfs-progs: ioctl.h: copy commented version of balance structures
btrfs-progs: ioctl.h: add EXTENT_SAME ioctl and structures
btrfs-progs: ioctl.h: add send flag mask
btrfs-progs: ioctl.h: update type annotations
btrfs-progs: ioctl.h: update btrfs_ioctl_fs_info_args
btrfs-progs: ioctl.h: add btrfs_ioctl_feature_flags struct
btrfs-progs: doc: update btrfs-rescue page and drop btrfs-zero-log
btrfs-progs: scrub: more friendly duration format in status
btrfs-progs: add function to stringify filesystem features
btrfs-progs: convert: allow to set filesystem features
btrfs-progs: tests: update convert tests to set fs features
btrfs-progs: cleanup and deprecate btrfs-zero-log
btrfs-progs: tests: add test for zero-log
btrfs-progs: mkfs: add option to make it quiet
btrfs-progs: mkfs: accept mixed case for profile names
btrfs-progs: mkfs: move info message out of btrfs_add_to_fsid
btrfs-progs: mkfs: do not try to register non-block-devices
btrfs-progs: mkfs, move uuid to the end of device summary
btrfs-progs: mkfs, drop UUID from device summary
btrfs-progs: mkfs: drop mixed from summary
btrfs-progs: mkfs: make the summary more compact
btrfs-progs: remove stray deprecation notice
btrfs-progs: fix argv0_buf handling
btrfs-progs: add command group info strings
btrfs-progs: print compact help for btrfs
btrfs-progs: receive: fix minor resource leak
btrfs-progs: cleanup after errors in open_file_or_dir3
btrfs-progs: subvol: let sync check only current deletions
btrfs-progs: add helper to wait for subvolume cleaning
btrfs-progs: add missing includes to header files
btrfs-progs: accept --help as option in the standalone utilities
btrfs-progs: print error within test_dev_for_mkfs
btrfs-progs: print error within test_num_disk_vs_raid
btrfs-progs: use less memory for pretty_size_mode buffers
btrfs-progs: receive: implement the update_extent callback
btrfs-progs: send: add option to for the no-data mode
btrfs-progs: use PATH_MAX instead of BTRFS_PATH_NAME_MAX
btrfs-progs: properly set up ioctl arguments
btrfs-progs: receive: use static buffer for cur_subvol
btrfs-progs: add helper for copying paths
btrfs-progs: receive: use static buffer for mount point option argument
btrfs-progs: receive: use static buffer for source file
btrfs-progs: send: use static buffer for output file name
btrfs-progs: and new path_cat helpers to send utils
btrfs-progs: send utils: switch callbacks to new helpers
btrfs-progs: unify naming of key types in print-tree
btrfs-progs: receive: use static buffer for root_subvol_path
btrfs-progs: receive: use static buffer for write_path
btrfs-progs: send utils: switch callbacks to patch_cat3_out
btrfs-progs: receive: use static buffer for cur_subvol path
btrfs-progs: send utils: deprecate path_cat and path_cat3
btrfs-progs: Makefile: fix typo, dependencies for cmds-* files
Btrfs progs v4.1-rc1
btrfs-progs: tests: verify btrfstune output during uuid-rewrite
btrfs-progs: btrfstune: print lowercase uuid during uuid rewrite
btrfs-progs: btrfstune: print correct current uuid during rewrite
btrfs-progs: configure: fix typo in summary
btrfs-progs: INSTALL: fix typos
Btrfs progs v4.1
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-22 15:00 Btrfs progs release 4.1 David Sterba
@ 2015-06-22 16:18 ` Goffredo Baroncelli
2015-06-22 18:01 ` David Sterba
2015-06-22 18:18 ` qgroup limit clearing, was " Christian Robottom Reis
` (4 subsequent siblings)
5 siblings, 1 reply; 24+ messages in thread
From: Goffredo Baroncelli @ 2015-06-22 16:18 UTC (permalink / raw)
To: dsterba, linux-btrfs, clm
On 2015-06-22 17:00, David Sterba wrote:
> Hi,
Many thanks for your work.
BTW just for curiosity: is it a coincidence that both Torvalds and you released the kernel 4.1/btrfs-progs 4.1 in the same day ? I know that the version are coupled, but also the same day....
BR
G.Baronelli
>
> btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual load of
> changes.
>
> Fixed since rc1:
> - uuid rewrite prints the correct original UUID
> - map-logical updated
> - fi show size units
> - typos
>
> * bugfixes
> - fsck.btrfs: no bash-isms
> - bugzilla 97171: invalid memory access (with tests)
> - receive:
> - cloning works with --chroot
> - capabilities not lost
> - mkfs: do not try to register bare file images
> - option --help accepted by the standalone utilities
>
> * enhancements
> - corrupt block: ability to remove csums
> - mkfs:
> - warn if metadata redundancy is lower than for data
> - options to make the output quiet (only errors)
> - mixed case names of raid profiles accepted
> - rework the output:
> - more comprehensive, 'key: value' format
> - subvol:
> - show:
> - print received uuid
> - update the output
> - new options to specify size units
> - sync:
> - grab all deleted ids and print them as they're removed,
> previous implementation only checked if there are any
> to be deleted - change in command semantics
> - scrub: print timestamps in days HMS format
> - receive:
> - can specify mount point, do not rely on /proc
> - can work inside subvolumes
> - send:
> - new option to send stream without data (NO_FILE_DATA)
> - convert:
> - specify incompat features on the new fs
> - qgroup:
> - show: distinguish no limits and 0 limit value
> - limit: ability to clear the limit
> - help for 'btrfs' is shorter, 1st level command overview
> - debug tree: print key names according to their C name
>
> * new
> - rescure zero-log
> - btrfsune:
> - rewrite uuid on a filesystem image
> - new option to turn on NO_HOLES incompat feature
>
> * deprecated
> - standalone btrfs-zero-log
>
> * other
> - testing framework updates
> - uuid rewrite test
> - btrfstune feature setting test
> - zero-log tests
> - more testing image formats
> - manual page updates
> - ioctl.h synced with current kernel uapi version
> - convert: preparatory works for more filesystems (reiserfs pending)
> - use static buffers for path handling where possible
> - add new helpers for send uilts that check memory allocations,
> switch all users, deprecate old helpers
> - Makefile: fix build dependency generation
> - map-logical: make it work again
>
> Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
> Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
>
> Shortlog:
>
> Anand Jain (2):
> btrfs-progs: add info about list-all to the help
> btrfs-progs: use function is_block_device() instead
>
> Dimitri John Ledkov (1):
> btrfs-progs: fsck.btrfs: Fix bashism and bad getopts processing
>
> Dongsheng Yang (4):
> btrfs-progs: qgroup: show 'none' when we did not limit it on this qgroup
> btrfs-progs: qgroup: allow user to clear some limitation on qgroup.
> btrfs-progs: qgroup limit: error out if input value is negative
> btrfs-progs: qgroup limit: add a check for invalid input of 'T/G/M/K'
>
> Emil Karlson (1):
> btrfs-progs: use openat for process_clone in receive
>
> Goffredo Baroncelli (4):
> btrfs-progs: add strdup in btrfs_add_to_fsid() to track the device path
> btrfs-progs: return the fsid from make_btrfs()
> btrfs-progs: mkfs: track sizes of created block groups
> btrfs-progs: mkfs: print the summary
>
> Jeff Mahoney (8):
> btrfs-progs: convert: clean up blk_iterate_data handling wrt record_file_blocks
> btrfs-progs: convert: remove unused fs argument from block_iterate_proc
> btrfs-progs: convert: remove unused inode_key in copy_single_inode
> btrfs-progs: convert: rename ext2_root to image_root
> btrfs-progs: compat: define DIV_ROUND_UP if not already defined
> btrfs-progs: convert: fix typo in btrfs_insert_dir_item call
> btrfs-progs: convert: factor out adding dirent into convert_insert_dirent
> btrfs-progs: convert: factor out block iteration callback
>
> Josef Bacik (3):
> Btrfs-progs: corrupt-block: add the ability to remove csums
> btrfs-progs: specify mountpoint for recieve
> btrfs-progs: make receive work inside of subvolumes
>
> Qu Wenruo (13):
> btrfs-progs: Enhance read_tree_block to avoid memory corruption
> btrfs-progs: btrfstune: rework change_uuid
> btrfs-progs: btrfstune: add ability to restore unfinished fsid change
> btrfs-progs: btrfstune: add '-U' and '-u' option to change fsid
> btrfs-progs: Documentation: uuid change
> btrfs-progs: btrfstune: fix a bug which makes unfinished fsid change unrecoverable
> btrfs-progs: export read_extent_data function
> btrfs-progs: map-logical: introduce map_one_extent function
> Btrfs-progs: map-logical: introduce print_mapping_info function
> Btrfs-progs: map-logical: introduce write_extent_content function
> btrfs-progs: map-logical: Rework map-logical logics
> btrfs-progs: Allow "filesystem show" command to handle different units
> btrfs-progs: docs: new size options for fi show
>
> Sam Tygier (1):
> btrfs-progs: mkfs: check metadata redundancy
>
> David Sterba (79):
> btrfs-progs: tests: log the test name in results file
> btrfs-progs: tests: support more formats of test images
> btrfs-progs: tests: use readlink -f to resolve path
> btrfs-progs: tests: sort image files
> btrfs-progs: test: 015-check-bad-memory-access
> btrfs-progs: tests: add script to clean intermediate images
> btrfs-progs: receive: restore capabilities after chown
> btrfs-progs: tests: add driver script for misc tests
> btrfs-progs: tests: common: add helper run_check_stdout
> btrfs-progs: tests: add misc test for fs features
> btrfs-progs: update .gitignore
> btrfs-progs: subvol show: print received uuid
> btrfs-progs: subvol show: tweak output
> btrfs-progs: btrfstune: rework option handling
> btrfs-progs: btrfstune: add option to enable NO_HOLES
> btrfs-progs: doc: update btrfstune manpage
> btrfs-progs: tests: add misc tests for uuid rewrite
> btrfs-progs: btrfstune: make uuid rewrite progress more verbose
> btrfs-progs: ioctl.h: reorder to match kernel uapi header
> btrfs-progs: ioctl.h: copy commented version of scrub_progress structure
> btrfs-progs: ioctl.h: copy commented version of balance structures
> btrfs-progs: ioctl.h: add EXTENT_SAME ioctl and structures
> btrfs-progs: ioctl.h: add send flag mask
> btrfs-progs: ioctl.h: update type annotations
> btrfs-progs: ioctl.h: update btrfs_ioctl_fs_info_args
> btrfs-progs: ioctl.h: add btrfs_ioctl_feature_flags struct
> btrfs-progs: doc: update btrfs-rescue page and drop btrfs-zero-log
> btrfs-progs: scrub: more friendly duration format in status
> btrfs-progs: add function to stringify filesystem features
> btrfs-progs: convert: allow to set filesystem features
> btrfs-progs: tests: update convert tests to set fs features
> btrfs-progs: cleanup and deprecate btrfs-zero-log
> btrfs-progs: tests: add test for zero-log
> btrfs-progs: mkfs: add option to make it quiet
> btrfs-progs: mkfs: accept mixed case for profile names
> btrfs-progs: mkfs: move info message out of btrfs_add_to_fsid
> btrfs-progs: mkfs: do not try to register non-block-devices
> btrfs-progs: mkfs, move uuid to the end of device summary
> btrfs-progs: mkfs, drop UUID from device summary
> btrfs-progs: mkfs: drop mixed from summary
> btrfs-progs: mkfs: make the summary more compact
> btrfs-progs: remove stray deprecation notice
> btrfs-progs: fix argv0_buf handling
> btrfs-progs: add command group info strings
> btrfs-progs: print compact help for btrfs
> btrfs-progs: receive: fix minor resource leak
> btrfs-progs: cleanup after errors in open_file_or_dir3
> btrfs-progs: subvol: let sync check only current deletions
> btrfs-progs: add helper to wait for subvolume cleaning
> btrfs-progs: add missing includes to header files
> btrfs-progs: accept --help as option in the standalone utilities
> btrfs-progs: print error within test_dev_for_mkfs
> btrfs-progs: print error within test_num_disk_vs_raid
> btrfs-progs: use less memory for pretty_size_mode buffers
> btrfs-progs: receive: implement the update_extent callback
> btrfs-progs: send: add option to for the no-data mode
> btrfs-progs: use PATH_MAX instead of BTRFS_PATH_NAME_MAX
> btrfs-progs: properly set up ioctl arguments
> btrfs-progs: receive: use static buffer for cur_subvol
> btrfs-progs: add helper for copying paths
> btrfs-progs: receive: use static buffer for mount point option argument
> btrfs-progs: receive: use static buffer for source file
> btrfs-progs: send: use static buffer for output file name
> btrfs-progs: and new path_cat helpers to send utils
> btrfs-progs: send utils: switch callbacks to new helpers
> btrfs-progs: unify naming of key types in print-tree
> btrfs-progs: receive: use static buffer for root_subvol_path
> btrfs-progs: receive: use static buffer for write_path
> btrfs-progs: send utils: switch callbacks to patch_cat3_out
> btrfs-progs: receive: use static buffer for cur_subvol path
> btrfs-progs: send utils: deprecate path_cat and path_cat3
> btrfs-progs: Makefile: fix typo, dependencies for cmds-* files
> Btrfs progs v4.1-rc1
> btrfs-progs: tests: verify btrfstune output during uuid-rewrite
> btrfs-progs: btrfstune: print lowercase uuid during uuid rewrite
> btrfs-progs: btrfstune: print correct current uuid during rewrite
> btrfs-progs: configure: fix typo in summary
> btrfs-progs: INSTALL: fix typos
> Btrfs progs v4.1
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>
--
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-22 16:18 ` Goffredo Baroncelli
@ 2015-06-22 18:01 ` David Sterba
0 siblings, 0 replies; 24+ messages in thread
From: David Sterba @ 2015-06-22 18:01 UTC (permalink / raw)
To: kreijack; +Cc: linux-btrfs, clm
On Mon, Jun 22, 2015 at 06:18:35PM +0200, Goffredo Baroncelli wrote:
> Many thanks for your work.
> BTW just for curiosity: is it a coincidence that both Torvalds and you
> released the kernel 4.1/btrfs-progs 4.1 in the same day ? I know that
> the version are coupled, but also the same day....
This time around I was ready to do the release on time so there was no
reason to delay it. Previous releases were delayed because of other work
or (my) insufficient confidence in the pending changes.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
^ permalink raw reply [flat|nested] 24+ messages in thread
* qgroup limit clearing, was Re: Btrfs progs release 4.1
2015-06-22 15:00 Btrfs progs release 4.1 David Sterba
2015-06-22 16:18 ` Goffredo Baroncelli
2015-06-22 18:18 ` qgroup limit clearing, was " Christian Robottom Reis
@ 2015-06-22 18:18 ` Christian Robottom Reis
2015-06-22 23:55 ` Tsutomu Itoh
2015-06-22 20:45 ` Martin Steigerwald
` (2 subsequent siblings)
5 siblings, 1 reply; 24+ messages in thread
From: Christian Robottom Reis @ 2015-06-22 18:18 UTC (permalink / raw)
To: dsterba, linux-btrfs
On Mon, Jun 22, 2015 at 05:00:23PM +0200, David Sterba wrote:
> - qgroup:
> - show: distinguish no limits and 0 limit value
> - limit: ability to clear the limit
I'm using kernel 4.1-rc7 as per:
root@riff:/var/lib/lxc/juju-trusty-lxc-template/rootfs# uname -a
Linux riff 4.1.0-040100rc7-generic #201506080035 SMP Mon Jun 8 04:36:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
But apart from still having major issues with qgroups (quota enforcement
triggers even when there seems to be plenty of free space) clearing
limits with btrfs-progs 4.1 doesn't revert back to 'none', instead
confusingly setting the quota to 16EiB. Using:
root@riff:/var/lib/lxc/juju-trusty-lxc-template/rootfs# btrfs version
btrfs-progs v4.1
I start from:
qgroupid rfer excl max_rfer max_excl
-------- ---- ---- -------- --------
0/5 2.15GiB 1.95GiB none none
0/261 1.42GiB 1.11GiB none 100.00GiB
0/265 1.09GiB 600.59MiB none 100.00GiB
0/271 793.32MiB 366.40MiB none 100.00GiB
0/274 514.96MiB 142.92MiB none 100.00GiB
I then issue:
root@riff# btrfs qgroup limit -e none 261 /var
root@riff# btrfs qgroup limit none 261 /var
I end up with:
qgroupid rfer excl max_rfer max_excl
-------- ---- ---- -------- --------
0/5 2.15GiB 1.95GiB none none
0/261 1.42GiB 1.11GiB 16.00EiB 16.00EiB
0/265 1.09GiB 600.59MiB none 100.00GiB
0/271 793.32MiB 366.40MiB none 100.00GiB
0/274 514.96MiB 142.92MiB none 100.00GiB
Is that expected?
--
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
CEO, Async Open Source | [+55 16] 9 9112 6430 | http://launchpad.net/~kiko
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
^ permalink raw reply [flat|nested] 24+ messages in thread
* qgroup limit clearing, was Re: Btrfs progs release 4.1
2015-06-22 15:00 Btrfs progs release 4.1 David Sterba
2015-06-22 16:18 ` Goffredo Baroncelli
@ 2015-06-22 18:18 ` Christian Robottom Reis
2015-06-22 18:18 ` Christian Robottom Reis
` (3 subsequent siblings)
5 siblings, 0 replies; 24+ messages in thread
From: Christian Robottom Reis @ 2015-06-22 18:18 UTC (permalink / raw)
To: dsterba, linux-btrfs
On Mon, Jun 22, 2015 at 05:00:23PM +0200, David Sterba wrote:
> - qgroup:
> - show: distinguish no limits and 0 limit value
> - limit: ability to clear the limit
I'm using kernel 4.1-rc7 as per:
root@riff:/var/lib/lxc/juju-trusty-lxc-template/rootfs# uname -a
Linux riff 4.1.0-040100rc7-generic #201506080035 SMP Mon Jun 8 04:36:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
But apart from still having major issues with qgroups (quota enforcement
triggers even when there seems to be plenty of free space) clearing
limits with btrfs-progs 4.1 doesn't revert back to 'none', instead
confusingly setting the quota to 16EiB. Using:
root@riff:/var/lib/lxc/juju-trusty-lxc-template/rootfs# btrfs version
btrfs-progs v4.1
I start from:
qgroupid rfer excl max_rfer max_excl
-------- ---- ---- -------- --------
0/5 2.15GiB 1.95GiB none none
0/261 1.42GiB 1.11GiB none 100.00GiB
0/265 1.09GiB 600.59MiB none 100.00GiB
0/271 793.32MiB 366.40MiB none 100.00GiB
0/274 514.96MiB 142.92MiB none 100.00GiB
I then issue:
root@riff# btrfs qgroup limit -e none 261 /var
root@riff# btrfs qgroup limit none 261 /var
I end up with:
qgroupid rfer excl max_rfer max_excl
-------- ---- ---- -------- --------
0/5 2.15GiB 1.95GiB none none
0/261 1.42GiB 1.11GiB 16.00EiB 16.00EiB
0/265 1.09GiB 600.59MiB none 100.00GiB
0/271 793.32MiB 366.40MiB none 100.00GiB
0/274 514.96MiB 142.92MiB none 100.00GiB
Is that expected?
--
Christian Robottom Reis | [+1] 612 888 4935 | http://launchpad.net/~kiko
Canonical VP Hyperscale | [+55 16] 9 9112 6430
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-22 15:00 Btrfs progs release 4.1 David Sterba
` (2 preceding siblings ...)
2015-06-22 18:18 ` Christian Robottom Reis
@ 2015-06-22 20:45 ` Martin Steigerwald
2015-06-23 12:53 ` David Sterba
2015-06-23 9:03 ` Sjoerd
2015-09-09 1:34 ` Qu Wenruo
5 siblings, 1 reply; 24+ messages in thread
From: Martin Steigerwald @ 2015-06-22 20:45 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs, clm
Wow, nice collection of changes!
Am Montag, 22. Juni 2015, 17:00:23 schrieb David Sterba:
> * new
> - rescure zero-log
> - btrfsune:
> - rewrite uuid on a filesystem image
> - new option to turn on NO_HOLES incompat feature
Did you think about folding btrfstune into btrfs command as well?
Thanks,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: qgroup limit clearing, was Re: Btrfs progs release 4.1
2015-06-22 18:18 ` Christian Robottom Reis
@ 2015-06-22 23:55 ` Tsutomu Itoh
2015-06-23 1:55 ` Qu Wenruo
2015-06-23 12:42 ` David Sterba
0 siblings, 2 replies; 24+ messages in thread
From: Tsutomu Itoh @ 2015-06-22 23:55 UTC (permalink / raw)
To: Christian Robottom Reis; +Cc: dsterba, linux-btrfs, Chris Mason
On 2015/06/23 3:18, Christian Robottom Reis wrote:
> On Mon, Jun 22, 2015 at 05:00:23PM +0200, David Sterba wrote:
>> - qgroup:
>> - show: distinguish no limits and 0 limit value
>> - limit: ability to clear the limit
>
> I'm using kernel 4.1-rc7 as per:
>
> root@riff:/var/lib/lxc/juju-trusty-lxc-template/rootfs# uname -a
> Linux riff 4.1.0-040100rc7-generic #201506080035 SMP Mon Jun 8 04:36:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> But apart from still having major issues with qgroups (quota enforcement
> triggers even when there seems to be plenty of free space) clearing
> limits with btrfs-progs 4.1 doesn't revert back to 'none', instead
> confusingly setting the quota to 16EiB. Using:
>
> root@riff:/var/lib/lxc/juju-trusty-lxc-template/rootfs# btrfs version
> btrfs-progs v4.1
>
> I start from:
>
> qgroupid rfer excl max_rfer max_excl
> -------- ---- ---- -------- --------
> 0/5 2.15GiB 1.95GiB none none
> 0/261 1.42GiB 1.11GiB none 100.00GiB
> 0/265 1.09GiB 600.59MiB none 100.00GiB
> 0/271 793.32MiB 366.40MiB none 100.00GiB
> 0/274 514.96MiB 142.92MiB none 100.00GiB
>
> I then issue:
>
> root@riff# btrfs qgroup limit -e none 261 /var
> root@riff# btrfs qgroup limit none 261 /var
>
> I end up with:
>
> qgroupid rfer excl max_rfer max_excl
> -------- ---- ---- -------- --------
> 0/5 2.15GiB 1.95GiB none none
> 0/261 1.42GiB 1.11GiB 16.00EiB 16.00EiB
> 0/265 1.09GiB 600.59MiB none 100.00GiB
> 0/271 793.32MiB 366.40MiB none 100.00GiB
> 0/274 514.96MiB 142.92MiB none 100.00GiB
>
> Is that expected?
>
The following fix is necessary for the kernel to display it correctly.
[PATCH] btrfs: qgroup: allow user to clear the limitation on qgroup
http://marc.info/?l=linux-btrfs&m=143331495409594&w=2
Thanks,
Tsutomu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: qgroup limit clearing, was Re: Btrfs progs release 4.1
2015-06-22 23:55 ` Tsutomu Itoh
@ 2015-06-23 1:55 ` Qu Wenruo
2015-06-23 12:42 ` David Sterba
1 sibling, 0 replies; 24+ messages in thread
From: Qu Wenruo @ 2015-06-23 1:55 UTC (permalink / raw)
To: Tsutomu Itoh, Christian Robottom Reis; +Cc: dsterba, linux-btrfs, Chris Mason
Tsutomu Itoh wrote on 2015/06/23 08:55 +0900:
> On 2015/06/23 3:18, Christian Robottom Reis wrote:
>> On Mon, Jun 22, 2015 at 05:00:23PM +0200, David Sterba wrote:
>>> - qgroup:
>>> - show: distinguish no limits and 0 limit value
>>> - limit: ability to clear the limit
>>
>> I'm using kernel 4.1-rc7 as per:
>>
>> root@riff:/var/lib/lxc/juju-trusty-lxc-template/rootfs# uname -a
>> Linux riff 4.1.0-040100rc7-generic #201506080035 SMP Mon Jun 8
>> 04:36:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>>
>> But apart from still having major issues with qgroups (quota enforcement
>> triggers even when there seems to be plenty of free space) clearing
>> limits with btrfs-progs 4.1 doesn't revert back to 'none', instead
>> confusingly setting the quota to 16EiB. Using:
>>
>> root@riff:/var/lib/lxc/juju-trusty-lxc-template/rootfs# btrfs
>> version
>> btrfs-progs v4.1
>>
>> I start from:
>>
>> qgroupid rfer excl max_rfer max_excl
>> -------- ---- ---- -------- --------
>> 0/5 2.15GiB 1.95GiB none none
>> 0/261 1.42GiB 1.11GiB none 100.00GiB
>> 0/265 1.09GiB 600.59MiB none 100.00GiB
>> 0/271 793.32MiB 366.40MiB none 100.00GiB
>> 0/274 514.96MiB 142.92MiB none 100.00GiB
>>
>> I then issue:
>>
>> root@riff# btrfs qgroup limit -e none 261 /var
>> root@riff# btrfs qgroup limit none 261 /var
>>
>> I end up with:
>>
>> qgroupid rfer excl max_rfer max_excl
>> -------- ---- ---- -------- --------
>> 0/5 2.15GiB 1.95GiB none none
>> 0/261 1.42GiB 1.11GiB 16.00EiB 16.00EiB
>> 0/265 1.09GiB 600.59MiB none 100.00GiB
>> 0/271 793.32MiB 366.40MiB none 100.00GiB
>> 0/274 514.96MiB 142.92MiB none 100.00GiB
>>
>> Is that expected?
>>
>
> The following fix is necessary for the kernel to display it correctly.
>
> [PATCH] btrfs: qgroup: allow user to clear the limitation on qgroup
> http://marc.info/?l=linux-btrfs&m=143331495409594&w=2
>
> Thanks,
> Tsutomu
I'll send a new pull request containing this patch when we done the full
test.
The pull will be mainly consisted of small cleanup and bug fixes, so it
should be quite safe, but I still want to make sure it's completely safe
anyway.
Thanks,
Qu
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-22 15:00 Btrfs progs release 4.1 David Sterba
` (3 preceding siblings ...)
2015-06-22 20:45 ` Martin Steigerwald
@ 2015-06-23 9:03 ` Sjoerd
2015-06-23 9:18 ` Qu Wenruo
2015-06-23 14:45 ` David Sterba
2015-09-09 1:34 ` Qu Wenruo
5 siblings, 2 replies; 24+ messages in thread
From: Sjoerd @ 2015-06-23 9:03 UTC (permalink / raw)
To: linux-btrfs
On Monday 22 June 2015 17:00:23 David Sterba wrote:
> Hi,
>
> btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual load
> of changes.
Would it be beneficial to update when using older kernels (3.19.x in my case)?
I can't upgrade to kernel 4.1 since my nvidia (propriety) drivers won't
support it (yet) :(
Cheers,
Sjoerd
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-23 9:03 ` Sjoerd
@ 2015-06-23 9:18 ` Qu Wenruo
2015-06-23 14:45 ` David Sterba
1 sibling, 0 replies; 24+ messages in thread
From: Qu Wenruo @ 2015-06-23 9:18 UTC (permalink / raw)
To: Sjoerd, linux-btrfs
Sjoerd wrote on 2015/06/23 11:03 +0200:
> On Monday 22 June 2015 17:00:23 David Sterba wrote:
>> Hi,
>>
>> btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual load
>> of changes.
>
> Would it be beneficial to update when using older kernels (3.19.x in my case)?
Part(most) of.
Some one like qgroup limit needs kernel support.
UI and offline tools change won't depend on kernel version.
But the UI change is most obvious one, so upgrade user-tool should give
your a quite fresh feel.
> I can't upgrade to kernel 4.1 since my nvidia (propriety) drivers won't
> support it (yet) :(
Same Nvidia binary driver here, but I just need to wait the kernel and
nvidia package enter testing repo of Arch. :)
Thanks,
Qu
>
> Cheers,
> Sjoerd
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: qgroup limit clearing, was Re: Btrfs progs release 4.1
2015-06-22 23:55 ` Tsutomu Itoh
2015-06-23 1:55 ` Qu Wenruo
@ 2015-06-23 12:42 ` David Sterba
1 sibling, 0 replies; 24+ messages in thread
From: David Sterba @ 2015-06-23 12:42 UTC (permalink / raw)
To: Tsutomu Itoh; +Cc: Christian Robottom Reis, linux-btrfs, Chris Mason
On Tue, Jun 23, 2015 at 08:55:02AM +0900, Tsutomu Itoh wrote:
> > qgroupid rfer excl max_rfer max_excl
> > -------- ---- ---- -------- --------
> > 0/261 1.42GiB 1.11GiB 16.00EiB 16.00EiB
> >
> > Is that expected?
No, thanks for the report.
> The following fix is necessary for the kernel to display it correctly.
>
> [PATCH] btrfs: qgroup: allow user to clear the limitation on qgroup
> http://marc.info/?l=linux-btrfs&m=143331495409594&w=2
I missed that it depended on a kernel change, so we should fix it in
userspace, but it's merely a cosmetic fix.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-22 20:45 ` Martin Steigerwald
@ 2015-06-23 12:53 ` David Sterba
0 siblings, 0 replies; 24+ messages in thread
From: David Sterba @ 2015-06-23 12:53 UTC (permalink / raw)
To: Martin Steigerwald; +Cc: linux-btrfs, clm
On Mon, Jun 22, 2015 at 10:45:32PM +0200, Martin Steigerwald wrote:
> Wow, nice collection of changes!
Thanks.
> Am Montag, 22. Juni 2015, 17:00:23 schrieb David Sterba:
> > * new
> > - rescure zero-log
> > - btrfsune:
> > - rewrite uuid on a filesystem image
> > - new option to turn on NO_HOLES incompat feature
>
> Did you think about folding btrfstune into btrfs command as well?
I'm a long-time proponent of deprecating standalone btrfstune utility,
but so far there's no satisfactory replacement in terms of user
interface.
We've tried to do the uuid rewrite through the properties but it was
kind of blocking the whole feature just because the interface was not
settled.
I'd like to avoid simply adding 'btrfs filesystem tune' with all the
functionality of btrfstune. The thoughts go towards a more generic
command group that could do active opertaions on the filesystem, eg.
similar the 'inspect-internal' to query various stuff.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-23 9:03 ` Sjoerd
2015-06-23 9:18 ` Qu Wenruo
@ 2015-06-23 14:45 ` David Sterba
2015-06-24 20:26 ` Sjoerd
1 sibling, 1 reply; 24+ messages in thread
From: David Sterba @ 2015-06-23 14:45 UTC (permalink / raw)
To: Sjoerd; +Cc: linux-btrfs
On Tue, Jun 23, 2015 at 11:03:47AM +0200, Sjoerd wrote:
> On Monday 22 June 2015 17:00:23 David Sterba wrote:
> > btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual load
> > of changes.
>
> Would it be beneficial to update when using older kernels (3.19.x in my case)?
> I can't upgrade to kernel 4.1 since my nvidia (propriety) drivers won't
> support it (yet) :(
The userspace tools do not depend on kernel and can be safely used with
older kernels, so I'd recommend to update.
https://btrfs.wiki.kernel.org/index.php/FAQ#Do_I_have_to_keep_my_btrfs-progs_at_the_same_version_as_my_kernel.3F
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-23 14:45 ` David Sterba
@ 2015-06-24 20:26 ` Sjoerd
2015-06-25 13:03 ` David Sterba
0 siblings, 1 reply; 24+ messages in thread
From: Sjoerd @ 2015-06-24 20:26 UTC (permalink / raw)
To: linux-btrfs
On Tuesday 23 June 2015 16:45:01 David Sterba wrote:
> On Tue, Jun 23, 2015 at 11:03:47AM +0200, Sjoerd wrote:
> > On Monday 22 June 2015 17:00:23 David Sterba wrote:
> > > btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual
> > > load
> > > of changes.
> >
> > Would it be beneficial to update when using older kernels (3.19.x in my
> > case)? I can't upgrade to kernel 4.1 since my nvidia (propriety) drivers
> > won't support it (yet) :(
>
> The userspace tools do not depend on kernel and can be safely used with
> older kernels, so I'd recommend to update.
>
> https://btrfs.wiki.kernel.org/index.php/FAQ#Do_I_have_to_keep_my_btrfs-progs
> _at_the_same_version_as_my_kernel.3F --
Thanks for the link...I updated it...The updated version breaks the
btrfsQuota.sh and btrfsQuota.py script from the wiki though :(
https://btrfs.wiki.kernel.org/index.php/Quota_support
I can live with that though ;)
Cheers,
Sjoerd
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-24 20:26 ` Sjoerd
@ 2015-06-25 13:03 ` David Sterba
0 siblings, 0 replies; 24+ messages in thread
From: David Sterba @ 2015-06-25 13:03 UTC (permalink / raw)
To: Sjoerd; +Cc: linux-btrfs
On Wed, Jun 24, 2015 at 10:26:05PM +0200, Sjoerd wrote:
> Thanks for the link...I updated it...The updated version breaks the
> btrfsQuota.sh and btrfsQuota.py script from the wiki though :(
I see, but the scripts are likely to break anytime due to changes in the
qgroup output interfaces. You should rather take them as an example. The
preferred way to is to ehnance the tools but this might be harder than
maintaining own script.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-06-22 15:00 Btrfs progs release 4.1 David Sterba
` (4 preceding siblings ...)
2015-06-23 9:03 ` Sjoerd
@ 2015-09-09 1:34 ` Qu Wenruo
2015-09-09 3:24 ` Qu Wenruo
2015-09-09 13:19 ` David Sterba
5 siblings, 2 replies; 24+ messages in thread
From: Qu Wenruo @ 2015-09-09 1:34 UTC (permalink / raw)
To: dsterba, linux-btrfs, clm
Hi David,
Sorry for the late reply, but I noticed something interesting.
David Sterba wrote on 2015/06/22 17:00 +0200:
> Hi,
>
> btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual load of
> changes.
>
> Fixed since rc1:
> - uuid rewrite prints the correct original UUID
> - map-logical updated
> - fi show size units
> - typos
>
> * bugfixes
> - fsck.btrfs: no bash-isms
> - bugzilla 97171: invalid memory access (with tests)
> - receive:
> - cloning works with --chroot
> - capabilities not lost
> - mkfs: do not try to register bare file images
> - option --help accepted by the standalone utilities
>
> * enhancements
> - corrupt block: ability to remove csums
> - mkfs:
> - warn if metadata redundancy is lower than for data
> - options to make the output quiet (only errors)
> - mixed case names of raid profiles accepted
> - rework the output:
> - more comprehensive, 'key: value' format
> - subvol:
> - show:
> - print received uuid
> - update the output
> - new options to specify size units
> - sync:
> - grab all deleted ids and print them as they're removed,
> previous implementation only checked if there are any
> to be deleted - change in command semantics
> - scrub: print timestamps in days HMS format
> - receive:
> - can specify mount point, do not rely on /proc
> - can work inside subvolumes
> - send:
> - new option to send stream without data (NO_FILE_DATA)
> - convert:
> - specify incompat features on the new fs
> - qgroup:
> - show: distinguish no limits and 0 limit value
> - limit: ability to clear the limit
> - help for 'btrfs' is shorter, 1st level command overview
> - debug tree: print key names according to their C name
>
> * new
> - rescure zero-log
> - btrfsune:
> - rewrite uuid on a filesystem image
> - new option to turn on NO_HOLES incompat feature
>
> * deprecated
> - standalone btrfs-zero-log
>
> * other
> - testing framework updates
> - uuid rewrite test
> - btrfstune feature setting test
> - zero-log tests
> - more testing image formats
> - manual page updates
> - ioctl.h synced with current kernel uapi version
> - convert: preparatory works for more filesystems (reiserfs pending)
> - use static buffers for path handling where possible
> - add new helpers for send uilts that check memory allocations,
> switch all users, deprecate old helpers
> - Makefile: fix build dependency generation
> - map-logical: make it work again
>
> Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
> Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
>
> Shortlog:
>
> Anand Jain (2):
> btrfs-progs: add info about list-all to the help
> btrfs-progs: use function is_block_device() instead
>
> Dimitri John Ledkov (1):
> btrfs-progs: fsck.btrfs: Fix bashism and bad getopts processing
>
> Dongsheng Yang (4):
> btrfs-progs: qgroup: show 'none' when we did not limit it on this qgroup
> btrfs-progs: qgroup: allow user to clear some limitation on qgroup.
> btrfs-progs: qgroup limit: error out if input value is negative
> btrfs-progs: qgroup limit: add a check for invalid input of 'T/G/M/K'
>
> Emil Karlson (1):
> btrfs-progs: use openat for process_clone in receive
>
> Goffredo Baroncelli (4):
> btrfs-progs: add strdup in btrfs_add_to_fsid() to track the device path
> btrfs-progs: return the fsid from make_btrfs()
> btrfs-progs: mkfs: track sizes of created block groups
> btrfs-progs: mkfs: print the summary
>
> Jeff Mahoney (8):
> btrfs-progs: convert: clean up blk_iterate_data handling wrt record_file_blocks
> btrfs-progs: convert: remove unused fs argument from block_iterate_proc
> btrfs-progs: convert: remove unused inode_key in copy_single_inode
> btrfs-progs: convert: rename ext2_root to image_root
> btrfs-progs: compat: define DIV_ROUND_UP if not already defined
> btrfs-progs: convert: fix typo in btrfs_insert_dir_item call
> btrfs-progs: convert: factor out adding dirent into convert_insert_dirent
> btrfs-progs: convert: factor out block iteration callback
>
> Josef Bacik (3):
> Btrfs-progs: corrupt-block: add the ability to remove csums
> btrfs-progs: specify mountpoint for recieve
> btrfs-progs: make receive work inside of subvolumes
>
> Qu Wenruo (13):
> btrfs-progs: Enhance read_tree_block to avoid memory corruption
> btrfs-progs: btrfstune: rework change_uuid
> btrfs-progs: btrfstune: add ability to restore unfinished fsid change
> btrfs-progs: btrfstune: add '-U' and '-u' option to change fsid
> btrfs-progs: Documentation: uuid change
> btrfs-progs: btrfstune: fix a bug which makes unfinished fsid change unrecoverable
> btrfs-progs: export read_extent_data function
> btrfs-progs: map-logical: introduce map_one_extent function
> Btrfs-progs: map-logical: introduce print_mapping_info function
> Btrfs-progs: map-logical: introduce write_extent_content function
> btrfs-progs: map-logical: Rework map-logical logics
> btrfs-progs: Allow "filesystem show" command to handle different units
> btrfs-progs: docs: new size options for fi show
>
> Sam Tygier (1):
> btrfs-progs: mkfs: check metadata redundancy
>
> David Sterba (79):
> btrfs-progs: tests: log the test name in results file
> btrfs-progs: tests: support more formats of test images
> btrfs-progs: tests: use readlink -f to resolve path
> btrfs-progs: tests: sort image files
> btrfs-progs: test: 015-check-bad-memory-access
> btrfs-progs: tests: add script to clean intermediate images
> btrfs-progs: receive: restore capabilities after chown
> btrfs-progs: tests: add driver script for misc tests
> btrfs-progs: tests: common: add helper run_check_stdout
> btrfs-progs: tests: add misc test for fs features
> btrfs-progs: update .gitignore
> btrfs-progs: subvol show: print received uuid
> btrfs-progs: subvol show: tweak output
> btrfs-progs: btrfstune: rework option handling
> btrfs-progs: btrfstune: add option to enable NO_HOLES
> btrfs-progs: doc: update btrfstune manpage
> btrfs-progs: tests: add misc tests for uuid rewrite
> btrfs-progs: btrfstune: make uuid rewrite progress more verbose
> btrfs-progs: ioctl.h: reorder to match kernel uapi header
> btrfs-progs: ioctl.h: copy commented version of scrub_progress structure
> btrfs-progs: ioctl.h: copy commented version of balance structures
> btrfs-progs: ioctl.h: add EXTENT_SAME ioctl and structures
> btrfs-progs: ioctl.h: add send flag mask
> btrfs-progs: ioctl.h: update type annotations
> btrfs-progs: ioctl.h: update btrfs_ioctl_fs_info_args
> btrfs-progs: ioctl.h: add btrfs_ioctl_feature_flags struct
> btrfs-progs: doc: update btrfs-rescue page and drop btrfs-zero-log
> btrfs-progs: scrub: more friendly duration format in status
> btrfs-progs: add function to stringify filesystem features
> btrfs-progs: convert: allow to set filesystem features
Tons of new patches, but at least I didn't find the patch in mail list
for the above convert patch, when I want to comment on it.
So I can only comment here.
[[Good fix even before we found the real bug]]
This convert patch is quite good, as it in fact addressed the
metadata-cross-map-boundary bug, even before we found it.
For the metadata crossing map boundary bug, the root cause was, for old
convert, we were using mixed block group by default but use wrong
node/leaf size.
For mixed block group case, node/leaf size must be the same with
sectorsize(4K on x86/x86_64), but we were using 16K leaf/node size in
that case. And causing tree blocks crossing map boundary.
I'm a little surprised kernel can still mount such image without major
bugs. But anyway, we still need to add new kernel check.
[[Side effect]]
On the other hand, for convert case, unlike btrfs, ext2/3 is not extent
based, and their data/metadata aren't batched together.
Ext4 is a little better, but still not so extent-based like btrf.
If not using mixed block group, even for a newly created ext4 image,
btrfs-convert will create 13 block groups for the scattered ext4 data.
And things will be even worse if the ext4 image contains some data.
[[True fix]]
IMHO, I still prefer to use mixed block group for btrfs-convert, as that
fits better for the use case.
And what we really need is, to keep fixing btrfs-convert and kernel.
Banning mixed bg for convert case is just avoiding the real cause, and I
think that should be the last method.
Thanks,
Qu
> btrfs-progs: tests: update convert tests to set fs features
> btrfs-progs: cleanup and deprecate btrfs-zero-log
> btrfs-progs: tests: add test for zero-log
> btrfs-progs: mkfs: add option to make it quiet
> btrfs-progs: mkfs: accept mixed case for profile names
> btrfs-progs: mkfs: move info message out of btrfs_add_to_fsid
> btrfs-progs: mkfs: do not try to register non-block-devices
> btrfs-progs: mkfs, move uuid to the end of device summary
> btrfs-progs: mkfs, drop UUID from device summary
> btrfs-progs: mkfs: drop mixed from summary
> btrfs-progs: mkfs: make the summary more compact
> btrfs-progs: remove stray deprecation notice
> btrfs-progs: fix argv0_buf handling
> btrfs-progs: add command group info strings
> btrfs-progs: print compact help for btrfs
> btrfs-progs: receive: fix minor resource leak
> btrfs-progs: cleanup after errors in open_file_or_dir3
> btrfs-progs: subvol: let sync check only current deletions
> btrfs-progs: add helper to wait for subvolume cleaning
> btrfs-progs: add missing includes to header files
> btrfs-progs: accept --help as option in the standalone utilities
> btrfs-progs: print error within test_dev_for_mkfs
> btrfs-progs: print error within test_num_disk_vs_raid
> btrfs-progs: use less memory for pretty_size_mode buffers
> btrfs-progs: receive: implement the update_extent callback
> btrfs-progs: send: add option to for the no-data mode
> btrfs-progs: use PATH_MAX instead of BTRFS_PATH_NAME_MAX
> btrfs-progs: properly set up ioctl arguments
> btrfs-progs: receive: use static buffer for cur_subvol
> btrfs-progs: add helper for copying paths
> btrfs-progs: receive: use static buffer for mount point option argument
> btrfs-progs: receive: use static buffer for source file
> btrfs-progs: send: use static buffer for output file name
> btrfs-progs: and new path_cat helpers to send utils
> btrfs-progs: send utils: switch callbacks to new helpers
> btrfs-progs: unify naming of key types in print-tree
> btrfs-progs: receive: use static buffer for root_subvol_path
> btrfs-progs: receive: use static buffer for write_path
> btrfs-progs: send utils: switch callbacks to patch_cat3_out
> btrfs-progs: receive: use static buffer for cur_subvol path
> btrfs-progs: send utils: deprecate path_cat and path_cat3
> btrfs-progs: Makefile: fix typo, dependencies for cmds-* files
> Btrfs progs v4.1-rc1
> btrfs-progs: tests: verify btrfstune output during uuid-rewrite
> btrfs-progs: btrfstune: print lowercase uuid during uuid rewrite
> btrfs-progs: btrfstune: print correct current uuid during rewrite
> btrfs-progs: configure: fix typo in summary
> btrfs-progs: INSTALL: fix typos
> Btrfs progs v4.1
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-09-09 1:34 ` Qu Wenruo
@ 2015-09-09 3:24 ` Qu Wenruo
2015-09-09 10:35 ` Vytautas D
2015-09-09 13:25 ` David Sterba
2015-09-09 13:19 ` David Sterba
1 sibling, 2 replies; 24+ messages in thread
From: Qu Wenruo @ 2015-09-09 3:24 UTC (permalink / raw)
To: dsterba, linux-btrfs, clm
Qu Wenruo wrote on 2015/09/09 09:34 +0800:
> Hi David,
>
> Sorry for the late reply, but I noticed something interesting.
>
> David Sterba wrote on 2015/06/22 17:00 +0200:
>> Hi,
>>
>> btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual
>> load of
>> changes.
>>
>> Fixed since rc1:
>> - uuid rewrite prints the correct original UUID
>> - map-logical updated
>> - fi show size units
>> - typos
>>
>> * bugfixes
>> - fsck.btrfs: no bash-isms
>> - bugzilla 97171: invalid memory access (with tests)
>> - receive:
>> - cloning works with --chroot
>> - capabilities not lost
>> - mkfs: do not try to register bare file images
>> - option --help accepted by the standalone utilities
>>
>> * enhancements
>> - corrupt block: ability to remove csums
>> - mkfs:
>> - warn if metadata redundancy is lower than for data
>> - options to make the output quiet (only errors)
>> - mixed case names of raid profiles accepted
>> - rework the output:
>> - more comprehensive, 'key: value' format
>> - subvol:
>> - show:
>> - print received uuid
>> - update the output
>> - new options to specify size units
>> - sync:
>> - grab all deleted ids and print them as they're removed,
>> previous implementation only checked if there are any
>> to be deleted - change in command semantics
>> - scrub: print timestamps in days HMS format
>> - receive:
>> - can specify mount point, do not rely on /proc
>> - can work inside subvolumes
>> - send:
>> - new option to send stream without data (NO_FILE_DATA)
>> - convert:
>> - specify incompat features on the new fs
>> - qgroup:
>> - show: distinguish no limits and 0 limit value
>> - limit: ability to clear the limit
>> - help for 'btrfs' is shorter, 1st level command overview
>> - debug tree: print key names according to their C name
>>
>> * new
>> - rescure zero-log
>> - btrfsune:
>> - rewrite uuid on a filesystem image
>> - new option to turn on NO_HOLES incompat feature
>>
>> * deprecated
>> - standalone btrfs-zero-log
>>
>> * other
>> - testing framework updates
>> - uuid rewrite test
>> - btrfstune feature setting test
>> - zero-log tests
>> - more testing image formats
>> - manual page updates
>> - ioctl.h synced with current kernel uapi version
>> - convert: preparatory works for more filesystems (reiserfs pending)
>> - use static buffers for path handling where possible
>> - add new helpers for send uilts that check memory allocations,
>> switch all users, deprecate old helpers
>> - Makefile: fix build dependency generation
>> - map-logical: make it work again
>>
>> Tarballs:
>> https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
>> Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
>>
>> Shortlog:
>>
>> Anand Jain (2):
>> btrfs-progs: add info about list-all to the help
>> btrfs-progs: use function is_block_device() instead
>>
>> Dimitri John Ledkov (1):
>> btrfs-progs: fsck.btrfs: Fix bashism and bad getopts processing
>>
>> Dongsheng Yang (4):
>> btrfs-progs: qgroup: show 'none' when we did not limit it on
>> this qgroup
>> btrfs-progs: qgroup: allow user to clear some limitation on
>> qgroup.
>> btrfs-progs: qgroup limit: error out if input value is negative
>> btrfs-progs: qgroup limit: add a check for invalid input of
>> 'T/G/M/K'
>>
>> Emil Karlson (1):
>> btrfs-progs: use openat for process_clone in receive
>>
>> Goffredo Baroncelli (4):
>> btrfs-progs: add strdup in btrfs_add_to_fsid() to track the
>> device path
>> btrfs-progs: return the fsid from make_btrfs()
>> btrfs-progs: mkfs: track sizes of created block groups
>> btrfs-progs: mkfs: print the summary
>>
>> Jeff Mahoney (8):
>> btrfs-progs: convert: clean up blk_iterate_data handling wrt
>> record_file_blocks
>> btrfs-progs: convert: remove unused fs argument from
>> block_iterate_proc
>> btrfs-progs: convert: remove unused inode_key in copy_single_inode
>> btrfs-progs: convert: rename ext2_root to image_root
>> btrfs-progs: compat: define DIV_ROUND_UP if not already defined
>> btrfs-progs: convert: fix typo in btrfs_insert_dir_item call
>> btrfs-progs: convert: factor out adding dirent into
>> convert_insert_dirent
>> btrfs-progs: convert: factor out block iteration callback
>>
>> Josef Bacik (3):
>> Btrfs-progs: corrupt-block: add the ability to remove csums
>> btrfs-progs: specify mountpoint for recieve
>> btrfs-progs: make receive work inside of subvolumes
>>
>> Qu Wenruo (13):
>> btrfs-progs: Enhance read_tree_block to avoid memory corruption
>> btrfs-progs: btrfstune: rework change_uuid
>> btrfs-progs: btrfstune: add ability to restore unfinished fsid
>> change
>> btrfs-progs: btrfstune: add '-U' and '-u' option to change fsid
>> btrfs-progs: Documentation: uuid change
>> btrfs-progs: btrfstune: fix a bug which makes unfinished fsid
>> change unrecoverable
>> btrfs-progs: export read_extent_data function
>> btrfs-progs: map-logical: introduce map_one_extent function
>> Btrfs-progs: map-logical: introduce print_mapping_info function
>> Btrfs-progs: map-logical: introduce write_extent_content function
>> btrfs-progs: map-logical: Rework map-logical logics
>> btrfs-progs: Allow "filesystem show" command to handle
>> different units
>> btrfs-progs: docs: new size options for fi show
>>
>> Sam Tygier (1):
>> btrfs-progs: mkfs: check metadata redundancy
>>
>> David Sterba (79):
>> btrfs-progs: tests: log the test name in results file
>> btrfs-progs: tests: support more formats of test images
>> btrfs-progs: tests: use readlink -f to resolve path
>> btrfs-progs: tests: sort image files
>> btrfs-progs: test: 015-check-bad-memory-access
>> btrfs-progs: tests: add script to clean intermediate images
>> btrfs-progs: receive: restore capabilities after chown
>> btrfs-progs: tests: add driver script for misc tests
>> btrfs-progs: tests: common: add helper run_check_stdout
>> btrfs-progs: tests: add misc test for fs features
>> btrfs-progs: update .gitignore
>> btrfs-progs: subvol show: print received uuid
>> btrfs-progs: subvol show: tweak output
>> btrfs-progs: btrfstune: rework option handling
>> btrfs-progs: btrfstune: add option to enable NO_HOLES
>> btrfs-progs: doc: update btrfstune manpage
>> btrfs-progs: tests: add misc tests for uuid rewrite
>> btrfs-progs: btrfstune: make uuid rewrite progress more verbose
>> btrfs-progs: ioctl.h: reorder to match kernel uapi header
>> btrfs-progs: ioctl.h: copy commented version of scrub_progress
>> structure
>> btrfs-progs: ioctl.h: copy commented version of balance structures
>> btrfs-progs: ioctl.h: add EXTENT_SAME ioctl and structures
>> btrfs-progs: ioctl.h: add send flag mask
>> btrfs-progs: ioctl.h: update type annotations
>> btrfs-progs: ioctl.h: update btrfs_ioctl_fs_info_args
>> btrfs-progs: ioctl.h: add btrfs_ioctl_feature_flags struct
>> btrfs-progs: doc: update btrfs-rescue page and drop btrfs-zero-log
>> btrfs-progs: scrub: more friendly duration format in status
>> btrfs-progs: add function to stringify filesystem features
>> btrfs-progs: convert: allow to set filesystem features
>
> Tons of new patches, but at least I didn't find the patch in mail list
> for the above convert patch, when I want to comment on it.
>
> So I can only comment here.
>
> [[Good fix even before we found the real bug]]
> This convert patch is quite good, as it in fact addressed the
> metadata-cross-map-boundary bug, even before we found it.
>
> For the metadata crossing map boundary bug, the root cause was, for old
> convert, we were using mixed block group by default but use wrong
> node/leaf size.
>
> For mixed block group case, node/leaf size must be the same with
> sectorsize(4K on x86/x86_64), but we were using 16K leaf/node size in
> that case. And causing tree blocks crossing map boundary.
>
> I'm a little surprised kernel can still mount such image without major
> bugs. But anyway, we still need to add new kernel check.
Sorry David, I found this explain is wrong, as kernel already has
checked this case.
The problem is that, for converted image, it's quite possible that data
and metadata extent are stored in one chunk even the chunk is not mixed.
I'll add fsck support for it soon.
BTW, the side effect still exists and may worth consideration.
Thanks,
Qu
>
> [[Side effect]]
> On the other hand, for convert case, unlike btrfs, ext2/3 is not extent
> based, and their data/metadata aren't batched together.
> Ext4 is a little better, but still not so extent-based like btrf.
>
> If not using mixed block group, even for a newly created ext4 image,
> btrfs-convert will create 13 block groups for the scattered ext4 data.
> And things will be even worse if the ext4 image contains some data.
>
> [[True fix]]
> IMHO, I still prefer to use mixed block group for btrfs-convert, as that
> fits better for the use case.
>
> And what we really need is, to keep fixing btrfs-convert and kernel.
> Banning mixed bg for convert case is just avoiding the real cause, and I
> think that should be the last method.
>
> Thanks,
> Qu
>
>> btrfs-progs: tests: update convert tests to set fs features
>> btrfs-progs: cleanup and deprecate btrfs-zero-log
>> btrfs-progs: tests: add test for zero-log
>> btrfs-progs: mkfs: add option to make it quiet
>> btrfs-progs: mkfs: accept mixed case for profile names
>> btrfs-progs: mkfs: move info message out of btrfs_add_to_fsid
>> btrfs-progs: mkfs: do not try to register non-block-devices
>> btrfs-progs: mkfs, move uuid to the end of device summary
>> btrfs-progs: mkfs, drop UUID from device summary
>> btrfs-progs: mkfs: drop mixed from summary
>> btrfs-progs: mkfs: make the summary more compact
>> btrfs-progs: remove stray deprecation notice
>> btrfs-progs: fix argv0_buf handling
>> btrfs-progs: add command group info strings
>> btrfs-progs: print compact help for btrfs
>> btrfs-progs: receive: fix minor resource leak
>> btrfs-progs: cleanup after errors in open_file_or_dir3
>> btrfs-progs: subvol: let sync check only current deletions
>> btrfs-progs: add helper to wait for subvolume cleaning
>> btrfs-progs: add missing includes to header files
>> btrfs-progs: accept --help as option in the standalone utilities
>> btrfs-progs: print error within test_dev_for_mkfs
>> btrfs-progs: print error within test_num_disk_vs_raid
>> btrfs-progs: use less memory for pretty_size_mode buffers
>> btrfs-progs: receive: implement the update_extent callback
>> btrfs-progs: send: add option to for the no-data mode
>> btrfs-progs: use PATH_MAX instead of BTRFS_PATH_NAME_MAX
>> btrfs-progs: properly set up ioctl arguments
>> btrfs-progs: receive: use static buffer for cur_subvol
>> btrfs-progs: add helper for copying paths
>> btrfs-progs: receive: use static buffer for mount point option
>> argument
>> btrfs-progs: receive: use static buffer for source file
>> btrfs-progs: send: use static buffer for output file name
>> btrfs-progs: and new path_cat helpers to send utils
>> btrfs-progs: send utils: switch callbacks to new helpers
>> btrfs-progs: unify naming of key types in print-tree
>> btrfs-progs: receive: use static buffer for root_subvol_path
>> btrfs-progs: receive: use static buffer for write_path
>> btrfs-progs: send utils: switch callbacks to patch_cat3_out
>> btrfs-progs: receive: use static buffer for cur_subvol path
>> btrfs-progs: send utils: deprecate path_cat and path_cat3
>> btrfs-progs: Makefile: fix typo, dependencies for cmds-* files
>> Btrfs progs v4.1-rc1
>> btrfs-progs: tests: verify btrfstune output during uuid-rewrite
>> btrfs-progs: btrfstune: print lowercase uuid during uuid rewrite
>> btrfs-progs: btrfstune: print correct current uuid during rewrite
>> btrfs-progs: configure: fix typo in summary
>> btrfs-progs: INSTALL: fix typos
>> Btrfs progs v4.1
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-09-09 3:24 ` Qu Wenruo
@ 2015-09-09 10:35 ` Vytautas D
2015-09-09 11:25 ` Duncan
2015-09-09 13:26 ` David Sterba
2015-09-09 13:25 ` David Sterba
1 sibling, 2 replies; 24+ messages in thread
From: Vytautas D @ 2015-09-09 10:35 UTC (permalink / raw)
To: Qu Wenruo; +Cc: dsterba, Btrfs BTRFS, clm
sorry for side question - but does this mean that btrfs-convert issues
others were having on kernel 4.0+ on this mailing list is now
addressed ?
On Wed, Sep 9, 2015 at 4:24 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> Qu Wenruo wrote on 2015/09/09 09:34 +0800:
>>
>> Hi David,
>>
>> Sorry for the late reply, but I noticed something interesting.
>>
>> David Sterba wrote on 2015/06/22 17:00 +0200:
>>>
>>> Hi,
>>>
>>> btrfs-progs 4.1 have been released (in time with kernel 4.1). Unusual
>>> load of
>>> changes.
>>>
>>> Fixed since rc1:
>>> - uuid rewrite prints the correct original UUID
>>> - map-logical updated
>>> - fi show size units
>>> - typos
>>>
>>> * bugfixes
>>> - fsck.btrfs: no bash-isms
>>> - bugzilla 97171: invalid memory access (with tests)
>>> - receive:
>>> - cloning works with --chroot
>>> - capabilities not lost
>>> - mkfs: do not try to register bare file images
>>> - option --help accepted by the standalone utilities
>>>
>>> * enhancements
>>> - corrupt block: ability to remove csums
>>> - mkfs:
>>> - warn if metadata redundancy is lower than for data
>>> - options to make the output quiet (only errors)
>>> - mixed case names of raid profiles accepted
>>> - rework the output:
>>> - more comprehensive, 'key: value' format
>>> - subvol:
>>> - show:
>>> - print received uuid
>>> - update the output
>>> - new options to specify size units
>>> - sync:
>>> - grab all deleted ids and print them as they're removed,
>>> previous implementation only checked if there are any
>>> to be deleted - change in command semantics
>>> - scrub: print timestamps in days HMS format
>>> - receive:
>>> - can specify mount point, do not rely on /proc
>>> - can work inside subvolumes
>>> - send:
>>> - new option to send stream without data (NO_FILE_DATA)
>>> - convert:
>>> - specify incompat features on the new fs
>>> - qgroup:
>>> - show: distinguish no limits and 0 limit value
>>> - limit: ability to clear the limit
>>> - help for 'btrfs' is shorter, 1st level command overview
>>> - debug tree: print key names according to their C name
>>>
>>> * new
>>> - rescure zero-log
>>> - btrfsune:
>>> - rewrite uuid on a filesystem image
>>> - new option to turn on NO_HOLES incompat feature
>>>
>>> * deprecated
>>> - standalone btrfs-zero-log
>>>
>>> * other
>>> - testing framework updates
>>> - uuid rewrite test
>>> - btrfstune feature setting test
>>> - zero-log tests
>>> - more testing image formats
>>> - manual page updates
>>> - ioctl.h synced with current kernel uapi version
>>> - convert: preparatory works for more filesystems (reiserfs pending)
>>> - use static buffers for path handling where possible
>>> - add new helpers for send uilts that check memory allocations,
>>> switch all users, deprecate old helpers
>>> - Makefile: fix build dependency generation
>>> - map-logical: make it work again
>>>
>>> Tarballs:
>>> https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
>>> Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
>>>
>>> Shortlog:
>>>
>>> Anand Jain (2):
>>> btrfs-progs: add info about list-all to the help
>>> btrfs-progs: use function is_block_device() instead
>>>
>>> Dimitri John Ledkov (1):
>>> btrfs-progs: fsck.btrfs: Fix bashism and bad getopts processing
>>>
>>> Dongsheng Yang (4):
>>> btrfs-progs: qgroup: show 'none' when we did not limit it on
>>> this qgroup
>>> btrfs-progs: qgroup: allow user to clear some limitation on
>>> qgroup.
>>> btrfs-progs: qgroup limit: error out if input value is negative
>>> btrfs-progs: qgroup limit: add a check for invalid input of
>>> 'T/G/M/K'
>>>
>>> Emil Karlson (1):
>>> btrfs-progs: use openat for process_clone in receive
>>>
>>> Goffredo Baroncelli (4):
>>> btrfs-progs: add strdup in btrfs_add_to_fsid() to track the
>>> device path
>>> btrfs-progs: return the fsid from make_btrfs()
>>> btrfs-progs: mkfs: track sizes of created block groups
>>> btrfs-progs: mkfs: print the summary
>>>
>>> Jeff Mahoney (8):
>>> btrfs-progs: convert: clean up blk_iterate_data handling wrt
>>> record_file_blocks
>>> btrfs-progs: convert: remove unused fs argument from
>>> block_iterate_proc
>>> btrfs-progs: convert: remove unused inode_key in copy_single_inode
>>> btrfs-progs: convert: rename ext2_root to image_root
>>> btrfs-progs: compat: define DIV_ROUND_UP if not already defined
>>> btrfs-progs: convert: fix typo in btrfs_insert_dir_item call
>>> btrfs-progs: convert: factor out adding dirent into
>>> convert_insert_dirent
>>> btrfs-progs: convert: factor out block iteration callback
>>>
>>> Josef Bacik (3):
>>> Btrfs-progs: corrupt-block: add the ability to remove csums
>>> btrfs-progs: specify mountpoint for recieve
>>> btrfs-progs: make receive work inside of subvolumes
>>>
>>> Qu Wenruo (13):
>>> btrfs-progs: Enhance read_tree_block to avoid memory corruption
>>> btrfs-progs: btrfstune: rework change_uuid
>>> btrfs-progs: btrfstune: add ability to restore unfinished fsid
>>> change
>>> btrfs-progs: btrfstune: add '-U' and '-u' option to change fsid
>>> btrfs-progs: Documentation: uuid change
>>> btrfs-progs: btrfstune: fix a bug which makes unfinished fsid
>>> change unrecoverable
>>> btrfs-progs: export read_extent_data function
>>> btrfs-progs: map-logical: introduce map_one_extent function
>>> Btrfs-progs: map-logical: introduce print_mapping_info function
>>> Btrfs-progs: map-logical: introduce write_extent_content function
>>> btrfs-progs: map-logical: Rework map-logical logics
>>> btrfs-progs: Allow "filesystem show" command to handle
>>> different units
>>> btrfs-progs: docs: new size options for fi show
>>>
>>> Sam Tygier (1):
>>> btrfs-progs: mkfs: check metadata redundancy
>>>
>>> David Sterba (79):
>>> btrfs-progs: tests: log the test name in results file
>>> btrfs-progs: tests: support more formats of test images
>>> btrfs-progs: tests: use readlink -f to resolve path
>>> btrfs-progs: tests: sort image files
>>> btrfs-progs: test: 015-check-bad-memory-access
>>> btrfs-progs: tests: add script to clean intermediate images
>>> btrfs-progs: receive: restore capabilities after chown
>>> btrfs-progs: tests: add driver script for misc tests
>>> btrfs-progs: tests: common: add helper run_check_stdout
>>> btrfs-progs: tests: add misc test for fs features
>>> btrfs-progs: update .gitignore
>>> btrfs-progs: subvol show: print received uuid
>>> btrfs-progs: subvol show: tweak output
>>> btrfs-progs: btrfstune: rework option handling
>>> btrfs-progs: btrfstune: add option to enable NO_HOLES
>>> btrfs-progs: doc: update btrfstune manpage
>>> btrfs-progs: tests: add misc tests for uuid rewrite
>>> btrfs-progs: btrfstune: make uuid rewrite progress more verbose
>>> btrfs-progs: ioctl.h: reorder to match kernel uapi header
>>> btrfs-progs: ioctl.h: copy commented version of scrub_progress
>>> structure
>>> btrfs-progs: ioctl.h: copy commented version of balance structures
>>> btrfs-progs: ioctl.h: add EXTENT_SAME ioctl and structures
>>> btrfs-progs: ioctl.h: add send flag mask
>>> btrfs-progs: ioctl.h: update type annotations
>>> btrfs-progs: ioctl.h: update btrfs_ioctl_fs_info_args
>>> btrfs-progs: ioctl.h: add btrfs_ioctl_feature_flags struct
>>> btrfs-progs: doc: update btrfs-rescue page and drop btrfs-zero-log
>>> btrfs-progs: scrub: more friendly duration format in status
>>> btrfs-progs: add function to stringify filesystem features
>>> btrfs-progs: convert: allow to set filesystem features
>>
>>
>> Tons of new patches, but at least I didn't find the patch in mail list
>> for the above convert patch, when I want to comment on it.
>>
>> So I can only comment here.
>>
>> [[Good fix even before we found the real bug]]
>> This convert patch is quite good, as it in fact addressed the
>> metadata-cross-map-boundary bug, even before we found it.
>>
>> For the metadata crossing map boundary bug, the root cause was, for old
>> convert, we were using mixed block group by default but use wrong
>> node/leaf size.
>>
>> For mixed block group case, node/leaf size must be the same with
>> sectorsize(4K on x86/x86_64), but we were using 16K leaf/node size in
>> that case. And causing tree blocks crossing map boundary.
>>
>> I'm a little surprised kernel can still mount such image without major
>> bugs. But anyway, we still need to add new kernel check.
>
> Sorry David, I found this explain is wrong, as kernel already has checked
> this case.
>
> The problem is that, for converted image, it's quite possible that data and
> metadata extent are stored in one chunk even the chunk is not mixed.
>
> I'll add fsck support for it soon.
>
> BTW, the side effect still exists and may worth consideration.
>
> Thanks,
> Qu
>
>>
>> [[Side effect]]
>> On the other hand, for convert case, unlike btrfs, ext2/3 is not extent
>> based, and their data/metadata aren't batched together.
>> Ext4 is a little better, but still not so extent-based like btrf.
>>
>> If not using mixed block group, even for a newly created ext4 image,
>> btrfs-convert will create 13 block groups for the scattered ext4 data.
>> And things will be even worse if the ext4 image contains some data.
>>
>> [[True fix]]
>> IMHO, I still prefer to use mixed block group for btrfs-convert, as that
>> fits better for the use case.
>>
>> And what we really need is, to keep fixing btrfs-convert and kernel.
>> Banning mixed bg for convert case is just avoiding the real cause, and I
>> think that should be the last method.
>>
>> Thanks,
>> Qu
>>
>>> btrfs-progs: tests: update convert tests to set fs features
>>> btrfs-progs: cleanup and deprecate btrfs-zero-log
>>> btrfs-progs: tests: add test for zero-log
>>> btrfs-progs: mkfs: add option to make it quiet
>>> btrfs-progs: mkfs: accept mixed case for profile names
>>> btrfs-progs: mkfs: move info message out of btrfs_add_to_fsid
>>> btrfs-progs: mkfs: do not try to register non-block-devices
>>> btrfs-progs: mkfs, move uuid to the end of device summary
>>> btrfs-progs: mkfs, drop UUID from device summary
>>> btrfs-progs: mkfs: drop mixed from summary
>>> btrfs-progs: mkfs: make the summary more compact
>>> btrfs-progs: remove stray deprecation notice
>>> btrfs-progs: fix argv0_buf handling
>>> btrfs-progs: add command group info strings
>>> btrfs-progs: print compact help for btrfs
>>> btrfs-progs: receive: fix minor resource leak
>>> btrfs-progs: cleanup after errors in open_file_or_dir3
>>> btrfs-progs: subvol: let sync check only current deletions
>>> btrfs-progs: add helper to wait for subvolume cleaning
>>> btrfs-progs: add missing includes to header files
>>> btrfs-progs: accept --help as option in the standalone utilities
>>> btrfs-progs: print error within test_dev_for_mkfs
>>> btrfs-progs: print error within test_num_disk_vs_raid
>>> btrfs-progs: use less memory for pretty_size_mode buffers
>>> btrfs-progs: receive: implement the update_extent callback
>>> btrfs-progs: send: add option to for the no-data mode
>>> btrfs-progs: use PATH_MAX instead of BTRFS_PATH_NAME_MAX
>>> btrfs-progs: properly set up ioctl arguments
>>> btrfs-progs: receive: use static buffer for cur_subvol
>>> btrfs-progs: add helper for copying paths
>>> btrfs-progs: receive: use static buffer for mount point option
>>> argument
>>> btrfs-progs: receive: use static buffer for source file
>>> btrfs-progs: send: use static buffer for output file name
>>> btrfs-progs: and new path_cat helpers to send utils
>>> btrfs-progs: send utils: switch callbacks to new helpers
>>> btrfs-progs: unify naming of key types in print-tree
>>> btrfs-progs: receive: use static buffer for root_subvol_path
>>> btrfs-progs: receive: use static buffer for write_path
>>> btrfs-progs: send utils: switch callbacks to patch_cat3_out
>>> btrfs-progs: receive: use static buffer for cur_subvol path
>>> btrfs-progs: send utils: deprecate path_cat and path_cat3
>>> btrfs-progs: Makefile: fix typo, dependencies for cmds-* files
>>> Btrfs progs v4.1-rc1
>>> btrfs-progs: tests: verify btrfstune output during uuid-rewrite
>>> btrfs-progs: btrfstune: print lowercase uuid during uuid rewrite
>>> btrfs-progs: btrfstune: print correct current uuid during rewrite
>>> btrfs-progs: configure: fix typo in summary
>>> btrfs-progs: INSTALL: fix typos
>>> Btrfs progs v4.1
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-09-09 10:35 ` Vytautas D
@ 2015-09-09 11:25 ` Duncan
2015-09-09 13:26 ` David Sterba
1 sibling, 0 replies; 24+ messages in thread
From: Duncan @ 2015-09-09 11:25 UTC (permalink / raw)
To: linux-btrfs
Vytautas D posted on Wed, 09 Sep 2015 11:35:50 +0100 as excerpted:
> sorry for side question - but does this mean that btrfs-convert issues
> others were having on kernel 4.0+ on this mailing list is now addressed
> ?
AFAIK they've not yet been entirely fixed, no. There's still active
(within the day) discussion on further patches. Based on that discussion
(I'm not a dev) the root problem is that ext* doesn't cleanly separate
data and metadata (tho ext4 does better than ext2/3) like btrfs does, and
both can end up in the same chunk on btrfs. That would require btrfs
mixed-bg mode (which is otherwise for small filesystems, the default at 1
GiB or smaller but often useful to say 32 GiB), but there were various
problems with that (possibly including chunks of the wrong size for mixed-
bg) and currently, convert apparently tries to do separate data and
metadata chunks. But because they're so mixed on ext*, that many small
chunks instead of the fewer large chunks btrfs would normally have. So
they're still debating mixed-bg or not.
So currently, the best recommendation is to simply create a brand new
btrfs, and copy to it from the existing ext*. That also has the benefit
of keeping the ext* as a backup of what is presumably the btrfs working
copy, and of course, as any good sysadmin has already integrated but
unfortunately we keep getting reports of people figuring out the hard
way, if it's not backed up, by definition of your lack of backup action,
you don't care about it, any post-loss claims to the contrary not
withstanding. So encouraging people to have at least the backup made as
a result of copying the old ext* to the new btrfs, instead of trying to
convert in place and finding out the hard way that lack of a backup means
the lost data wasn't of value, can be seen as a good thing. =:^)
Actually, even in the absence of known convert bugs I'd recommend a clean
mkfs, simply because that way you /know/ you start out with a clean
filesystem, and you end up with better btrfs defaults that way too. I
know that's the reason I steered clear of convert here. I simply don't
trust it, and wanted to start clean in any case.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-09-09 1:34 ` Qu Wenruo
2015-09-09 3:24 ` Qu Wenruo
@ 2015-09-09 13:19 ` David Sterba
1 sibling, 0 replies; 24+ messages in thread
From: David Sterba @ 2015-09-09 13:19 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, clm
On Wed, Sep 09, 2015 at 09:34:53AM +0800, Qu Wenruo wrote:
> > David Sterba (79):
> > btrfs-progs: tests: log the test name in results file
> > btrfs-progs: tests: support more formats of test images
> > btrfs-progs: tests: use readlink -f to resolve path
> > btrfs-progs: tests: sort image files
> > btrfs-progs: test: 015-check-bad-memory-access
> > btrfs-progs: tests: add script to clean intermediate images
> > btrfs-progs: receive: restore capabilities after chown
> > btrfs-progs: tests: add driver script for misc tests
> > btrfs-progs: tests: common: add helper run_check_stdout
> > btrfs-progs: tests: add misc test for fs features
> > btrfs-progs: update .gitignore
> > btrfs-progs: subvol show: print received uuid
> > btrfs-progs: subvol show: tweak output
> > btrfs-progs: btrfstune: rework option handling
> > btrfs-progs: btrfstune: add option to enable NO_HOLES
> > btrfs-progs: doc: update btrfstune manpage
> > btrfs-progs: tests: add misc tests for uuid rewrite
> > btrfs-progs: btrfstune: make uuid rewrite progress more verbose
> > btrfs-progs: ioctl.h: reorder to match kernel uapi header
> > btrfs-progs: ioctl.h: copy commented version of scrub_progress structure
> > btrfs-progs: ioctl.h: copy commented version of balance structures
> > btrfs-progs: ioctl.h: add EXTENT_SAME ioctl and structures
> > btrfs-progs: ioctl.h: add send flag mask
> > btrfs-progs: ioctl.h: update type annotations
> > btrfs-progs: ioctl.h: update btrfs_ioctl_fs_info_args
> > btrfs-progs: ioctl.h: add btrfs_ioctl_feature_flags struct
> > btrfs-progs: doc: update btrfs-rescue page and drop btrfs-zero-log
> > btrfs-progs: scrub: more friendly duration format in status
> > btrfs-progs: add function to stringify filesystem features
> > btrfs-progs: convert: allow to set filesystem features
>
> Tons of new patches, but at least I didn't find the patch in mail list
> for the above convert patch, when I want to comment on it.
Well, I'm doing lots of small and boring patches that I consider noisy
for the mailinglist, OTOH I introduced some bugs that would be caught by
reviews.
> So I can only comment here.
>
> [[Good fix even before we found the real bug]]
> This convert patch is quite good, as it in fact addressed the
> metadata-cross-map-boundary bug, even before we found it.
I was totally unaware of this consequence. So this narrows the bug to
* introduced: 3.19.1 (16k nodesize for convert)
* last not fixed: 4.0.1
So anything 4.1+ should be fine.
> [[True fix]]
> IMHO, I still prefer to use mixed block group for btrfs-convert, as that
> fits better for the use case.
While I don't mind adding mixed-bg support to convert options, I think
we first need to add support to mixed bg balance filters. The typical
filesystem size being converted from extN to btrfs is much larger than
the forced mixed-bg flag (1G nowadays).
> And what we really need is, to keep fixing btrfs-convert and kernel.
> Banning mixed bg for convert case is just avoiding the real cause, and I
> think that should be the last method.
The allowed convert features were added to the mask if they looked safe
to me at the time.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-09-09 3:24 ` Qu Wenruo
2015-09-09 10:35 ` Vytautas D
@ 2015-09-09 13:25 ` David Sterba
2015-09-09 14:01 ` Austin S Hemmelgarn
2015-09-10 0:50 ` Qu Wenruo
1 sibling, 2 replies; 24+ messages in thread
From: David Sterba @ 2015-09-09 13:25 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, clm
On Wed, Sep 09, 2015 at 11:24:17AM +0800, Qu Wenruo wrote:
> The problem is that, for converted image, it's quite possible that data
> and metadata extent are stored in one chunk even the chunk is not mixed.
>
> I'll add fsck support for it soon.
That would be great, I have no idea how the data/metadata can get mixed.
Does this involve inlied file data that come from ext4?
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-09-09 10:35 ` Vytautas D
2015-09-09 11:25 ` Duncan
@ 2015-09-09 13:26 ` David Sterba
1 sibling, 0 replies; 24+ messages in thread
From: David Sterba @ 2015-09-09 13:26 UTC (permalink / raw)
To: Vytautas D; +Cc: Qu Wenruo, Btrfs BTRFS, clm
On Wed, Sep 09, 2015 at 11:35:50AM +0100, Vytautas D wrote:
> sorry for side question - but does this mean that btrfs-convert issues
> others were having on kernel 4.0+ on this mailing list is now
> addressed ?
Per Qu's analysis yes, the broken versions are 3.19.1-4.0.1.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-09-09 13:25 ` David Sterba
@ 2015-09-09 14:01 ` Austin S Hemmelgarn
2015-09-10 0:50 ` Qu Wenruo
1 sibling, 0 replies; 24+ messages in thread
From: Austin S Hemmelgarn @ 2015-09-09 14:01 UTC (permalink / raw)
To: dsterba, Qu Wenruo, linux-btrfs, clm
[-- Attachment #1: Type: text/plain, Size: 799 bytes --]
On 2015-09-09 09:25, David Sterba wrote:
> On Wed, Sep 09, 2015 at 11:24:17AM +0800, Qu Wenruo wrote:
>> The problem is that, for converted image, it's quite possible that data
>> and metadata extent are stored in one chunk even the chunk is not mixed.
>>
>> I'll add fsck support for it soon.
>
> That would be great, I have no idea how the data/metadata can get mixed.
> Does this involve inlied file data that come from ext4?
Hmm, I'm actually kind of curious about this too. AFAIK, the inline
file data limit on ext* is significantly smaller than on btrfs. I could
possibly see a really badly fragmented ext* filesystem causing this to
happen because of the chunk size requirements in btrfs, but I would
expect such a case to fail in new and unexpected ways as a result.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Btrfs progs release 4.1
2015-09-09 13:25 ` David Sterba
2015-09-09 14:01 ` Austin S Hemmelgarn
@ 2015-09-10 0:50 ` Qu Wenruo
1 sibling, 0 replies; 24+ messages in thread
From: Qu Wenruo @ 2015-09-10 0:50 UTC (permalink / raw)
To: dsterba, linux-btrfs, clm
David Sterba wrote on 2015/09/09 15:25 +0200:
> On Wed, Sep 09, 2015 at 11:24:17AM +0800, Qu Wenruo wrote:
>> The problem is that, for converted image, it's quite possible that data
>> and metadata extent are stored in one chunk even the chunk is not mixed.
>>
>> I'll add fsck support for it soon.
>
> That would be great, I have no idea how the data/metadata can get mixed.
> Does this involve inlied file data that come from ext4?
>
Not completely sure yet, but in my case, I didn't put any data into the
ext4 fs, just newly created one and convert.
And no inline data extent in converted btrfs, checked by debug-tree.
So I think the problem is the chunk allocation algorithm for btrfs-convert.
The algorithm just doesn't check whether the allocated chunk will cover
ext* data or not, and determine chunk type by a strange data/metadata
threshold.
To fix it, we'd better rework the algorithm to add more flexibility to
handle scattered ext* data.
But at the bottom line, every ext* used space should be covered by DATA
chunk.
Like the following case:
0 1G 2G 3G
|//| |///| |//| |//|
Here For range 0~1G, the beginning is some ext* data, but doesn't take
too much space.
And in range 1G~2G, some ext* extents are scattered there.
Lastly 2G~3G is not used.
For 0~1G, we only need to ensure the beginning part(including the ext*
range) be covered by a data chunk, the resting part can be either
allocated for metadata or data of newer btrfs chunk.
For 1G~2G, better to allocate the whole 1G into data chunk, as too many
scattered ext* space.
For 2G~3G, either DATA or METADATA is OK for further allocation.
But for now, as a hot fix, I prefer the quick but dirty fix that I
already submitted, just to force btrfs-convert to use mixed-bg patchset
(with fsck support).
The roadmap for convert fix will follow then:
1) Add kernel balance support to convert mixed-bg.
Seems easier than 2) for me.
2) Add btrfs-convert support to use correct non mixed-bg mode.
But I'm still not sure about whether the bug mail list reported is
related to the bug we exposed.
Thanks,
Qu
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2015-09-10 0:50 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-22 15:00 Btrfs progs release 4.1 David Sterba
2015-06-22 16:18 ` Goffredo Baroncelli
2015-06-22 18:01 ` David Sterba
2015-06-22 18:18 ` qgroup limit clearing, was " Christian Robottom Reis
2015-06-22 18:18 ` Christian Robottom Reis
2015-06-22 23:55 ` Tsutomu Itoh
2015-06-23 1:55 ` Qu Wenruo
2015-06-23 12:42 ` David Sterba
2015-06-22 20:45 ` Martin Steigerwald
2015-06-23 12:53 ` David Sterba
2015-06-23 9:03 ` Sjoerd
2015-06-23 9:18 ` Qu Wenruo
2015-06-23 14:45 ` David Sterba
2015-06-24 20:26 ` Sjoerd
2015-06-25 13:03 ` David Sterba
2015-09-09 1:34 ` Qu Wenruo
2015-09-09 3:24 ` Qu Wenruo
2015-09-09 10:35 ` Vytautas D
2015-09-09 11:25 ` Duncan
2015-09-09 13:26 ` David Sterba
2015-09-09 13:25 ` David Sterba
2015-09-09 14:01 ` Austin S Hemmelgarn
2015-09-10 0:50 ` Qu Wenruo
2015-09-09 13:19 ` David Sterba
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.