python-btrfs v10

python-btrfs v10

[Hans van Kranenburg]

I just tagged v10 of the python btrfs library.

https://github.com/knorrie/python-btrfs/

Somewhere in the next days I'll update pypi and will prepare debian
packages for unstable and stretch-backports.

Note: I will be converting the branches in git for debian packaging to
the DEP14 standard, also see
https://github.com/knorrie/python-btrfs/issues/14

---- >8 ----

Here's the summary of changes (from CHANGES):

python-btrfs v10, Jan 18, 2019
  * Switch to LGPL-3.0 license.
  * Add reference documentation to all code, using sphinx autodoc. An
    online version of the result in html format is available at:
    https://python-btrfs.readthedocs.io/en/latest/btrfs.html
  * All parts of the code that are covered by reference documentation
    are now considered to be the public API of the library. Effort will
    be done to not break this API in the future.
  * The FileSystem class now provides a context manager, to help prevent
    leaking the open file descriptor used internally.
  * The mounted_filesystems function in btrfs.utils module has been
    replaced by mounted_filesystem_paths. It is the responsibility of
    the caller to create the FileSystem objects for these paths and use
    the new context manager while doing do.
  * Add the fs_usage module, containing the FsUsage class, providing
    detailed usage reporting for a filesystem. This code replaces old
    similar functions, which were incomplete and buggy.
  * Adopt munin and nagios plugins to use the new FsUsage object.
  * IOCTLs: IOC_SYNC, FIDEDUPERANGE, IOC_GET_FEATURES
  * Data structures: FreeSpaceInfo, FreeSpaceExtent, FreeSpaceBitmap.
  * Introduce ItemNotFoundError, currently only raised by the block
    group lookup function in FileSystem. It inherits from IndexError for
    backwards compatibility.
  * Many small changes to improve the object pretty printer.
  * Fixes:
    - By default the maximum amount of items generated by a single call
      to search or search_v2 would be limited to 2^32-1. Fix this, so
      it's unlimited. The search functions return an iterator and call
      the search ioctl of the linux kernel multiple times internally.
  * Examples added:
    - A new show_usage, which shows information from the new usage
      reporting.
    - space_calculator: Offline usable space calculator, based on disk
      sizes and profiles provided on the command line.
    - file_dedupe_range: Example about using the deduplication ioctl.

---- >8 ----

My favourite one for this release is the new detailed usage reporting.

By using an algorithm that is based on the behaviour of the chunk
allocator in kernel code it simulates what would happen if the rest of
all available unallocated raw disk space for the filesystem would be
used up by newly written data. While doing so, it takes the current
usage pattern of the filesystem into account (data vs metadata ratio).

https://github.com/knorrie/python-btrfs/commit/41f7f3ca8f32566cf2182949c3724295e3776fc8

After running the simulation of the chunk allocator algorithm, there
might still be unallocated raw disk space left when no further chunk
allocations using the current target profiles for data and metadata are
possible. These amounts of "unallocatable" raw disk space are also
available in the report.

Moreover, when re-running the same simulation, using the current usage
pattern, and starting with all available disk space as unallocated, we
can figure out the amounts of raw disk space that would be unallocatable
when converting all data to the target profiles, while taking into
account that there might be different sized attached block devices
spoiling the fun.

By combining both of those results, it's possible to report the amount
of unallocatable raw disk space that could be "reclaimed" for use when
doing btrfs balance to rewrite data.

And if your head is not yet spinning right now, by combining all of
that, it's quite easy to predict if a btrfs balance convert will have
any chance of completion. Just watch output of the new show_usage.py
example, or use your own cobbled up script that does things with
attributes of your fs.usage() to follow progress.

The code works for all currently implemented block group profiles in btrfs.

But it's python, so isn't this very slow? I have a 72TiB btrfs
filesystem here with about 69k block groups, and crawling through the
whole chunk tree while retrieving it from the kernel, as well as doing
both the chunk allocator simulations for remaining unallocated space and
for a fully empty filesystem takes two and a halve seconds in total. I
think that isn't bad.

---- >8 ----

So, it has been more than a year... What happened? Well, at first I
started writing tutorial style documentation. It can be seen in still
highly WIP state at
https://github.com/knorrie/python-btrfs/blob/tutorial/tutorial/README.md

While doing so, I ran into a bug, and another one... And a lot of things
that just didn't work for RAID0/5/6 yet.

Then I started fixing that by writing the new usage reporting code. When
this got into shape, I realized that noone else than myself would ever
be able to use all of that without proper reference documentation about
all objects and attributes.

So, I started writing reference documentation for it, and then for the
rest of the code. While doing so, I went over all of the code in the
library again, and fixed a huge amount of small issues.

And, I'm happy with the end result. :) I think the reference docs will
help users a lot to find out what can be done using the library, and
have a fun time exploring.

---- >8 ----

Future work (or RSN and SIYH):

1) There has been a repeated question to promote some of the things that
are in examples/ to programs that will be put in /usr/bin. There are
quite a bunch of things that are very useful for end users. I will start
a separate discussion about this RSN on this mailing list. It will
require improving command line argument handling, writing man pages and
I don't know if the btrfs team would like me to start polluting the
btrfs-* tab space again after they've tried to get rid of all of those
btrfs-* commands.

This is the very first thing I want to work on now.

2) Tutorial style documentation.

I will move the current work on tutorial style documentation as
referenced above into the sphinx stuff, so I can cross reference
everything. No ETA, since this is just my hobby project. ;]

Have fun, and don't hesitate to talk to me on IRC (Knorrie) or email me
if you run into problems using all of this. o/

--
Hans van Kranenburg