linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Hans van Kranenburg <Hans.van.Kranenburg@mendix.com>
To: Tobias Reinhard <trtracer@gmail.com>,
	Qu Wenruo <quwenruo.btrfs@gmx.com>,
	"linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>
Subject: Re: Retrieving CSUM-Tree
Date: Fri, 25 Jan 2019 20:49:24 +0000	[thread overview]
Message-ID: <29e7b71e-ba07-2b25-3474-1352ee38677b@mendix.com> (raw)
In-Reply-To: <e56eccf6-7c6e-3601-5068-95d700c60351@gmail.com>

On 1/25/19 9:45 PM, Tobias Reinhard wrote:
> Am 25.01.2019 um 19:05 schrieb Hans van Kranenburg:
>> On 1/25/19 5:59 PM, Tobias Reinhard wrote:
>>> Am 13.01.2019 um 12:02 schrieb Qu Wenruo:
>>>> On 2019/1/13 下午6:19, Tobias Reinhard wrote:
>>>>> Hi,
>>>>>
>>>>> I want to read the complete CSUM-Tree from userspace. I tried it
>>>>> via the
>>>>> ioctl. This is what the code looks like:
>>>>>
>>>>> struct btrfs_sv2_args sv2_args;
>>>>> int fd = open(filename, O_RDONLY);
>>>>> sv2_args.key.tree_id = BTRFS_CSUM_TREE_OBJECTID;
>>>>> sv2_args.key.min_objectid = 0;
>>>>> sv2_args.key.max_objectid = -1;
>>>>> sv2_args.key.min_offset = 0;
>>>>> sv2_args.key.max_offset = -1;
>>>>> sv2_args.key.min_transid = 0;
>>>>> sv2_args.key.max_transid = -1;
>>>>> sv2_args.key.min_type = BTRFS_CSUM_ITEM_KEY;
>>>>> sv2_args.key.max_type = BTRFS_CSUM_ITEM_KEY;
>>>>> sv2_args.key.nr_items = -1;
>>>>> sv2_args.buf_size = sizeof(sv2_args.buf);
>>>>> ioctl(fd, BTRFS_IOC_TREE_SEARCH_V2, &sv2_args);
>>>>>
>>>>> But the device is not small and I hit the limit of the
>>>>> btrfs_sv2_args.buf which seems to be 16 MB.
>>>>>
>>>>> How can I get the *complete* CSUM-Tree?
>>>>>
>>>>> Limiting to offset does not work (My first idea was to do it this way
>>>>> and get it in chunks).
>>>> That's strange.
>>>>
>>>> Are you still using 0~-1 objectid and 0~-1 type, just last_offset~-1?
>>>>
>>>> Have tried searching using the following parameters?
>>>> min_objectid = max_objectid = BTRFS_EXTENT_CSUM_OBJECTID
>>>> min_type = max_type = BTRFS_CSUM_ITEM_KEY;
>>>> min_offset = last_found_csum_offset
>>>> max_offset = -1
>>> Sorry for my late response.
>>>
>>> If I set
>>>
>>> min_objectid = max_objectid = BTRFS_EXTENT_CSUM_OBJECTID
>>>
>>> I don't get anything. I have to set it to max=-1 (min doesn't matter).
>>>
>>> And in that I case, min_offset and max_offset doesn't matter - I always
>>> get the same result. I can even use "wrong" filters like min=1000
>>> max=500.
>> First, it's important to understand how all these min/max values play
>> together:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/btrfs.h#n441
>>
>>
>> So, you define a single start key, and a single end key, and then you
>> get everything that's in between (including the end value).
>>
>> So, this...
>>
>> sv2_args.key.min_objectid = 0;
>> sv2_args.key.max_objectid = -1;
>> sv2_args.key.min_offset = 0;
>> sv2_args.key.max_offset = -1;
>> sv2_args.key.min_transid = 0;
>> sv2_args.key.max_transid = -1;
>> sv2_args.key.min_type = BTRFS_CSUM_ITEM_KEY;
>> sv2_args.key.max_type = BTRFS_CSUM_ITEM_KEY;
>>
>> ...translates to:
>>
>> min key: (0, CSUM_ITEM_KEY, 0)
>> max key: (18446744073709551615, CSUM_ITEM_KEY, 18446744073709551615)
>>
>> Since the keys end up being just a single 136 bit number, it makes no
>> sense to do anything with the middle field, if the first field, objectid
>> is not the same in both start and end key. The search space is linear,
>> not 3 dimensional. The invidual min/max values for objectid, type and
>> offset cannot be used to filter the result, they only define the
>> endpoints of an interval.
>>
>> Since all csum items have the same objectid number anyway, the second
>> suggestion is fine, and gives you this start and end:
>>
>> min key: (EXTENT_CSUM_OBJECTID, CSUM_ITEM_KEY, 0)
>> max key: (EXTENT_CSUM_OBJECTID, CSUM_ITEM_KEY, 18446744073709551615)
>>
>> Works for me (here in python, but using same ioctl):
>>
>> -$ cat show_csum_keys.py
>> #!/usr/bin/python3
>>
>> import btrfs
>> from btrfs.ctree import Key, CSUM_TREE_OBJECTID, \
>>          EXTENT_CSUM_OBJECTID, EXTENT_CSUM_KEY
>>
>> with btrfs.FileSystem('/mnt/tutorial') as fs:
>>      min_key = Key(EXTENT_CSUM_OBJECTID, EXTENT_CSUM_KEY, 0)
>>      max_key = Key(EXTENT_CSUM_OBJECTID, EXTENT_CSUM_KEY + 1, 0) - 1
>>      print("Searching from {} to {}".format(min_key, max_key))
>>      for header, data in btrfs.ioctl.search_v2(fs.fd, CSUM_TREE_OBJECTID,
>>                                                min_key, max_key):
>>          print(Key(header.objectid, header.type, header.offset))
>>
>> -# ./show_csum_keys.py
>> Searching from (EXTENT_CSUM EXTENT_CSUM 0) to (EXTENT_CSUM EXTENT_CSUM
>> -1)
>> (EXTENT_CSUM EXTENT_CSUM 5700059136)
>> (EXTENT_CSUM EXTENT_CSUM 5700321280)
>> (EXTENT_CSUM EXTENT_CSUM 5700583424)
>> (EXTENT_CSUM EXTENT_CSUM 5700845568)
>> (EXTENT_CSUM EXTENT_CSUM 5701107712)
>> (EXTENT_CSUM EXTENT_CSUM 5704646656)
>> (EXTENT_CSUM EXTENT_CSUM 5705039872)
>> (EXTENT_CSUM EXTENT_CSUM 5706350592)
>> [...]
>>
> Reading your example, I noticed my mistake.
> 
> I took  BTRFS_CSUM_ITEM_KEY  instead of BTRFS_EXTENT_CSUM_KEY for the
> type, doh.
> 
> Now, it seems to works.

Hah! Well done. I also didn't spot that. :)

Probably too much CSUM dancing before our eyes... Wait until you start
unpacking the search results, will only get worse...

https://reactiongifs.me/wp-content/uploads/2014/06/reading-ikea-intructions-big-lebowski-confused.gif

-- 
Hans van Kranenburg

  reply	other threads:[~2019-01-25 20:49 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-13 10:19 Retrieving Tobias Reinhard
2019-01-13 11:02 ` Retrieving Qu Wenruo
2019-01-25 16:59   ` Retrieving CSUM-Tree Tobias Reinhard
2019-01-25 18:05     ` Hans van Kranenburg
2019-01-25 20:45       ` Tobias Reinhard
2019-01-25 20:49         ` Hans van Kranenburg [this message]
2019-01-25 21:22 ` Retrieving Zygo Blaxell

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=29e7b71e-ba07-2b25-3474-1352ee38677b@mendix.com \
    --to=hans.van.kranenburg@mendix.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=quwenruo.btrfs@gmx.com \
    --cc=trtracer@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).