linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Omar Sandoval <osandov@osandov.com>
To: dsterba@suse.cz, linux-btrfs@vger.kernel.org,
	linux-fsdevel@vger.kernel.org
Subject: Re: [PATCH 0/9] btrfs: implement send/receive of compressed extents without decompressing
Date: Mon, 24 Aug 2020 15:16:39 -0700	[thread overview]
Message-ID: <20200824221639.GF197795@exodia.localdomain> (raw)
In-Reply-To: <20200824195754.GQ2026@twin.jikos.cz>

On Mon, Aug 24, 2020 at 09:57:55PM +0200, David Sterba wrote:
> On Fri, Aug 21, 2020 at 12:39:50AM -0700, Omar Sandoval wrote:
> > Protocol Updates
> > ================
> > 
> > This series makes some changes to the send stream protocol beyond adding
> > the encoded write command/attributes and bumping the version. Namely, v1
> > has a 64k limit on the size of a write due to the 16-bit attribute
> > length. This is not enough for encoded writes, as compressed extents may
> > be up to 128k and cannot be split up. To address this, the
> > BTRFS_SEND_A_DATA is treated specially in v2: its length is implicitly
> > the remaining length of the command (which has a 32-bit length). This
> > was the last bad of the options I considered.
> > 
> > There are other commands that we've been wanting to add to the protocol:
> > fallocate and FS_IOC_SETFLAGS. This series reserves their command and
> > attribute numbers but does not implement kernel support for emitting
> > them. However, it does implement support in receive for them, so the
> > kernel can start emitting those whenever we get around to implementing
> > them.
> 
> Can you please outline the protocol changes (as a bullet list) and
> eventually cross-ref with items
> https://btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive#Send_stream_v2_draft
> 
> I'd like to know which and why you did not implement. The decision here
> is between get v2 out with most desired options and rev v3 later with
> the rest, or do v2 as complete as possible.

The short version is that I didn't implement the kernel side of any of
those :) the RWF_ENCODED series + this series is already big, and I
didn't want to make it even bigger. I figured updating the
protocol/receive now and doing the kernel side later was a good
compromise (rather than doing a huge code dump or constantly bumping the
protocol version). Is there some reason you don't like this approach?
I'm of course happy to go about this in whatever way you think is best.

Here's a breakdown of the list from the wiki:

* Send extent holes, send preallocated extents: both require fallocate.
  Boris implemented the receive side. I have some old patches
  implementing the send side [1], but they're a largish rework of extent
  tracking in send.
* Extent clones within one file: as far as I can tell, this is already
  possible with v1, it just sends redundant file paths.
* Send otime for inodes: the consensus when I posted patches to enable
  this [2] was that we don't want this after all.
* Send file flags (FS_IOC_GETFLAGS/FS_IOC_SETFLAGS): again, Boris
  implemented the receive side. I previously took a stab at the send
  side, but it's really annoying because of all of the interactions
  between directory inheritance, writes vs. NOCOW/append-only/immutable,
  etc. It's do-able, it would just take a lot of care.
* Optionally send owner/group as strings: this one I wasn't aware of.
* "block device is not sent over the stream": I don't know what this is
  referring to. It looks like we send block device nodes with mknod.

In my opinion, fallocate support is the most important, SETFLAGS would
be good but is a lot of effort, and the rest are nice-to-have.

Let me know how you'd like me to go about this.

1: https://github.com/osandov/linux/commits/btrfs-send-v2
2: https://lore.kernel.org/linux-btrfs/cover.1550136164.git.osandov@fb.com/

  reply	other threads:[~2020-08-24 22:16 UTC|newest]

Thread overview: 37+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-21  7:39 [PATCH 0/9] btrfs: implement send/receive of compressed extents without decompressing Omar Sandoval
2020-08-21  7:39 ` [PATCH 1/9] btrfs: send: get rid of i_size logic in send_write() Omar Sandoval
2020-08-21 17:26   ` Filipe Manana
2020-08-24 17:39   ` Josef Bacik
2020-08-21  7:39 ` [PATCH 2/9] btrfs: send: avoid copying file data Omar Sandoval
2020-08-21 17:29   ` Filipe Manana
2020-08-24 21:34     ` Omar Sandoval
2020-08-24 17:47   ` Josef Bacik
2020-09-11 14:13   ` David Sterba
2020-09-14 22:04     ` Omar Sandoval
2020-09-15  8:14       ` David Sterba
2020-08-21  7:39 ` [PATCH 3/9] btrfs: send: use btrfs_file_extent_end() in send_write_or_clone() Omar Sandoval
2020-08-21 17:30   ` Filipe Manana
2020-08-21  7:39 ` [PATCH 4/9] btrfs: add send_stream_version attribute to sysfs Omar Sandoval
2020-08-21  7:39 ` [PATCH 5/9] btrfs: add send stream v2 definitions Omar Sandoval
2020-08-24 17:49   ` Josef Bacik
2020-08-21  7:39 ` [PATCH 6/9] btrfs: send: write larger chunks when using stream v2 Omar Sandoval
2020-08-24 17:57   ` Josef Bacik
2020-08-21  7:39 ` [PATCH 7/9] btrfs: send: allocate send buffer with alloc_page() and vmap() for v2 Omar Sandoval
2020-08-21  7:39 ` [PATCH 8/9] btrfs: send: send compressed extents with encoded writes Omar Sandoval
2020-08-24 17:32   ` Josef Bacik
2020-08-24 17:52     ` Omar Sandoval
2020-08-21  7:39 ` [PATCH 9/9] btrfs: send: enable support for stream v2 and compressed writes Omar Sandoval
2020-08-21  7:40 ` [PATCH 01/11] btrfs-progs: receive: support v2 send stream larger tlv_len Omar Sandoval
2020-08-21  7:40 ` [PATCH 02/11] btrfs-progs: receive: dynamically allocate sctx->read_buf Omar Sandoval
2020-08-21  7:40 ` [PATCH 03/11] btrfs-progs: receive: support v2 send stream DATA tlv format Omar Sandoval
2020-08-21  7:40 ` [PATCH 04/11] btrfs-progs: receive: add send stream v2 cmds and attrs to send.h Omar Sandoval
2020-08-21  7:40 ` [PATCH 05/11] btrfs-progs: receive: add stub implementation for pwritev2 Omar Sandoval
2020-08-21  7:40 ` [PATCH 06/11] btrfs-progs: receive: process encoded_write commands Omar Sandoval
2020-08-21  7:40 ` [PATCH 07/11] btrfs-progs: receive: encoded_write fallback to explicit decode and write Omar Sandoval
2020-08-21  7:40 ` [PATCH 08/11] btrfs-progs: receive: process fallocate commands Omar Sandoval
2020-08-21  7:40 ` [PATCH 09/11] btrfs-progs: receive: process setflags ioctl commands Omar Sandoval
2020-08-21  7:40 ` [PATCH 10/11] btrfs-progs: send: stream v2 ioctl flags Omar Sandoval
2020-08-21  7:40 ` [PATCH 11/11] btrfs-progs: receive: add tests for basic encoded_write send/receive Omar Sandoval
2020-08-24 19:57 ` [PATCH 0/9] btrfs: implement send/receive of compressed extents without decompressing David Sterba
2020-08-24 22:16   ` Omar Sandoval [this message]
2020-09-10 11:28 ` David Sterba

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200824221639.GF197795@exodia.localdomain \
    --to=osandov@osandov.com \
    --cc=dsterba@suse.cz \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).