Re: Adding compression support for bluestore.

[Igor Fedotov <ifedotov@mirantis.com>]

On 17.03.2016 18:28, Allen Samuels wrote:
>> flow for EC and replicated pools. The scheme that you propose means that
>> EC chunking boundaries become fluid and data-sensitive -- destroying the
>> "seek" capability (i.e., you no longer know which node has any given logical
>> address within the object). Essentially you'll need an entirely different
>> backend flow for EC pools (at this level) with a complicated metadata
>> mapping scheme. That seems MUCH more complicated and run-time
>> expensive to me.
>> Wouldn't agree with this statement.  Perhaps I improperly presented my
>> ideas or missing something...
>> IMHO current EC pool's write pattern is just a regular append only mode.
> This is where we diverge. Sam and I worked out a blueprint for doing non append-only writes into EC pools (i.e., partial and/or complete overwrites).
>
> See https://github.com/athanatos/ceph/blob/wip-ec-overwrites/doc/dev/osd_internals/ec_overwrites.rst
>
> This allows all of the current restrictions in the usages of EC Pools to be eliminated, enabling all protocols to directly utilize EC pools.
>
> If you do compression BEFORE you do EC, then you have a real problem with landing your data across the different nodes of an EC stripe in the non-append case.
>
> BTW, it's BlueStore itself that enables this new capability to be implemented efficiently, it's very expensive to do this with FileStore (an additional full copy of the data is required, i.e., 3x write-amp on FileStore)
>   
Got it.  That's where we diverge:
Actually speaking of compression layer segregation in this thread I 
meant having it AFTER EC/Replicated pools and BEFORE the bluestore. Not 
BEFORE pools ... IMO in my case such a layer is absolutely similar for 
both replicated and EC pools as EC write patterns are always (for both 
append only and overwrite modes) a subset of replicated pool one.

Anyway thanks a lot for your clarifications. I highly appreciate your 
help...

>
>> And read pattern is partially random - EC reads data in arbitrary order at
>> specific offsets only. As long as some layer is able to handle such patterns it's
>> probably OK for EC pool. And I don't see any reasons why compression layer
>> is unable to do that and what's the difference comparing to replicated pools.
>> Actually my idea  about segregation was mainly about reusing existing
>> bluestore rather than modifying it. Compression engine should somehow
>> (e.g. by inheriting from bluestore and overriding _do_write/_do_read
>> methods ) intercept write/read requests and maintain its OWN block
>> management independent from bluestore one. Bluestore is left untouched
>> and exposes its functionality ( via Read/Write handlers) AS-IS to the
>> compression layer instead of pools. The key thing is that compressed blocks
>> map and bluestore extents map use the same logical offset, i.e.
>> if some compressed block starts at offset X it's written to bluestore at offset
>> X too. But written block is shorter than original one and thus store space is
>> saved.
>> I would agree with the comment that this probably complicates metadata
>> handling - compression layer metadata has to be handled similar to bluestore
>> ones ( proper sync, WAL, transaction, etc). But I don't see any issues specific
>> to EC here...
>> Have I missed something?
> No doubt the metadata gets more complicated in the presence of compression. Especially if we want to enable the sort of lazy partial-overwrite with background cleanup that seems to be the most desirable.
Agree.

Thanks,
Igor