[LSF/MM/BPF TOPIC] eBFP for block devices

[LSF/MM/BPF TOPIC] eBFP for block devices

[Hannes Reinecke]

Hi Omar,

and another topic which came up during discussion yesterday:

eBPF for block devices
It would be useful to enable eBPF for block devices, such that we could 
do things like filtering bios on bio type, do error injection by 
modifying the bio result etc.
This topic should be around how it could be implemented and what 
additional use-cases could be supported.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Gabriel Krisman Bertazi]

Hannes Reinecke <hare@suse.de> writes:

> Hi Omar,
>
> and another topic which came up during discussion yesterday:
>
> eBPF for block devices
> It would be useful to enable eBPF for block devices, such that we could
> do things like filtering bios on bio type, do error injection by 
> modifying the bio result etc.
> This topic should be around how it could be implemented and what
> additional use-cases could be supported.

Cc'ing Khazhy since Google might have some applications for this to
filter IOs based on the blocks being accessed, in the context of
safeguarding specific regions from accidental overwrites / application
error.

-- 
Gabriel Krisman Bertazi

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Martin K. Petersen]

Gabriel,

> Cc'ing Khazhy since Google might have some applications for this to
> filter IOs based on the blocks being accessed, in the context of
> safeguarding specific regions from accidental overwrites / application
> error.

We've been working in this area too. It possible to write BPF filters to
protect block ranges using should_fail_bio().

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Khazhy Kumykov]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

On Mon, May 2, 2022 at 11:53 AM Martin K. Petersen
<martin.petersen@oracle.com> wrote:
>
>
> Gabriel,
>
> > Cc'ing Khazhy since Google might have some applications for this to
> > filter IOs based on the blocks being accessed, in the context of
> > safeguarding specific regions from accidental overwrites / application
> > error.
Yeah, we have a few potential use cases for something like this - one
is interested in filtering based on issuing user. A separate use case
would be filtering based on regions - for this I do wonder how
appropriate something like eBPF would be, especially if we wanted the
filters to have some guarantees of existing so long as the data we're
protecting exists.
>
> We've been working in this area too. It possible to write BPF filters to
> protect block ranges using should_fail_bio().

It'd be nice to have a "proper" api vs. ALLOW_ERROR_INJECTION, which
feels more debug-y and has the drawback of
CONFIG_FUNCTION_ERROR_INJECTION being all-or-nothing

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3999 bytes --]

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Martin K. Petersen]

Khazhy,

>> We've been working in this area too. It possible to write BPF filters to
>> protect block ranges using should_fail_bio().
>
> It'd be nice to have a "proper" api vs. ALLOW_ERROR_INJECTION, which
> feels more debug-y and has the drawback of
> CONFIG_FUNCTION_ERROR_INJECTION being all-or-nothing

I agree that it would be nice for this to be detached from error
injection.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Hannes Reinecke]

On 5/2/22 13:34, Martin K. Petersen wrote:
> 
> Khazhy,
> 
>>> We've been working in this area too. It possible to write BPF filters to
>>> protect block ranges using should_fail_bio().
>>
>> It'd be nice to have a "proper" api vs. ALLOW_ERROR_INJECTION, which
>> feels more debug-y and has the drawback of
>> CONFIG_FUNCTION_ERROR_INJECTION being all-or-nothing
> 
> I agree that it would be nice for this to be detached from error
> injection.
> 
Actually, I'd rather turn this around such that error injection is 
running _on top_ of eBPF...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Martin K. Petersen]

Hannes,

> Actually, I'd rather turn this around such that error injection is
> running _on top_ of eBPF...

That would be great. Relying on eBPF does add quite a bit of complexity
in the dependencies department compared to 'echo foo > /sys/debug/bar',
though...

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Omar Sandoval]

On Mon, May 02, 2022 at 09:21:31AM -0700, Hannes Reinecke wrote:
> Hi Omar,
> 
> and another topic which came up during discussion yesterday:
> 
> eBPF for block devices
> It would be useful to enable eBPF for block devices, such that we could do
> things like filtering bios on bio type, do error injection by modifying the
> bio result etc.
> This topic should be around how it could be implemented and what additional
> use-cases could be supported.
> 
> Cheers,
> 
> Hannes

Do you want to try to coordinate a joint session with BPF, or were you
planning on brainstorming what we need just in the IO track and tracking
down the BPF folks offline?

+Alexei and Daniel

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Hannes Reinecke]

On 5/2/22 11:17, Omar Sandoval wrote:
> On Mon, May 02, 2022 at 09:21:31AM -0700, Hannes Reinecke wrote:
>> Hi Omar,
>>
>> and another topic which came up during discussion yesterday:
>>
>> eBPF for block devices
>> It would be useful to enable eBPF for block devices, such that we could do
>> things like filtering bios on bio type, do error injection by modifying the
>> bio result etc.
>> This topic should be around how it could be implemented and what additional
>> use-cases could be supported.
>>
>> Cheers,
>>
>> Hannes
> 
> Do you want to try to coordinate a joint session with BPF, or were you
> planning on brainstorming what we need just in the IO track and tracking
> down the BPF folks offline?
> 
> +Alexei and Daniel

Coordinated session, please.
We need to get some common understandig of those things we're trying to 
do are properly eBPF-ish, and alse make the eBPF people aware of the 
issue we're facing.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [LSF/MM/BPF TOPIC] eBFP for block devices

[Luis Chamberlain]

On Mon, May 02, 2022 at 12:24:04PM -0700, Hannes Reinecke wrote:
> On 5/2/22 11:17, Omar Sandoval wrote:
> > On Mon, May 02, 2022 at 09:21:31AM -0700, Hannes Reinecke wrote:
> > > Hi Omar,
> > > 
> > > and another topic which came up during discussion yesterday:
> > > 
> > > eBPF for block devices
> > > It would be useful to enable eBPF for block devices, such that we could do
> > > things like filtering bios on bio type, do error injection by modifying the
> > > bio result etc.
> > > This topic should be around how it could be implemented and what additional
> > > use-cases could be supported.
> > > 
> > > Cheers,
> > > 
> > > Hannes
> > 
> > Do you want to try to coordinate a joint session with BPF, or were you
> > planning on brainstorming what we need just in the IO track and tracking
> > down the BPF folks offline?
> > 
> > +Alexei and Daniel
> 
> Coordinated session, please.
> We need to get some common understandig of those things we're trying to do
> are properly eBPF-ish, and alse make the eBPF people aware of the issue
> we're facing.

We last spoke about error injection prospects with eBPF at last year's LSFMM,
and I had explained the lay of the land of pros cons, and the last incarnation
I had tried was the one with the least amount of code affected upstream, but
that was still not a viable accepted approach as we still had to sprinkle a
few branches here and there. And so curious if things have changed.

An alternative that comes to mind recently is to do use something like
ldflags-y += --wrap=upstream_call as the CXL folks do with tools/testing/cxl/Kbuild
however I'm in no way shape or form a fan of what how they did that without proper
kconfig semantics -- that has already proven to easily crash. One *can* do this
with proper Kconfig semantics.

The only downfall to this would be that we'd have live with a copy of
the routine we want elsewhere say block/alt/foo.c to debug with the deltas we
want spinkled onto it. So we'd have to then try to keep both in sync and
that seems like another pain in the ass.

Either way, consider me interested in beating the horse on error injection.

  Luis