Fwd: Adding crates

Fwd: Adding crates

[Chris Suter]

Hey folks,

I'm playing with a kernel module written in Rust, and for the thing
I'm working on, I need to pull in a number of crates. The most
significant of those might be some async ones: (e.g. futures,
async_trait). What's the best way to pull those in and get them
building with my module? In particular, I'm interested in how to
structure the Makefile to make it work. Are there any examples I could
follow?

Cheers,

Chris

Re: Fwd: Adding crates

[Geert Stappers]

On Mon, Mar 28, 2022 at 11:41:43AM +1100, Chris Suter wrote:
> Hey folks,
> 
> I'm playing with a kernel module written in Rust, and for the thing
> I'm working on, I need to pull in a number of crates. The most
> significant of those might be some async ones: (e.g. futures,
> async_trait). What's the best way to pull those in and get them
> building with my module? In particular, I'm interested in how to
> structure the Makefile to make it work. Are there any examples I could
> follow?

See https://lore.kernel.org/rust-for-linux/CANiq72my9N51hhryv-Cp79eUmPVCx6q=0fJrhkzmvT+nVPq+WQ@mail.gmail.com/T/#t
for a recent thread "Using 3rd party crates in an out-of-tree module"
 

> Cheers,
> Chris

 
Groeten
Geert Stappers
-- 
Silence is hard to parse

Re: Fwd: Adding crates

[Chris Suter]

I skimmed that thread, but unless I'm missing something, it seems to
be pretty much "don't do it", which doesn't work for me right now. I'm
just looking to get something working and from there I can figure out
the best way to integrate it — I'm just looking for the easiest (and
not necessarily the best/long-term) way to bring in dependent crates.

— Chris

On Mon, Mar 28, 2022 at 4:41 PM Geert Stappers <stappers@stappers.nl> wrote:
>
> On Mon, Mar 28, 2022 at 11:41:43AM +1100, Chris Suter wrote:
> > Hey folks,
> >
> > I'm playing with a kernel module written in Rust, and for the thing
> > I'm working on, I need to pull in a number of crates. The most
> > significant of those might be some async ones: (e.g. futures,
> > async_trait). What's the best way to pull those in and get them
> > building with my module? In particular, I'm interested in how to
> > structure the Makefile to make it work. Are there any examples I could
> > follow?
>
> See https://lore.kernel.org/rust-for-linux/CANiq72my9N51hhryv-Cp79eUmPVCx6q=0fJrhkzmvT+nVPq+WQ@mail.gmail.com/T/#t
> for a recent thread "Using 3rd party crates in an out-of-tree module"
>
>
> > Cheers,
> > Chris
>
>
> Groeten
> Geert Stappers
> --
> Silence is hard to parse

Re: Fwd: Adding crates

[Greg KH]

On Tue, Mar 29, 2022 at 08:38:29AM +1100, Chris Suter wrote:
> I skimmed that thread, but unless I'm missing something, it seems to
> be pretty much "don't do it", which doesn't work for me right now.

Why not?  What is the deadline?

And what specific crates do you need and why?  There is no "generic"
solution here other than "do not do this."

thanks,

greg k-h

Re: Fwd: Adding crates

[Chris Suter]

On Tue, Mar 29, 2022 at 4:23 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Tue, Mar 29, 2022 at 08:38:29AM +1100, Chris Suter wrote:
> > I skimmed that thread, but unless I'm missing something, it seems to
> > be pretty much "don't do it", which doesn't work for me right now.
>
> Why not?  What is the deadline?
>
> And what specific crates do you need and why?  There is no "generic"
> solution here other than "do not do this."

I'm interested in quite a few crates but the first and perhaps most
significant would be futures. I'm not necessarily looking for a
"generic" solution at this point, any solution would do — just enough
to help me make a little progress and then maybe I could contribute
back later. I can probably figure it out myself, but I was hoping to
save some time.

— Chris

>
> thanks,
>
> greg k-h

Re: Fwd: Adding crates

[Miguel Ojeda]

On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote:
>
> I'm interested in quite a few crates but the first and perhaps most
> significant would be futures. I'm not necessarily looking for a
> "generic" solution at this point, any solution would do — just enough
> to help me make a little progress and then maybe I could contribute
> back later. I can probably figure it out myself, but I was hoping to
> save some time.

If you are just looking to experiment with some third-party crates,
then as mentioned in the other thread, what you could do is copy and
adapt the code. For instance, if the crates you need do not use Cargo
build scripts, this may be as simple as copying the sources of the
Rust modules into your kernel module (which is a crate), so that you
do not need to deal with build systems.

But if you are looking to do something else, then please note there is
no plan to add support for building/linking against third-party
crates. This is why we ask what crates would be useful and why -- so
that we can consider adding similar facilities to the kernel (possibly
importing those third-party crates).

Cheers,
Miguel

Re: Fwd: Adding crates

[Chris Suter]

Hi Miguel,

On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote:
> >
> > I'm interested in quite a few crates but the first and perhaps most
> > significant would be futures. I'm not necessarily looking for a
> > "generic" solution at this point, any solution would do — just enough
> > to help me make a little progress and then maybe I could contribute
> > back later. I can probably figure it out myself, but I was hoping to
> > save some time.
>
> If you are just looking to experiment with some third-party crates,
> then as mentioned in the other thread, what you could do is copy and
> adapt the code. For instance, if the crates you need do not use Cargo
> build scripts, this may be as simple as copying the sources of the
> Rust modules into your kernel module (which is a crate), so that you
> do not need to deal with build systems.

I'm working with an existing code base and there are quite a few
crates I need to bring in or find replacements for. I'm hoping to
minimise or isolate the changes I need to make to the existing code
base. Obviously, some changes are going to be unavoidable.

To start with, what I'd like to do is bring in a crate and build it as
a library and then link it with my module. I just need to work out the
magic incantation to do that.

> But if you are looking to do something else, then please note there is
> no plan to add support for building/linking against third-party
> crates. This is why we ask what crates would be useful and why -- so
> that we can consider adding similar facilities to the kernel (possibly
> importing those third-party crates).

Like I said, I'm interested in futures. Why it's useful: async Rust is
arguably more common and easier to use than other forms of
multi-threaded processing. Other crates that I'd like: anyhow,
bincode, byteorder, log, once_cell, pin-project, rand, serde, slab,
static_assertions, uuid plus some more esoteric ones.

At this stage, I'm really just experimenting to see what's possible.

Anyway, thanks for your response. I'll keep playing and see where I get to.

Kind regards,

Chris

>
> Cheers,
> Miguel

Re: Fwd: Adding crates

[Greg KH]

On Wed, Mar 30, 2022 at 11:03:50AM +1100, Chris Suter wrote:
> Hi Miguel,
> 
> On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda
> <miguel.ojeda.sandonis@gmail.com> wrote:
> >
> > On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote:
> > >
> > > I'm interested in quite a few crates but the first and perhaps most
> > > significant would be futures. I'm not necessarily looking for a
> > > "generic" solution at this point, any solution would do — just enough
> > > to help me make a little progress and then maybe I could contribute
> > > back later. I can probably figure it out myself, but I was hoping to
> > > save some time.
> >
> > If you are just looking to experiment with some third-party crates,
> > then as mentioned in the other thread, what you could do is copy and
> > adapt the code. For instance, if the crates you need do not use Cargo
> > build scripts, this may be as simple as copying the sources of the
> > Rust modules into your kernel module (which is a crate), so that you
> > do not need to deal with build systems.
> 
> I'm working with an existing code base and there are quite a few
> crates I need to bring in or find replacements for. I'm hoping to
> minimise or isolate the changes I need to make to the existing code
> base. Obviously, some changes are going to be unavoidable.

What type of existing Rust codebase should be ported to be inside the
kernel?

> > But if you are looking to do something else, then please note there is
> > no plan to add support for building/linking against third-party
> > crates. This is why we ask what crates would be useful and why -- so
> > that we can consider adding similar facilities to the kernel (possibly
> > importing those third-party crates).
> 
> Like I said, I'm interested in futures. Why it's useful: async Rust is
> arguably more common and easier to use than other forms of
> multi-threaded processing. Other crates that I'd like: anyhow,
> bincode, byteorder, log, once_cell, pin-project, rand, serde, slab,
> static_assertions, uuid plus some more esoteric ones.

Almost none of those make sense when you are writing kernel code.  Are
you sure you want all of this within the kernel itself?

What does this codebase do?  Have a link to it anywhere?

thanks,

greg k-h

Re: Fwd: Adding crates

[Kent Overstreet]

On Wed, Mar 30, 2022 at 06:31:28AM +0200, Greg KH wrote:
> On Wed, Mar 30, 2022 at 11:03:50AM +1100, Chris Suter wrote:
> > Hi Miguel,
> > 
> > On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda
> > <miguel.ojeda.sandonis@gmail.com> wrote:
> > >
> > > On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote:
> > > >
> > > > I'm interested in quite a few crates but the first and perhaps most
> > > > significant would be futures. I'm not necessarily looking for a
> > > > "generic" solution at this point, any solution would do — just enough
> > > > to help me make a little progress and then maybe I could contribute
> > > > back later. I can probably figure it out myself, but I was hoping to
> > > > save some time.
> > >
> > > If you are just looking to experiment with some third-party crates,
> > > then as mentioned in the other thread, what you could do is copy and
> > > adapt the code. For instance, if the crates you need do not use Cargo
> > > build scripts, this may be as simple as copying the sources of the
> > > Rust modules into your kernel module (which is a crate), so that you
> > > do not need to deal with build systems.
> > 
> > I'm working with an existing code base and there are quite a few
> > crates I need to bring in or find replacements for. I'm hoping to
> > minimise or isolate the changes I need to make to the existing code
> > base. Obviously, some changes are going to be unavoidable.
> 
> What type of existing Rust codebase should be ported to be inside the
> kernel?

Greg, I think your point of view made sense in the past, but the world is
changing. We used to not want to share code between kernel space and userspace
as much in the first place, and in C it has historically been impractical, but
consider the following:

 - We already have large swaths of code being imported from userspace into the
   kernel as-is - e.g. zstd, and I bet I could find a lot more if I went
   digging.

   Right now we do this by just importing the code wholesale, but when changes
   end up needing to be applied for it to work in the kernel, this is a
   maintenance nightmare when we then need to re-merge from upstream.

   It would be far better to get the changes needed for use in the kernel merged
   with upstream, and then just use upstream directly like any other package
   (with version pinning, so that when upstream changes, we review those changes
   before updating the version the kernel depends on). And Rust's conditional
   compilation & generics story is _much_ better than what we've had in the past
   in C and C++, meaning writing Rust code that works in both userspace and
   kernelspace is much saner than in the past.

 - People are also starting to want to take kernel code and use it in userspace
   more often. In filesystem land, where I want, you need to do this or else you
   have massive duplication (& corresponding maintenance nightmare) between
   kernelspace and e.g. userspace fsck. In other areas, people just want to be
   able to write and run tests in userspace - e.g. xarrays, maple trees, and
   this is something we should encourage!

   But, when the kernel space & userspace environments are completely different,
   this is a massive undertaking. For bcachefs I had to write shim
   implementations for a _ton_ of code, most of which was quick and dirty
   throwaway code that's a pain in the ass to work with and that I would really
   like to delete.

The solution to problems like these are to stop thinking that kernelspace and
userspace _have_ to be completely different beasts - they really don't have to
be! and start encouraging different thinking and tooling improvements that will
make our lives easier.

In the kernel we also reinvent the wheel a _lot_ with our core data structures.
Some of the stuff we have is awesome, some is... amazing in some ways but
terrifying to have to deal with when it breaks (rhashtables, I'm looking at
you), but often the glaringly obvious basic stuff is missing - why do we have
nothing like CCAN darray? never mind, I finally just wrote one...

But code sharing for core data structures is finally starting to look tractable:
kernel side GFP flags is getting deprecated in favour of
memalloc_no*_save/restore (thank you Willy!) - that's a big one. And the Rust
people, to their credit, have been taking seriously kernel land's needs to
always check for memory allocation failure - Rust is explicitly targeted at bare
metal, kernel stuff!

Anyways. Just try and keep an open mind...

Re: Fwd: Adding crates

[James Bottomley]

[added cc to linux-fsdevel, since they've long grappled with this
problem]
On Wed, 2022-03-30 at 16:43 -0400, Kent Overstreet wrote:
> On Wed, Mar 30, 2022 at 06:31:28AM +0200, Greg KH wrote:
> > On Wed, Mar 30, 2022 at 11:03:50AM +1100, Chris Suter wrote:
> > > Hi Miguel,
> > > 
> > > On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda
> > > <miguel.ojeda.sandonis@gmail.com> wrote:
> > > > On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me>
> > > > wrote:
> > > > > I'm interested in quite a few crates but the first and
> > > > > perhaps most significant would be futures. I'm not
> > > > > necessarily looking for a "generic" solution at this point,
> > > > > any solution would do — just enough to help me make a little
> > > > > progress and then maybe I could contribute back later. I can
> > > > > probably figure it out myself, but I was hoping to save some
> > > > > time.
> > > > 
> > > > If you are just looking to experiment with some third-party
> > > > crates, then as mentioned in the other thread, what you could
> > > > do is copy and adapt the code. For instance, if the crates you
> > > > need do not use Cargo build scripts, this may be as simple as
> > > > copying the sources of the Rust modules into your kernel module
> > > > (which is a crate), so that you do not need to deal with build
> > > > systems.
> > > 
> > > I'm working with an existing code base and there are quite a few
> > > crates I need to bring in or find replacements for. I'm hoping to
> > > minimise or isolate the changes I need to make to the existing
> > > code base. Obviously, some changes are going to be unavoidable.
> > 
> > What type of existing Rust codebase should be ported to be inside
> > the kernel?
> 
> Greg, I think your point of view made sense in the past, but the
> world is changing. We used to not want to share code between kernel
> space and userspace as much in the first place, and in C it has
> historically been impractical, but consider the following:
> 
>  - We already have large swaths of code being imported from userspace
> into the kernel as-is - e.g. zstd, and I bet I could find a lot more
> if I went digging.

Compression algorithms are somewhat "special".  The reference
implementation can often be imported as is with a few minor
modifications because if we're lucky it only really needs heap
alloc/free which we can provide.  The bigger problem is the use API. 
most compression algorithms eventually find themselves reimplemented in
many ways in many crypto systems even in userspace ... if zstd ever
gets standardized for compressed TLS, it will find itself wrappered in
openssl, for instance.  So, really, what the kernel is doing is no
different from what a lot of user space will do.

>    Right now we do this by just importing the code wholesale, but
> when changes end up needing to be applied for it to work in the
> kernel, this is a    maintenance nightmare when we then need to re-
> merge from upstream.

lib/zstd is ~300 lines of code, it's it doesn't seem to be that
difficult to maintain.  The bigger problem is that the "reference"
implementation is controlled by facebook and arbitrarily modified by
them to support their applications (Facebook isn't unique here, this
happens to a lot of reference implementations).  If you do a wc -l in
lib/ in the zstd repository it's ~74,000 lines.  So basically all we
care about is 0.4% of the supposed "reference" implementation.  Pulling
in the other 99.6% would serve no purpose other than to bloat our
attack surface, confuse the security analysis of the code and generate
a huge amount of static checking noise.

>    It would be far better to get the changes needed for use in the
> kernel merged  with upstream, and then just use upstream directly
> like any other package (with version pinning, so that when upstream
> changes, we review those changes before updating the version the
> kernel depends on).

As I already think I demonstrated, we don't want or need 99.4% of the
upstream ... most of the changes if you read the commit log are in API
areas the core compression algorithm doesn't change that much.  An even
bigger problem is a lot of that API surface uses C library interfaces
we don't support so we'd either have to grow pointless support or find
a way of stripping them anyway.

The point I'm making is that in its current form we really, really
don't want to sync with upstream.  It would be really great if upstream
maintained a usable core of pure compression algorithms that we could
sync with, but it currently doesn't.

The above is also the rust crate problem in miniature: the crates grow
API features the kernel will never care about and importing them
wholesale is going to take forever because of the internal kernel
support issue.  In the end, to take rust async as an example, it will
be much better to do for rust what we've done for zlib: take the core
that can support the kernel threading model and reimplement that in the
kernel crate.  The act of doing that will a) prove people care enough
about the functionality and b) allow us to refine it nicely.

I also don't think rust would really want to import crates wholesale. 
The reason for no_std is that rust is trying to adapt to embedded
environments, which the somewhat harsh constraints of the kernel is
very similar to.

>  And Rust's conditional    compilation & generics story is _much_
> better than what we've had in the past in C and C++, meaning writing
> Rust code that works in both userspace and kernelspace is much saner
> than in the past.
> 
>  - People are also starting to want to take kernel code and use it in
> userspace more often. In filesystem land, where I want, you need to
> do this or else you have massive duplication (& corresponding
> maintenance nightmare) between kernelspace and e.g. userspace fsck.
> In other areas, people just want to be able to write and run tests in
> userspace - e.g. xarrays, maple trees, and this is something we
> should encourage!

But the key point is if there were a reference core of just the bare
implementation separated from API wrappers, we could share it way more
easily.  It's not about having to import any old crap into the kernel,
it's about the maintainers of the reference implementation having to be
aware of all the possible uses of their code.

The counter example to everything you cite below is libxfs.  That's a
reference library that's shared between the xfs user space tools and
the kernel.  The point being that if you're willing to maintain just
the core of what you need in a style that's easily importable in
various environments, the task of importing it as the reference
implementation becomes easier.

>    But, when the kernel space & userspace environments are completely
> different, this is a massive undertaking. For bcachefs I had to write
> shim implementations for a _ton_ of code, most of which was quick and
> dirty  throwaway code that's a pain in the ass to work with and that
> I would really like to delete.
> 
> The solution to problems like these are to stop thinking that
> kernelspace and userspace _have_ to be completely different beasts -
> they really don't have to be! and start encouraging different
> thinking and tooling improvements that will make our lives easier.
> 
> In the kernel we also reinvent the wheel a _lot_ with our core data
> structures. Some of the stuff we have is awesome, some is... amazing
> in some ways but terrifying to have to deal with when it breaks
> (rhashtables, I'm looking at you), but often the glaringly obvious
> basic stuff is missing - why do we have nothing like CCAN darray?
> never mind, I finally just wrote one...
> 
> But code sharing for core data structures is finally starting to look
> tractable: kernel side GFP flags is getting deprecated in favour of
> memalloc_no*_save/restore (thank you Willy!) - that's a big one. And
> the Rust people, to their credit, have been taking seriously kernel
> land's needs to always check for memory allocation failure - Rust is
> explicitly targeted at bare metal, kernel stuff!
> 
> Anyways. Just try and keep an open mind...

I think the solution to the problem is to try to maintain highly mobile
reference implementations and also keep constantly considering what the
requirements are for mobility (both on the part of the reference
implementation and the kernel).  I also think that if we ever gave
impementors the magic ability just to dump any old code in the kernel,
they'd use it to take the lazy way out because it's a lot easier than
trying to keep the reference implementation separate.

The fact that most "reference" implementations don't conform to the
above isn't something we should be encouraging by supplying
compatibility APIs that paper over the problem and encourage API bloat.

James

Re: Fwd: Adding crates

[Kent Overstreet]

On Fri, Apr 15, 2022 at 09:31:24AM -0400, James Bottomley wrote:
> I think the solution to the problem is to try to maintain highly mobile
> reference implementations and also keep constantly considering what the
> requirements are for mobility (both on the part of the reference
> implementation and the kernel).  I also think that if we ever gave
> impementors the magic ability just to dump any old code in the kernel,
> they'd use it to take the lazy way out because it's a lot easier than
> trying to keep the reference implementation separate.
> 
> The fact that most "reference" implementations don't conform to the
> above isn't something we should be encouraging by supplying
> compatibility APIs that paper over the problem and encourage API bloat.

I think it might help if we had a
 - standard set of review guidelines
 - standard workflow

for pulling in (vendoring, however it gets done) code from external
repositories.

Generally what I see in the community is that people do want to support the
kernel better, but then people see things like Greg's response of "don't do
that" and it turns people off. Instead, we could
 - recognize that it's already being done (e.g. with zstd)
 - put some things in writing about how it _should_ be done

This could help a lot with e.g. the way Facebook maintains ZSTD - if we say
"look, we want this code factored out better so we only have to review the stuff
that's relevant" - the engineers generally won't mind doing that work, and now
they'll have something to take to their managers as justification.

Another thing I'd like to see is more of what the RCU folks have done - liburcu
is _amazing_ in that, at least for me, it's been a drop in replacement for the
kernel RCU implementation - and I would imagine it's as much as possible the
same code. A good candidate would be the kernel workqueue code: workqueues are
_great_ and I don't know of anything like them in userspace. That code should be
seeing wider use! And if it was made an external library that was consumed by
both the kernel and userspace, it could be - but it would require buy in from
kernel people.

But as you were saying about Facebook (and I discovered when I was at Google
many moons ago, as a young engineer) - large organizations tend to be insular,
they like te pretend the outside world doesn't exist. The kernel is one such
organization :) We could be a better citizen by encouraging and enabling efforts
such as this.

Re: Fwd: Adding crates

[James Bottomley]

On Fri, 2022-04-15 at 16:39 -0400, Kent Overstreet wrote:
> On Fri, Apr 15, 2022 at 09:31:24AM -0400, James Bottomley wrote:
> > I think the solution to the problem is to try to maintain highly
> > mobile reference implementations and also keep constantly
> > considering what the requirements are for mobility (both on the
> > part of the reference implementation and the kernel).  I also think
> > that if we ever gave impementors the magic ability just to dump any
> > old code in the kernel, they'd use it to take the lazy way out
> > because it's a lot easier than trying to keep the reference
> > implementation separate.
> > 
> > The fact that most "reference" implementations don't conform to the
> > above isn't something we should be encouraging by supplying
> > compatibility APIs that paper over the problem and encourage API
> > bloat.
> 
> I think it might help if we had a
>  - standard set of review guidelines
>  - standard workflow
> 
> for pulling in (vendoring, however it gets done) code from external
> repositories.

But that's precisely what we shouldn't have.  For most standard
implementations it shouldn't be done.  zlib is a case in point: even if
we could pull in 74k lines from an external repo to try to extract the
300 we need that's not going to be via anything like a standard
process.  Now there might be something we could assist with in
standardising how reference implementations should be done to make them
mobile to environments like embedded and the kernel, but if the project
isn't structured like this, it's already pretty much too late.

> Generally what I see in the community is that people do want to
> support the kernel better, but then people see things like Greg's
> response of "don't do that" and it turns people off. Instead, we
> could 
>  - recognize that it's already being done (e.g. with zstd)

It hasn't been done with zstd.  What we have looks like a reference
implementation based on RFC 8478, it's nothing like the facebook github
for zstd and I'm sure more people than Greg (me included) would object
greatly to trying to swap our 300 line implementation for the facebook
74k line one.

>  - put some things in writing about how it _should_ be done

This is where I think we could help.  As I said: libxfs (and librcu as
you point out below) already exists as a blueprint.  For rust async,
it's a bit more complex because it depends on whether the async support
in the compiler can truly be made compatible with the kernel threading
model rather than simply vendoring the async crates.

> This could help a lot with e.g. the way Facebook maintains ZSTD - if
> we say "look, we want this code factored out better so we only have
> to review the stuff that's relevant" - the engineers generally won't
> mind doing that work, and now they'll have something to take to their
> managers as justification.

Well, they do, they just don't expect to do it by us vendoring the
reference implementation:

https://github.com/facebook/zstd/issues/1638

In the above, the facebook maintainer is willing to extract the
reference core and add it to the linux kernel.  That seems to be a
great result for us.

> Another thing I'd like to see is more of what the RCU folks have done
> - liburcu is _amazing_ in that, at least for me, it's been a drop in
> replacement for the kernel RCU implementation - and I would imagine
> it's as much as possible the same code. A good candidate would be the
> kernel workqueue code: workqueues are _great_ and I don't know of
> anything like them in userspace. That code should be
> seeing wider use! And if it was made an external library that was
> consumed by both the kernel and userspace, it could be - but it would
> require buy in from kernel people.

That might work for , but most other languages need something different
(and often have their own implementation, like python:

https://docs.python.org/3/library/queue.html

or rabbitMQ or something).

> But as you were saying about Facebook (and I discovered when I was at
> Google many moons ago, as a young engineer) - large organizations
> tend to be insular, they like te pretend the outside world doesn't
> exist. The kernel is one such organization :) We could be a better
> citizen by encouraging and enabling efforts such as this.

Well, I think they are.  To be clear, I think the way zstd is supported
in the kernel is great.  If it works for facebook (having the
maintainer extract it to the kernel and send us pull requests), I don't
see a problem with keeping it the way it is rather than trying to
vendor the existing code.

Regards,

James