* Fwd: Adding crates [not found] <CAKfU0DKvQYjhgiKN83+DZW3CQOkSEdQcy9nXUfFLVn1Uju6GkQ@mail.gmail.com> @ 2022-03-28 0:41 ` Chris Suter 2022-03-28 5:41 ` Geert Stappers 0 siblings, 1 reply; 12+ messages in thread From: Chris Suter @ 2022-03-28 0:41 UTC (permalink / raw) To: rust-for-linux Hey folks, I'm playing with a kernel module written in Rust, and for the thing I'm working on, I need to pull in a number of crates. The most significant of those might be some async ones: (e.g. futures, async_trait). What's the best way to pull those in and get them building with my module? In particular, I'm interested in how to structure the Makefile to make it work. Are there any examples I could follow? Cheers, Chris ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-28 0:41 ` Fwd: Adding crates Chris Suter @ 2022-03-28 5:41 ` Geert Stappers 2022-03-28 21:38 ` Chris Suter 0 siblings, 1 reply; 12+ messages in thread From: Geert Stappers @ 2022-03-28 5:41 UTC (permalink / raw) To: Chris Suter; +Cc: rust-for-linux On Mon, Mar 28, 2022 at 11:41:43AM +1100, Chris Suter wrote: > Hey folks, > > I'm playing with a kernel module written in Rust, and for the thing > I'm working on, I need to pull in a number of crates. The most > significant of those might be some async ones: (e.g. futures, > async_trait). What's the best way to pull those in and get them > building with my module? In particular, I'm interested in how to > structure the Makefile to make it work. Are there any examples I could > follow? See https://lore.kernel.org/rust-for-linux/CANiq72my9N51hhryv-Cp79eUmPVCx6q=0fJrhkzmvT+nVPq+WQ@mail.gmail.com/T/#t for a recent thread "Using 3rd party crates in an out-of-tree module" > Cheers, > Chris Groeten Geert Stappers -- Silence is hard to parse ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-28 5:41 ` Geert Stappers @ 2022-03-28 21:38 ` Chris Suter 2022-03-29 5:23 ` Greg KH 0 siblings, 1 reply; 12+ messages in thread From: Chris Suter @ 2022-03-28 21:38 UTC (permalink / raw) To: Geert Stappers; +Cc: rust-for-linux I skimmed that thread, but unless I'm missing something, it seems to be pretty much "don't do it", which doesn't work for me right now. I'm just looking to get something working and from there I can figure out the best way to integrate it — I'm just looking for the easiest (and not necessarily the best/long-term) way to bring in dependent crates. — Chris On Mon, Mar 28, 2022 at 4:41 PM Geert Stappers <stappers@stappers.nl> wrote: > > On Mon, Mar 28, 2022 at 11:41:43AM +1100, Chris Suter wrote: > > Hey folks, > > > > I'm playing with a kernel module written in Rust, and for the thing > > I'm working on, I need to pull in a number of crates. The most > > significant of those might be some async ones: (e.g. futures, > > async_trait). What's the best way to pull those in and get them > > building with my module? In particular, I'm interested in how to > > structure the Makefile to make it work. Are there any examples I could > > follow? > > See https://lore.kernel.org/rust-for-linux/CANiq72my9N51hhryv-Cp79eUmPVCx6q=0fJrhkzmvT+nVPq+WQ@mail.gmail.com/T/#t > for a recent thread "Using 3rd party crates in an out-of-tree module" > > > > Cheers, > > Chris > > > Groeten > Geert Stappers > -- > Silence is hard to parse ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-28 21:38 ` Chris Suter @ 2022-03-29 5:23 ` Greg KH 2022-03-29 6:02 ` Chris Suter 0 siblings, 1 reply; 12+ messages in thread From: Greg KH @ 2022-03-29 5:23 UTC (permalink / raw) To: Chris Suter; +Cc: Geert Stappers, rust-for-linux On Tue, Mar 29, 2022 at 08:38:29AM +1100, Chris Suter wrote: > I skimmed that thread, but unless I'm missing something, it seems to > be pretty much "don't do it", which doesn't work for me right now. Why not? What is the deadline? And what specific crates do you need and why? There is no "generic" solution here other than "do not do this." thanks, greg k-h ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-29 5:23 ` Greg KH @ 2022-03-29 6:02 ` Chris Suter 2022-03-29 15:47 ` Miguel Ojeda 0 siblings, 1 reply; 12+ messages in thread From: Chris Suter @ 2022-03-29 6:02 UTC (permalink / raw) To: Greg KH; +Cc: Geert Stappers, rust-for-linux On Tue, Mar 29, 2022 at 4:23 PM Greg KH <gregkh@linuxfoundation.org> wrote: > > On Tue, Mar 29, 2022 at 08:38:29AM +1100, Chris Suter wrote: > > I skimmed that thread, but unless I'm missing something, it seems to > > be pretty much "don't do it", which doesn't work for me right now. > > Why not? What is the deadline? > > And what specific crates do you need and why? There is no "generic" > solution here other than "do not do this." I'm interested in quite a few crates but the first and perhaps most significant would be futures. I'm not necessarily looking for a "generic" solution at this point, any solution would do — just enough to help me make a little progress and then maybe I could contribute back later. I can probably figure it out myself, but I was hoping to save some time. — Chris > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-29 6:02 ` Chris Suter @ 2022-03-29 15:47 ` Miguel Ojeda 2022-03-30 0:03 ` Chris Suter 0 siblings, 1 reply; 12+ messages in thread From: Miguel Ojeda @ 2022-03-29 15:47 UTC (permalink / raw) To: Chris Suter; +Cc: Greg KH, Geert Stappers, rust-for-linux On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote: > > I'm interested in quite a few crates but the first and perhaps most > significant would be futures. I'm not necessarily looking for a > "generic" solution at this point, any solution would do — just enough > to help me make a little progress and then maybe I could contribute > back later. I can probably figure it out myself, but I was hoping to > save some time. If you are just looking to experiment with some third-party crates, then as mentioned in the other thread, what you could do is copy and adapt the code. For instance, if the crates you need do not use Cargo build scripts, this may be as simple as copying the sources of the Rust modules into your kernel module (which is a crate), so that you do not need to deal with build systems. But if you are looking to do something else, then please note there is no plan to add support for building/linking against third-party crates. This is why we ask what crates would be useful and why -- so that we can consider adding similar facilities to the kernel (possibly importing those third-party crates). Cheers, Miguel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-29 15:47 ` Miguel Ojeda @ 2022-03-30 0:03 ` Chris Suter 2022-03-30 4:31 ` Greg KH 0 siblings, 1 reply; 12+ messages in thread From: Chris Suter @ 2022-03-30 0:03 UTC (permalink / raw) To: Miguel Ojeda; +Cc: Greg KH, Geert Stappers, rust-for-linux Hi Miguel, On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote: > > On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote: > > > > I'm interested in quite a few crates but the first and perhaps most > > significant would be futures. I'm not necessarily looking for a > > "generic" solution at this point, any solution would do — just enough > > to help me make a little progress and then maybe I could contribute > > back later. I can probably figure it out myself, but I was hoping to > > save some time. > > If you are just looking to experiment with some third-party crates, > then as mentioned in the other thread, what you could do is copy and > adapt the code. For instance, if the crates you need do not use Cargo > build scripts, this may be as simple as copying the sources of the > Rust modules into your kernel module (which is a crate), so that you > do not need to deal with build systems. I'm working with an existing code base and there are quite a few crates I need to bring in or find replacements for. I'm hoping to minimise or isolate the changes I need to make to the existing code base. Obviously, some changes are going to be unavoidable. To start with, what I'd like to do is bring in a crate and build it as a library and then link it with my module. I just need to work out the magic incantation to do that. > But if you are looking to do something else, then please note there is > no plan to add support for building/linking against third-party > crates. This is why we ask what crates would be useful and why -- so > that we can consider adding similar facilities to the kernel (possibly > importing those third-party crates). Like I said, I'm interested in futures. Why it's useful: async Rust is arguably more common and easier to use than other forms of multi-threaded processing. Other crates that I'd like: anyhow, bincode, byteorder, log, once_cell, pin-project, rand, serde, slab, static_assertions, uuid plus some more esoteric ones. At this stage, I'm really just experimenting to see what's possible. Anyway, thanks for your response. I'll keep playing and see where I get to. Kind regards, Chris > > Cheers, > Miguel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-30 0:03 ` Chris Suter @ 2022-03-30 4:31 ` Greg KH 2022-03-30 20:43 ` Kent Overstreet 0 siblings, 1 reply; 12+ messages in thread From: Greg KH @ 2022-03-30 4:31 UTC (permalink / raw) To: Chris Suter; +Cc: Miguel Ojeda, Geert Stappers, rust-for-linux On Wed, Mar 30, 2022 at 11:03:50AM +1100, Chris Suter wrote: > Hi Miguel, > > On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda > <miguel.ojeda.sandonis@gmail.com> wrote: > > > > On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote: > > > > > > I'm interested in quite a few crates but the first and perhaps most > > > significant would be futures. I'm not necessarily looking for a > > > "generic" solution at this point, any solution would do — just enough > > > to help me make a little progress and then maybe I could contribute > > > back later. I can probably figure it out myself, but I was hoping to > > > save some time. > > > > If you are just looking to experiment with some third-party crates, > > then as mentioned in the other thread, what you could do is copy and > > adapt the code. For instance, if the crates you need do not use Cargo > > build scripts, this may be as simple as copying the sources of the > > Rust modules into your kernel module (which is a crate), so that you > > do not need to deal with build systems. > > I'm working with an existing code base and there are quite a few > crates I need to bring in or find replacements for. I'm hoping to > minimise or isolate the changes I need to make to the existing code > base. Obviously, some changes are going to be unavoidable. What type of existing Rust codebase should be ported to be inside the kernel? > > But if you are looking to do something else, then please note there is > > no plan to add support for building/linking against third-party > > crates. This is why we ask what crates would be useful and why -- so > > that we can consider adding similar facilities to the kernel (possibly > > importing those third-party crates). > > Like I said, I'm interested in futures. Why it's useful: async Rust is > arguably more common and easier to use than other forms of > multi-threaded processing. Other crates that I'd like: anyhow, > bincode, byteorder, log, once_cell, pin-project, rand, serde, slab, > static_assertions, uuid plus some more esoteric ones. Almost none of those make sense when you are writing kernel code. Are you sure you want all of this within the kernel itself? What does this codebase do? Have a link to it anywhere? thanks, greg k-h ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-30 4:31 ` Greg KH @ 2022-03-30 20:43 ` Kent Overstreet 2022-04-15 13:31 ` James Bottomley 0 siblings, 1 reply; 12+ messages in thread From: Kent Overstreet @ 2022-03-30 20:43 UTC (permalink / raw) To: Greg KH; +Cc: Chris Suter, Miguel Ojeda, Geert Stappers, rust-for-linux On Wed, Mar 30, 2022 at 06:31:28AM +0200, Greg KH wrote: > On Wed, Mar 30, 2022 at 11:03:50AM +1100, Chris Suter wrote: > > Hi Miguel, > > > > On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda > > <miguel.ojeda.sandonis@gmail.com> wrote: > > > > > > On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote: > > > > > > > > I'm interested in quite a few crates but the first and perhaps most > > > > significant would be futures. I'm not necessarily looking for a > > > > "generic" solution at this point, any solution would do — just enough > > > > to help me make a little progress and then maybe I could contribute > > > > back later. I can probably figure it out myself, but I was hoping to > > > > save some time. > > > > > > If you are just looking to experiment with some third-party crates, > > > then as mentioned in the other thread, what you could do is copy and > > > adapt the code. For instance, if the crates you need do not use Cargo > > > build scripts, this may be as simple as copying the sources of the > > > Rust modules into your kernel module (which is a crate), so that you > > > do not need to deal with build systems. > > > > I'm working with an existing code base and there are quite a few > > crates I need to bring in or find replacements for. I'm hoping to > > minimise or isolate the changes I need to make to the existing code > > base. Obviously, some changes are going to be unavoidable. > > What type of existing Rust codebase should be ported to be inside the > kernel? Greg, I think your point of view made sense in the past, but the world is changing. We used to not want to share code between kernel space and userspace as much in the first place, and in C it has historically been impractical, but consider the following: - We already have large swaths of code being imported from userspace into the kernel as-is - e.g. zstd, and I bet I could find a lot more if I went digging. Right now we do this by just importing the code wholesale, but when changes end up needing to be applied for it to work in the kernel, this is a maintenance nightmare when we then need to re-merge from upstream. It would be far better to get the changes needed for use in the kernel merged with upstream, and then just use upstream directly like any other package (with version pinning, so that when upstream changes, we review those changes before updating the version the kernel depends on). And Rust's conditional compilation & generics story is _much_ better than what we've had in the past in C and C++, meaning writing Rust code that works in both userspace and kernelspace is much saner than in the past. - People are also starting to want to take kernel code and use it in userspace more often. In filesystem land, where I want, you need to do this or else you have massive duplication (& corresponding maintenance nightmare) between kernelspace and e.g. userspace fsck. In other areas, people just want to be able to write and run tests in userspace - e.g. xarrays, maple trees, and this is something we should encourage! But, when the kernel space & userspace environments are completely different, this is a massive undertaking. For bcachefs I had to write shim implementations for a _ton_ of code, most of which was quick and dirty throwaway code that's a pain in the ass to work with and that I would really like to delete. The solution to problems like these are to stop thinking that kernelspace and userspace _have_ to be completely different beasts - they really don't have to be! and start encouraging different thinking and tooling improvements that will make our lives easier. In the kernel we also reinvent the wheel a _lot_ with our core data structures. Some of the stuff we have is awesome, some is... amazing in some ways but terrifying to have to deal with when it breaks (rhashtables, I'm looking at you), but often the glaringly obvious basic stuff is missing - why do we have nothing like CCAN darray? never mind, I finally just wrote one... But code sharing for core data structures is finally starting to look tractable: kernel side GFP flags is getting deprecated in favour of memalloc_no*_save/restore (thank you Willy!) - that's a big one. And the Rust people, to their credit, have been taking seriously kernel land's needs to always check for memory allocation failure - Rust is explicitly targeted at bare metal, kernel stuff! Anyways. Just try and keep an open mind... ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-03-30 20:43 ` Kent Overstreet @ 2022-04-15 13:31 ` James Bottomley 2022-04-15 20:39 ` Kent Overstreet 0 siblings, 1 reply; 12+ messages in thread From: James Bottomley @ 2022-04-15 13:31 UTC (permalink / raw) To: Kent Overstreet, Greg KH Cc: Chris Suter, Miguel Ojeda, Geert Stappers, rust-for-linux, linux-fsdevel [added cc to linux-fsdevel, since they've long grappled with this problem] On Wed, 2022-03-30 at 16:43 -0400, Kent Overstreet wrote: > On Wed, Mar 30, 2022 at 06:31:28AM +0200, Greg KH wrote: > > On Wed, Mar 30, 2022 at 11:03:50AM +1100, Chris Suter wrote: > > > Hi Miguel, > > > > > > On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda > > > <miguel.ojeda.sandonis@gmail.com> wrote: > > > > On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> > > > > wrote: > > > > > I'm interested in quite a few crates but the first and > > > > > perhaps most significant would be futures. I'm not > > > > > necessarily looking for a "generic" solution at this point, > > > > > any solution would do — just enough to help me make a little > > > > > progress and then maybe I could contribute back later. I can > > > > > probably figure it out myself, but I was hoping to save some > > > > > time. > > > > > > > > If you are just looking to experiment with some third-party > > > > crates, then as mentioned in the other thread, what you could > > > > do is copy and adapt the code. For instance, if the crates you > > > > need do not use Cargo build scripts, this may be as simple as > > > > copying the sources of the Rust modules into your kernel module > > > > (which is a crate), so that you do not need to deal with build > > > > systems. > > > > > > I'm working with an existing code base and there are quite a few > > > crates I need to bring in or find replacements for. I'm hoping to > > > minimise or isolate the changes I need to make to the existing > > > code base. Obviously, some changes are going to be unavoidable. > > > > What type of existing Rust codebase should be ported to be inside > > the kernel? > > Greg, I think your point of view made sense in the past, but the > world is changing. We used to not want to share code between kernel > space and userspace as much in the first place, and in C it has > historically been impractical, but consider the following: > > - We already have large swaths of code being imported from userspace > into the kernel as-is - e.g. zstd, and I bet I could find a lot more > if I went digging. Compression algorithms are somewhat "special". The reference implementation can often be imported as is with a few minor modifications because if we're lucky it only really needs heap alloc/free which we can provide. The bigger problem is the use API. most compression algorithms eventually find themselves reimplemented in many ways in many crypto systems even in userspace ... if zstd ever gets standardized for compressed TLS, it will find itself wrappered in openssl, for instance. So, really, what the kernel is doing is no different from what a lot of user space will do. > Right now we do this by just importing the code wholesale, but > when changes end up needing to be applied for it to work in the > kernel, this is a maintenance nightmare when we then need to re- > merge from upstream. lib/zstd is ~300 lines of code, it's it doesn't seem to be that difficult to maintain. The bigger problem is that the "reference" implementation is controlled by facebook and arbitrarily modified by them to support their applications (Facebook isn't unique here, this happens to a lot of reference implementations). If you do a wc -l in lib/ in the zstd repository it's ~74,000 lines. So basically all we care about is 0.4% of the supposed "reference" implementation. Pulling in the other 99.6% would serve no purpose other than to bloat our attack surface, confuse the security analysis of the code and generate a huge amount of static checking noise. > It would be far better to get the changes needed for use in the > kernel merged with upstream, and then just use upstream directly > like any other package (with version pinning, so that when upstream > changes, we review those changes before updating the version the > kernel depends on). As I already think I demonstrated, we don't want or need 99.4% of the upstream ... most of the changes if you read the commit log are in API areas the core compression algorithm doesn't change that much. An even bigger problem is a lot of that API surface uses C library interfaces we don't support so we'd either have to grow pointless support or find a way of stripping them anyway. The point I'm making is that in its current form we really, really don't want to sync with upstream. It would be really great if upstream maintained a usable core of pure compression algorithms that we could sync with, but it currently doesn't. The above is also the rust crate problem in miniature: the crates grow API features the kernel will never care about and importing them wholesale is going to take forever because of the internal kernel support issue. In the end, to take rust async as an example, it will be much better to do for rust what we've done for zlib: take the core that can support the kernel threading model and reimplement that in the kernel crate. The act of doing that will a) prove people care enough about the functionality and b) allow us to refine it nicely. I also don't think rust would really want to import crates wholesale. The reason for no_std is that rust is trying to adapt to embedded environments, which the somewhat harsh constraints of the kernel is very similar to. > And Rust's conditional compilation & generics story is _much_ > better than what we've had in the past in C and C++, meaning writing > Rust code that works in both userspace and kernelspace is much saner > than in the past. > > - People are also starting to want to take kernel code and use it in > userspace more often. In filesystem land, where I want, you need to > do this or else you have massive duplication (& corresponding > maintenance nightmare) between kernelspace and e.g. userspace fsck. > In other areas, people just want to be able to write and run tests in > userspace - e.g. xarrays, maple trees, and this is something we > should encourage! But the key point is if there were a reference core of just the bare implementation separated from API wrappers, we could share it way more easily. It's not about having to import any old crap into the kernel, it's about the maintainers of the reference implementation having to be aware of all the possible uses of their code. The counter example to everything you cite below is libxfs. That's a reference library that's shared between the xfs user space tools and the kernel. The point being that if you're willing to maintain just the core of what you need in a style that's easily importable in various environments, the task of importing it as the reference implementation becomes easier. > But, when the kernel space & userspace environments are completely > different, this is a massive undertaking. For bcachefs I had to write > shim implementations for a _ton_ of code, most of which was quick and > dirty throwaway code that's a pain in the ass to work with and that > I would really like to delete. > > The solution to problems like these are to stop thinking that > kernelspace and userspace _have_ to be completely different beasts - > they really don't have to be! and start encouraging different > thinking and tooling improvements that will make our lives easier. > > In the kernel we also reinvent the wheel a _lot_ with our core data > structures. Some of the stuff we have is awesome, some is... amazing > in some ways but terrifying to have to deal with when it breaks > (rhashtables, I'm looking at you), but often the glaringly obvious > basic stuff is missing - why do we have nothing like CCAN darray? > never mind, I finally just wrote one... > > But code sharing for core data structures is finally starting to look > tractable: kernel side GFP flags is getting deprecated in favour of > memalloc_no*_save/restore (thank you Willy!) - that's a big one. And > the Rust people, to their credit, have been taking seriously kernel > land's needs to always check for memory allocation failure - Rust is > explicitly targeted at bare metal, kernel stuff! > > Anyways. Just try and keep an open mind... I think the solution to the problem is to try to maintain highly mobile reference implementations and also keep constantly considering what the requirements are for mobility (both on the part of the reference implementation and the kernel). I also think that if we ever gave impementors the magic ability just to dump any old code in the kernel, they'd use it to take the lazy way out because it's a lot easier than trying to keep the reference implementation separate. The fact that most "reference" implementations don't conform to the above isn't something we should be encouraging by supplying compatibility APIs that paper over the problem and encourage API bloat. James ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-04-15 13:31 ` James Bottomley @ 2022-04-15 20:39 ` Kent Overstreet 2022-04-16 3:27 ` James Bottomley 0 siblings, 1 reply; 12+ messages in thread From: Kent Overstreet @ 2022-04-15 20:39 UTC (permalink / raw) To: James Bottomley Cc: Greg KH, Chris Suter, Miguel Ojeda, Geert Stappers, rust-for-linux, linux-fsdevel On Fri, Apr 15, 2022 at 09:31:24AM -0400, James Bottomley wrote: > I think the solution to the problem is to try to maintain highly mobile > reference implementations and also keep constantly considering what the > requirements are for mobility (both on the part of the reference > implementation and the kernel). I also think that if we ever gave > impementors the magic ability just to dump any old code in the kernel, > they'd use it to take the lazy way out because it's a lot easier than > trying to keep the reference implementation separate. > > The fact that most "reference" implementations don't conform to the > above isn't something we should be encouraging by supplying > compatibility APIs that paper over the problem and encourage API bloat. I think it might help if we had a - standard set of review guidelines - standard workflow for pulling in (vendoring, however it gets done) code from external repositories. Generally what I see in the community is that people do want to support the kernel better, but then people see things like Greg's response of "don't do that" and it turns people off. Instead, we could - recognize that it's already being done (e.g. with zstd) - put some things in writing about how it _should_ be done This could help a lot with e.g. the way Facebook maintains ZSTD - if we say "look, we want this code factored out better so we only have to review the stuff that's relevant" - the engineers generally won't mind doing that work, and now they'll have something to take to their managers as justification. Another thing I'd like to see is more of what the RCU folks have done - liburcu is _amazing_ in that, at least for me, it's been a drop in replacement for the kernel RCU implementation - and I would imagine it's as much as possible the same code. A good candidate would be the kernel workqueue code: workqueues are _great_ and I don't know of anything like them in userspace. That code should be seeing wider use! And if it was made an external library that was consumed by both the kernel and userspace, it could be - but it would require buy in from kernel people. But as you were saying about Facebook (and I discovered when I was at Google many moons ago, as a young engineer) - large organizations tend to be insular, they like te pretend the outside world doesn't exist. The kernel is one such organization :) We could be a better citizen by encouraging and enabling efforts such as this. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Fwd: Adding crates 2022-04-15 20:39 ` Kent Overstreet @ 2022-04-16 3:27 ` James Bottomley 0 siblings, 0 replies; 12+ messages in thread From: James Bottomley @ 2022-04-16 3:27 UTC (permalink / raw) To: Kent Overstreet Cc: Greg KH, Chris Suter, Miguel Ojeda, Geert Stappers, rust-for-linux, linux-fsdevel On Fri, 2022-04-15 at 16:39 -0400, Kent Overstreet wrote: > On Fri, Apr 15, 2022 at 09:31:24AM -0400, James Bottomley wrote: > > I think the solution to the problem is to try to maintain highly > > mobile reference implementations and also keep constantly > > considering what the requirements are for mobility (both on the > > part of the reference implementation and the kernel). I also think > > that if we ever gave impementors the magic ability just to dump any > > old code in the kernel, they'd use it to take the lazy way out > > because it's a lot easier than trying to keep the reference > > implementation separate. > > > > The fact that most "reference" implementations don't conform to the > > above isn't something we should be encouraging by supplying > > compatibility APIs that paper over the problem and encourage API > > bloat. > > I think it might help if we had a > - standard set of review guidelines > - standard workflow > > for pulling in (vendoring, however it gets done) code from external > repositories. But that's precisely what we shouldn't have. For most standard implementations it shouldn't be done. zlib is a case in point: even if we could pull in 74k lines from an external repo to try to extract the 300 we need that's not going to be via anything like a standard process. Now there might be something we could assist with in standardising how reference implementations should be done to make them mobile to environments like embedded and the kernel, but if the project isn't structured like this, it's already pretty much too late. > Generally what I see in the community is that people do want to > support the kernel better, but then people see things like Greg's > response of "don't do that" and it turns people off. Instead, we > could > - recognize that it's already being done (e.g. with zstd) It hasn't been done with zstd. What we have looks like a reference implementation based on RFC 8478, it's nothing like the facebook github for zstd and I'm sure more people than Greg (me included) would object greatly to trying to swap our 300 line implementation for the facebook 74k line one. > - put some things in writing about how it _should_ be done This is where I think we could help. As I said: libxfs (and librcu as you point out below) already exists as a blueprint. For rust async, it's a bit more complex because it depends on whether the async support in the compiler can truly be made compatible with the kernel threading model rather than simply vendoring the async crates. > This could help a lot with e.g. the way Facebook maintains ZSTD - if > we say "look, we want this code factored out better so we only have > to review the stuff that's relevant" - the engineers generally won't > mind doing that work, and now they'll have something to take to their > managers as justification. Well, they do, they just don't expect to do it by us vendoring the reference implementation: https://github.com/facebook/zstd/issues/1638 In the above, the facebook maintainer is willing to extract the reference core and add it to the linux kernel. That seems to be a great result for us. > Another thing I'd like to see is more of what the RCU folks have done > - liburcu is _amazing_ in that, at least for me, it's been a drop in > replacement for the kernel RCU implementation - and I would imagine > it's as much as possible the same code. A good candidate would be the > kernel workqueue code: workqueues are _great_ and I don't know of > anything like them in userspace. That code should be > seeing wider use! And if it was made an external library that was > consumed by both the kernel and userspace, it could be - but it would > require buy in from kernel people. That might work for , but most other languages need something different (and often have their own implementation, like python: https://docs.python.org/3/library/queue.html or rabbitMQ or something). > But as you were saying about Facebook (and I discovered when I was at > Google many moons ago, as a young engineer) - large organizations > tend to be insular, they like te pretend the outside world doesn't > exist. The kernel is one such organization :) We could be a better > citizen by encouraging and enabling efforts such as this. Well, I think they are. To be clear, I think the way zstd is supported in the kernel is great. If it works for facebook (having the maintainer extract it to the kernel and send us pull requests), I don't see a problem with keeping it the way it is rather than trying to vendor the existing code. Regards, James ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2022-04-16 3:27 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CAKfU0DKvQYjhgiKN83+DZW3CQOkSEdQcy9nXUfFLVn1Uju6GkQ@mail.gmail.com> 2022-03-28 0:41 ` Fwd: Adding crates Chris Suter 2022-03-28 5:41 ` Geert Stappers 2022-03-28 21:38 ` Chris Suter 2022-03-29 5:23 ` Greg KH 2022-03-29 6:02 ` Chris Suter 2022-03-29 15:47 ` Miguel Ojeda 2022-03-30 0:03 ` Chris Suter 2022-03-30 4:31 ` Greg KH 2022-03-30 20:43 ` Kent Overstreet 2022-04-15 13:31 ` James Bottomley 2022-04-15 20:39 ` Kent Overstreet 2022-04-16 3:27 ` James Bottomley
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).