Merging virtualbox shared-folder VFS driver through drivers/staging?

Merging virtualbox shared-folder VFS driver through drivers/staging?

[Hans de Goede]

Hi Al,

I've been trying to get the vboxsf fs code upstream for 1.5 years now,
it seems (to me) that the main problem is that no-one has time to
review it. You're reviewed it a couple of times and David Howells
has reviewed it 2 times. Al reviews have lead to various improvments
and have definitely been useful, so thank you for that.

But ATM, since posting v12 of the patch, it has again been quiet for
2 months again. Since this driver is already being used as addon /
our of tree driver by various distros, I would really like to get it
into mainline, to make live easier for distros and to make sure that
they use the latest version.

Since I do not see the lack of reviewing capacity problem get solved
anytime soon, I was wondering if you are ok with putting the code
in drivers/staging/vboxsf for now, until someone can review it and ack it
for moving over to sf/vboxsf ?

Regards,

Hans

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Matthew Wilcox]

On Sun, Aug 11, 2019 at 12:25:03AM +0200, Hans de Goede wrote:
> But ATM, since posting v12 of the patch, it has again been quiet for
> 2 months again. Since this driver is already being used as addon /
> our of tree driver by various distros, I would really like to get it
> into mainline, to make live easier for distros and to make sure that
> they use the latest version.

fwiw, v12 never made it to the list.  0/1 did, but 1/1 didn't.

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Greg Kroah-Hartman]

On Sun, Aug 11, 2019 at 12:25:03AM +0200, Hans de Goede wrote:
> Hi Al,
> 
> I've been trying to get the vboxsf fs code upstream for 1.5 years now,
> it seems (to me) that the main problem is that no-one has time to
> review it. You're reviewed it a couple of times and David Howells
> has reviewed it 2 times. Al reviews have lead to various improvments
> and have definitely been useful, so thank you for that.
> 
> But ATM, since posting v12 of the patch, it has again been quiet for
> 2 months again. Since this driver is already being used as addon /
> our of tree driver by various distros, I would really like to get it
> into mainline, to make live easier for distros and to make sure that
> they use the latest version.
> 
> Since I do not see the lack of reviewing capacity problem get solved
> anytime soon, I was wondering if you are ok with putting the code
> in drivers/staging/vboxsf for now, until someone can review it and ack it
> for moving over to sf/vboxsf ?

I have no objection to that if the vfs developers do not mind.

thanks,

greg k-h

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Christoph Hellwig]

On Sun, Aug 11, 2019 at 09:40:05AM +0200, Greg Kroah-Hartman wrote:
> > Since I do not see the lack of reviewing capacity problem get solved
> > anytime soon, I was wondering if you are ok with putting the code
> > in drivers/staging/vboxsf for now, until someone can review it and ack it
> > for moving over to sf/vboxsf ?
> 
> I have no objection to that if the vfs developers do not mind.

We had really bad experiences with fs code in staging.  I think it is
a bad idea that should not be repeated.

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Greg Kroah-Hartman]

On Sun, Aug 11, 2019 at 12:43:48AM -0700, Christoph Hellwig wrote:
> On Sun, Aug 11, 2019 at 09:40:05AM +0200, Greg Kroah-Hartman wrote:
> > > Since I do not see the lack of reviewing capacity problem get solved
> > > anytime soon, I was wondering if you are ok with putting the code
> > > in drivers/staging/vboxsf for now, until someone can review it and ack it
> > > for moving over to sf/vboxsf ?
> > 
> > I have no objection to that if the vfs developers do not mind.
> 
> We had really bad experiences with fs code in staging.  I think it is
> a bad idea that should not be repeated.

Lustre was a mistake.  erofs is better in that there are active
developers working to get it out of staging.  We would also need that
here for this to be successful.

Hans, is it just review that is keeping this from being merged or is
there "real work" that has to be done?

thanks,

greg k-h

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Gao Xiang]

Hi hch,

On Sun, Aug 11, 2019 at 12:43:48AM -0700, Christoph Hellwig wrote:
> On Sun, Aug 11, 2019 at 09:40:05AM +0200, Greg Kroah-Hartman wrote:
> > > Since I do not see the lack of reviewing capacity problem get solved
> > > anytime soon, I was wondering if you are ok with putting the code
> > > in drivers/staging/vboxsf for now, until someone can review it and ack it
> > > for moving over to sf/vboxsf ?
> > 
> > I have no objection to that if the vfs developers do not mind.
> 
> We had really bad experiences with fs code in staging.  I think it is
> a bad idea that should not be repeated.

I had no intention to join this topic. However, out of curiousty, I'd like
to hear your opinion about EROFS. I personally think that's not so bad?

EROFS is designed for the specific goal, and it is proven that it has much
better performance than other compress filesystems even uncompressed
filesystems with proper hardware combinations. And we have an active team
(with paid job) and some other companies (primary in Android scope) have
contacted with me in private about using this as well.

We are doing our best efforts on moving out of staging, and I personally
think the code seems not bad... Can you give me some more hints in advance?

Thank you very much,
Gao Xiang

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Hans de Goede]

Hi,

On 8/11/19 2:50 AM, Matthew Wilcox wrote:
> On Sun, Aug 11, 2019 at 12:25:03AM +0200, Hans de Goede wrote:
>> But ATM, since posting v12 of the patch, it has again been quiet for
>> 2 months again. Since this driver is already being used as addon /
>> our of tree driver by various distros, I would really like to get it
>> into mainline, to make live easier for distros and to make sure that
>> they use the latest version.
> 
> fwiw, v12 never made it to the list.  0/1 did, but 1/1 didn't.

Hmm, looks like you are right, weird. I will resend v12 right away.

Regards,

Hans

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Hans de Goede]

Hi,

On 8/11/19 9:50 AM, Greg Kroah-Hartman wrote:
> On Sun, Aug 11, 2019 at 12:43:48AM -0700, Christoph Hellwig wrote:
>> On Sun, Aug 11, 2019 at 09:40:05AM +0200, Greg Kroah-Hartman wrote:
>>>> Since I do not see the lack of reviewing capacity problem get solved
>>>> anytime soon, I was wondering if you are ok with putting the code
>>>> in drivers/staging/vboxsf for now, until someone can review it and ack it
>>>> for moving over to sf/vboxsf ?
>>>
>>> I have no objection to that if the vfs developers do not mind.
>>
>> We had really bad experiences with fs code in staging.  I think it is
>> a bad idea that should not be repeated.
> 
> Lustre was a mistake.  erofs is better in that there are active
> developers working to get it out of staging.  We would also need that
> here for this to be successful.
> 
> Hans, is it just review that is keeping this from being merged or is
> there "real work" that has to be done?

AFAIK it is just the reveiw which is keeping this from being merged.

During the first few revision Al Viro made some very good suggestions
which have all been addressed, v10 was reviewed by David Howell, and the
main thing to fix for that was switching over to the new mountfd APIs,
v11 was also revieded by David and had some minor issues with the use
of the new mountfd APIs. Those were all addressed for v12. So currently
the TODO list for this fs code is empty.

Regards,

Hans

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Greg Kroah-Hartman]

On Sun, Aug 11, 2019 at 03:43:01PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 8/11/19 9:50 AM, Greg Kroah-Hartman wrote:
> > On Sun, Aug 11, 2019 at 12:43:48AM -0700, Christoph Hellwig wrote:
> > > On Sun, Aug 11, 2019 at 09:40:05AM +0200, Greg Kroah-Hartman wrote:
> > > > > Since I do not see the lack of reviewing capacity problem get solved
> > > > > anytime soon, I was wondering if you are ok with putting the code
> > > > > in drivers/staging/vboxsf for now, until someone can review it and ack it
> > > > > for moving over to sf/vboxsf ?
> > > > 
> > > > I have no objection to that if the vfs developers do not mind.
> > > 
> > > We had really bad experiences with fs code in staging.  I think it is
> > > a bad idea that should not be repeated.
> > 
> > Lustre was a mistake.  erofs is better in that there are active
> > developers working to get it out of staging.  We would also need that
> > here for this to be successful.
> > 
> > Hans, is it just review that is keeping this from being merged or is
> > there "real work" that has to be done?
> 
> AFAIK it is just the reveiw which is keeping this from being merged.
> 
> During the first few revision Al Viro made some very good suggestions
> which have all been addressed, v10 was reviewed by David Howell, and the
> main thing to fix for that was switching over to the new mountfd APIs,
> v11 was also revieded by David and had some minor issues with the use
> of the new mountfd APIs. Those were all addressed for v12. So currently
> the TODO list for this fs code is empty.

Then in that case it doesn't sound like putting it in staging makes any
sense.  It should just be merged to the "correct" place right away as
nothing is left to be done on it.

thanks,

greg k-h

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Christoph Hellwig]

On Sun, Aug 11, 2019 at 09:50:42AM +0200, Greg Kroah-Hartman wrote:
> Lustre was a mistake.  erofs is better in that there are active
> developers working to get it out of staging.  We would also need that
> here for this to be successful.

I think erofs could have been handled much easier with a bunch of
iterations of normal submissions.  Bet yes, the biggest problem was
lustre.

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Gao Xiang]

On Mon, Aug 12, 2019 at 04:22:47AM -0700, Christoph Hellwig wrote:
> On Sun, Aug 11, 2019 at 09:50:42AM +0200, Greg Kroah-Hartman wrote:
> > Lustre was a mistake.  erofs is better in that there are active
> > developers working to get it out of staging.  We would also need that
> > here for this to be successful.
> 
> I think erofs could have been handled much easier with a bunch of
> iterations of normal submissions.  Bet yes, the biggest problem was
> lustre.

(I am trying...to add some word on this...)

As a new opensource file system, an unavoidable thing is trying to get
people opinions on this... It could be better to prove that's an interesting /
useful stuff for community so that we can do more actively with fun...
(Hopefully we got some attentions and no silence at first... thanks to
 Richard Weinberger, Pavel Machek, Joey Pabalinas... [1] p.s. I have no
 idea whether Richard still has some interest in it... but EROFS is
 ready now...)

At first, I submitted EROFS to linuxfs mailing list at the very early
stage, although the on-disk format is almost fixed, but I have to admit
that there were still stuffes needing to be done (Note that I updated
LZ4 (lib/lz4) later as well [2].)

we weren't quite sure that such an unclean/incomplete stuff was quite
good for better review at that time, and it was also lack of implementation
of some core runtime concepts such as decompression inplace (which was
only in my mind without real code) But we think this direction is
practical and we want to do iterations by time.

Also, we noticed there are many new stuffs such as fscontext, XArray,
multi-page bvec, which were introduced by time, we hope to keep up
with the mainline kernel...

Staging seems to be such a place for an incomplete but workable stuff
after I noticed what zram did before, therefore I took a try and
thanks Greg as well...

Thanks to merging into staging, many checkpatch/styling/functional
issues have been fixed by contributers... we can also have chance
to test in linux-next for many linux versions... and it's much
easier for us to do latest EROFS backport to 4.19LTS for our products...

On the other hand, since I have a paid job for a commerical company,
I need to apply EROFS unpainfully to our products. And I got many
useful running logs from our internal beta users, which helps us
make EROFS stable... Nowadays, almost all on-service HUAWEI mobile
phones on the market have been integrated with EROFS...

Now EROFS is ready for review, and the main code of EROFS is about
7KLOC... Sadly, it still haven't gotten some explicit external
ACKs till now... we don't know how we should do next to make a
difference... perhaps I thought it is relatively long thus I spilted
into 24 individual patches...

We really hope that it can be merged into fs/ in 5.4 so that we
can improve it even further and gain more users...

Thanks,
Gao Xiang

[1] https://lore.kernel.org/lkml/1670077.cnVahIradn@blindfold/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/lib/lz4

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Hans de Goede]

Hi all,

On 11-08-19 09:43, Christoph Hellwig wrote:
> On Sun, Aug 11, 2019 at 09:40:05AM +0200, Greg Kroah-Hartman wrote:
>>> Since I do not see the lack of reviewing capacity problem get solved
>>> anytime soon, I was wondering if you are ok with putting the code
>>> in drivers/staging/vboxsf for now, until someone can review it and ack it
>>> for moving over to sf/vboxsf ?
>>
>> I have no objection to that if the vfs developers do not mind.
> 
> We had really bad experiences with fs code in staging.  I think it is
> a bad idea that should not be repeated.

So after resolving the problem with the patch not making it through
to the list because of the patch-size, v12 got reviewed (thank you
for that Christoph) and I did a v13.

Then there was some discussion about read cache-coherency vs writes
done on the host side underneath us (the guest) and in the end
Christoph agreed that what was done in v13 was as good as it would
get given the limitations of the shared folder API offered by
the VirtualBox hypervisor, but Christoph did request to add a big
comment explaining these issues in more detail.

So I posted a v14, adding the big comment and addressing 2 very minor
other issue spotted by Christoph, that was 10 days ago and things
have gotten quiet again since. I realiz that 10 days is not very
long but for previous revisions I have been waiting upto 60 days
sometimes and the exfat in staging discussion reminded me of this.

So what is the plan going forward for vboxsf now?

Regards,

Hans

Re: Merging virtualbox shared-folder VFS driver through drivers/staging?

[Christoph Hellwig]

On Thu, Aug 29, 2019 at 05:05:49PM +0200, Hans de Goede wrote:
> So what is the plan going forward for vboxsf now?

ping Al and/or Andrew.