[RFC] dm-lc: plan to go to staging

[RFC] dm-lc: plan to go to staging

[Akira Hayakawa]

Hi,

I am happy to see that dm-lc is picked up by LWN.
http://lwn.net/Articles/562938/

Now I think I should consider the next step forward.


As Joe Thornber said before, what I have to do next are
1. Get some feedback from 3rd party users.
2. Reviewed by Mike Snitzer.
As in
http://www.redhat.com/archives/dm-devel/2013-July/msg00137.html

To achieve the first goal,
I am thinking of going to staging tree
for dm-lc to be widely available to people in community.

dm-lc is now managed in Github repo as a kernel module
however, getting available in the tree means a lot
to get 3rd party users.

Does this plan sound nice?

If so, what is the next step to go to staging?
I have a patch against 3.11-rc7 in linux-next locally.
Sending the patch to Greg is the next step?
And who and what should be included in CC?

For your information,
there is a precedent that DM target
gets in staging tree. That is enhanceio, also a caching target.

Any comments will be appreciated.

Thanks,
Akira

Re: [RFC] dm-lc: plan to go to staging

[Greg KH]

On Thu, Aug 29, 2013 at 10:35:35AM +0900, Akira Hayakawa wrote:
> Hi,
> 
> I am happy to see that dm-lc is picked up by LWN.
> http://lwn.net/Articles/562938/
> 
> Now I think I should consider the next step forward.
> 
> 
> As Joe Thornber said before, what I have to do next are
> 1. Get some feedback from 3rd party users.
> 2. Reviewed by Mike Snitzer.
> As in
> http://www.redhat.com/archives/dm-devel/2013-July/msg00137.html
> 
> To achieve the first goal,
> I am thinking of going to staging tree
> for dm-lc to be widely available to people in community.
> 
> dm-lc is now managed in Github repo as a kernel module
> however, getting available in the tree means a lot
> to get 3rd party users.
> 
> Does this plan sound nice?
> 
> If so, what is the next step to go to staging?
> I have a patch against 3.11-rc7 in linux-next locally.
> Sending the patch to Greg is the next step?
> And who and what should be included in CC?

That's fine with me.  For staging drivers, I need a TODO file that lists
what needs to be done to the code to get it into a mergable state for
the "real" part of the kernel, and an email address to cc: patches on.
Also, a MAINTAINERS entry is good as that way the
scripts/get_maintainer.pl tool will properly pick up your email address.

You should generate the patch against the linux-next tree, as I have
lots of new staging drivers in there, ready for 3.12, and if you make a
patch against 3.11-rc7, odds are it will not apply properly.

If you have any other questions, please let me know.

thanks,

greg k-h

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Alasdair G Kergon]

On Wed, Aug 28, 2013 at 07:05:55PM -0700, Greg KH wrote:
> For staging drivers, I need a TODO file that lists
> what needs to be done to the code to get it into a mergable state for
> the "real" part of the kernel,

Two simple requirements before putting your proof-of-concept into staging
if you want to work that way:

1) Drop the major version number to 0.  Version 1 is reserved for
supported modules.

2) Agree a new and meaningful target name with us so you don't have to
change it later.  "lc" means nothing, I'm afraid.

Then in general terms, you should continue to compare your device-mapper
target with the existing targets and where there are differences, either
change your target to be like something that already exists, or be ready
to explain why that can't or shouldn't be done.

In particular, the interface and architecture will need substantial
changes and working these out should be your highest priority.

For example, you write:

  Be careful, you MUST create all the LVs as the destinations of
  the dirty blocks on the cache device before this operation.  Otherwise,
  the kernel may crash.

I read a statement like that as an indication of an interface or
architectural problem.  The device-mapper approach is to 'design out'
problems, rather than relying on users not doing bad things.
Study the existing interfaces used by other targets to understand
some approaches that proved successful, then decide which ones
come closest to your needs.

(Your code also needs many more comments inline to explain what it does
and how it works.)

The documentation file will eventually need rewriting to follow the same
format as the other targets recently added to the kernel.  We document
the kernel interface rather than any particular userspace tools, which
just have the status of convenient examples.

Another little thing I noticed: look into using something like
__dm_bless_for_disk() for your metadata and clearly segregate your
on-disk structures and document the layout.

Alasdair

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Mikulas Patocka]

Another idea:

Make the interface of dm-lc (the arguments to constructor, messages and 
the status line) the same as dm-cache, so that they can be driven by the 
same userspace code.

Mikulas


On Thu, 29 Aug 2013, Alasdair G Kergon wrote:

> On Wed, Aug 28, 2013 at 07:05:55PM -0700, Greg KH wrote:
> > For staging drivers, I need a TODO file that lists
> > what needs to be done to the code to get it into a mergable state for
> > the "real" part of the kernel,
> 
> Two simple requirements before putting your proof-of-concept into staging
> if you want to work that way:
> 
> 1) Drop the major version number to 0.  Version 1 is reserved for
> supported modules.
> 
> 2) Agree a new and meaningful target name with us so you don't have to
> change it later.  "lc" means nothing, I'm afraid.
> 
> Then in general terms, you should continue to compare your device-mapper
> target with the existing targets and where there are differences, either
> change your target to be like something that already exists, or be ready
> to explain why that can't or shouldn't be done.
> 
> In particular, the interface and architecture will need substantial
> changes and working these out should be your highest priority.
> 
> For example, you write:
> 
>   Be careful, you MUST create all the LVs as the destinations of
>   the dirty blocks on the cache device before this operation.  Otherwise,
>   the kernel may crash.
> 
> I read a statement like that as an indication of an interface or
> architectural problem.  The device-mapper approach is to 'design out'
> problems, rather than relying on users not doing bad things.
> Study the existing interfaces used by other targets to understand
> some approaches that proved successful, then decide which ones
> come closest to your needs.
> 
> (Your code also needs many more comments inline to explain what it does
> and how it works.)
> 
> The documentation file will eventually need rewriting to follow the same
> format as the other targets recently added to the kernel.  We document
> the kernel interface rather than any particular userspace tools, which
> just have the status of convenient examples.
> 
> Another little thing I noticed: look into using something like
> __dm_bless_for_disk() for your metadata and clearly segregate your
> on-disk structures and document the layout.
> 
> Alasdair
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Akira Hayakawa]

Thank you for your constructive comments,

I want to first settle an agreement on the new name for current dm-lc
according to Alasdair's comment:
>> 2) Agree a new and meaningful target name with us so you don't have to
>> change it later.  "lc" means nothing, I'm afraid.

After we agree on this issue, I will start tackling on the design issues.
Choosing this issue first is because it may change the whole codebase
and rename the Github repo which can be one of the elements in MAINTAINERS file.

Name is very important.

First of all, let me make a tiny objection:
1) The name dm-lc does means. lc is the initials from Log-structured Caching.
2) LC is also the name of my a favorite anime character from "God only knows".
   I did want to use her name for my creation.

But,
I can agree on Alasdair's opinion that dm-lc means nothing,
for the potential future users.
Abbreviation of some technical word to a software name looks nice
but the unfortunate is that LC is not widely common like RAID or LFS.

I have three candidates on my mind.
I am thinking of removing the dm- prefix as bcache and enhanceio do.

The Points are:
1) The name should not be too long.
   A name explaining too much about the software is noisy.
2) Making clear the characteristics and the difference from other targets.
   The characteristics to note are:
   a) It writes in-coming writes in log-structured manner.
   b) It is extremely fast in write.
   c) It is caching software.

Candidates:
1) logcache
Not too long but explaining enough
but looks little bit dull to me.
Futhermore, there still is dm-log
and I feel like duplicating a little bit.

2) lscache
lscache sounds cool to me.
But the problem is that there is a preceding one with the same name
as to caching web-server workload.
Unique name makes it easy to search for a web documents.

3) writeboost or wboost
Concentrating on the characteristic b)
but doesn't mention cache.

Up to this point,
which do you think the best for the new name?
or other candidate?

I suppose logcache is an acceptable compromise.

Thanks for reading,
Akira

On 8/29/13 11:45 PM, Mikulas Patocka wrote:
> Another idea:
> 
> Make the interface of dm-lc (the arguments to constructor, messages and 
> the status line) the same as dm-cache, so that they can be driven by the 
> same userspace code.
> 
> Mikulas
> 
> 
> On Thu, 29 Aug 2013, Alasdair G Kergon wrote:
> 
>> On Wed, Aug 28, 2013 at 07:05:55PM -0700, Greg KH wrote:
>>> For staging drivers, I need a TODO file that lists
>>> what needs to be done to the code to get it into a mergable state for
>>> the "real" part of the kernel,
>>
>> Two simple requirements before putting your proof-of-concept into staging
>> if you want to work that way:
>>
>> 1) Drop the major version number to 0.  Version 1 is reserved for
>> supported modules.
>>
>> 2) Agree a new and meaningful target name with us so you don't have to
>> change it later.  "lc" means nothing, I'm afraid.
>>
>> Then in general terms, you should continue to compare your device-mapper
>> target with the existing targets and where there are differences, either
>> change your target to be like something that already exists, or be ready
>> to explain why that can't or shouldn't be done.
>>
>> In particular, the interface and architecture will need substantial
>> changes and working these out should be your highest priority.
>>
>> For example, you write:
>>
>>   Be careful, you MUST create all the LVs as the destinations of
>>   the dirty blocks on the cache device before this operation.  Otherwise,
>>   the kernel may crash.
>>
>> I read a statement like that as an indication of an interface or
>> architectural problem.  The device-mapper approach is to 'design out'
>> problems, rather than relying on users not doing bad things.
>> Study the existing interfaces used by other targets to understand
>> some approaches that proved successful, then decide which ones
>> come closest to your needs.
>>
>> (Your code also needs many more comments inline to explain what it does
>> and how it works.)
>>
>> The documentation file will eventually need rewriting to follow the same
>> format as the other targets recently added to the kernel.  We document
>> the kernel interface rather than any particular userspace tools, which
>> just have the status of convenient examples.
>>
>> Another little thing I noticed: look into using something like
>> __dm_bless_for_disk() for your metadata and clearly segregate your
>> on-disk structures and document the layout.
>>
>> Alasdair
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Alasdair G Kergon]

On Fri, Aug 30, 2013 at 09:33:29PM +0900, Akira Hayakawa wrote:
> I have three candidates on my mind.
> I am thinking of removing the dm- prefix as bcache and enhanceio do.
 
Yes, that's fine for the general way of referring to this.
(Within the codebase, the filename and module name would still need the
prefix: we use this for auto-module loading, function name/error message
scoping etc.)

>    a) It writes in-coming writes in log-structured manner.
>    b) It is extremely fast in write.
>    c) It is caching software.

Another candidate: 'lcache' - contrast with 'bcache
 
> 3) writeboost or wboost
> Concentrating on the characteristic b)
> but doesn't mention cache.

I quite like that option.

Alasdair

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Akira Hayakawa]

> (Within the codebase, the filename and module name would still need the
> prefix: we use this for auto-module loading, function name/error message
> scoping etc.)

>> 3) writeboost or wboost
>> Concentrating on the characteristic b)
>> but doesn't mention cache.
> 
> I quite like that option.

OK,
I would like to rename from dm-lc to dm-writeboost.
Do you agree?

Akira

On 8/30/13 9:50 PM, Alasdair G Kergon wrote:
> On Fri, Aug 30, 2013 at 09:33:29PM +0900, Akira Hayakawa wrote:
>> I have three candidates on my mind.
>> I am thinking of removing the dm- prefix as bcache and enhanceio do.
>  
> Yes, that's fine for the general way of referring to this.
> (Within the codebase, the filename and module name would still need the
> prefix: we use this for auto-module loading, function name/error message
> scoping etc.)
> 
>>    a) It writes in-coming writes in log-structured manner.
>>    b) It is extremely fast in write.
>>    c) It is caching software.
> 
> Another candidate: 'lcache' - contrast with 'bcache
>  
>> 3) writeboost or wboost
>> Concentrating on the characteristic b)
>> but doesn't mention cache.
> 
> I quite like that option.
> 
> Alasdair
> 

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

On Fri, 30 Aug 2013 21:33:29 +0900, Akira Hayakawa said:

> Candidates:
> 1) logcache
> Not too long but explaining enough
> but looks little bit dull to me.

Descriptive but dull naming is a much underrated quality in software packages.

Consider the package HarfBuzz.

As my daughter put it, "Sounds like a bad coffee name".

rpm -qi harfbuzz says:
Description :
HarfBuzz is an implementation of the OpenType Layout engine.

(And don't get me started on the people who name packages enca, fros, serd,
yajl, or yaml.  We aren't using ASR-33's anymore, brevity isn't as crucial :)

dm-logcache works just fine for me as a name for this.  Or dm-lcache.



[-- Attachment #2: Type: application/pgp-signature, Size: 865 bytes --]

[dm-devel] [RFC] dm-lc: plan to go to staging

[Valdis.Kletnieks at vt.edu]

On Fri, 30 Aug 2013 21:33:29 +0900, Akira Hayakawa said:

> Candidates:
> 1) logcache
> Not too long but explaining enough
> but looks little bit dull to me.

Descriptive but dull naming is a much underrated quality in software packages.

Consider the package HarfBuzz.

As my daughter put it, "Sounds like a bad coffee name".

rpm -qi harfbuzz says:
Description :
HarfBuzz is an implementation of the OpenType Layout engine.

(And don't get me started on the people who name packages enca, fros, serd,
yajl, or yaml.  We aren't using ASR-33's anymore, brevity isn't as crucial :)

dm-logcache works just fine for me as a name for this.  Or dm-lcache.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 865 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130830/18db74b6/attachment.bin 

Re: [dm-devel] [RFC] dm-writeboost: plan to go to staging

[Akira Hayakawa]

Hi.

Progress report for staging dm-writeboost.

(1) Renaming to dm-writeboost
Thanks for comments on renaming.

By a simple democratic,
dm-lc will be renamed to dm-writeboost.
I am really happy with the new name.

(2) Idea for smelling limitation on the kind of backing store
Now I am tackling on the design issues.
As Alasdair pointed out
the statement below from my document really smells
and seems to be the first priority at this point.

  Be careful, you MUST create all the LVs as the destinations of
  the dirty blocks on the cache device before this operation.  Otherwise,
  the kernel may crash.

Aside from the two small requirements by Alasdair,
I may have to shoot out this problem before sending patch to staging tree.
> Two simple requirements before putting your proof-of-concept into staging
> if you want to work that way:
> 
> 1) Drop the major version number to 0.  Version 1 is reserved for
> supported modules.
> 
> 2) Agree a new and meaningful target name with us so you don't have to
> change it later.  "lc" means nothing, I'm afraid.

My investigation today revealed that this statement is too pessimistic.
The truth is only that
the backing store should not be some partition out of a disk,
say /dev/sdd1 out of /dev/sdd.

With the partition number,
my userland script works badly in attaching a device to a cache.
It failed to get the correct device name from sysfs.

I am thinking of addressing this problem by
adding new sysfs member to show device number(major:minor) of the backing store
which is likely to maintain backward-compatibility and
doing some modification on the Python scripts.

That's very easy.
I will do this before sending a patch for staging
aside said two requirements.

(3) Draft for TODO and MAINTAINERS files
These are the current drafts for TODO and MAINTAINERS files.
Please let me know if you find more to write or simple mistakes.

(3-1) TODO file
TODO:
    - Get feedback from 3rd party users.
    - Reviewed by Mike Snitzer.
    - Audit userspace interfaces to make sure they are sane.
      Should use the same approach that is proven successful.
    - Fix document.
      Should document the kernel interfaces
      rather than particular userspace tools.
    - Add more comments inline
      to explain what it does and how it works.

Please send patches to Greg Kroah-Hartman <gregkh@linuxfoundation.org> and Cc:
Akira Hayakawa <ruby.wktk@gmail.com>

(3-2) part of MAINTAINERS file
M:      Greg Kroah-Hartman <gregkh@linuxfoundation.org>
M:      Akira Hayakawa <ruby.wktk@gmail.com>
S:      Maintained
L:      dm-devel@redhat.com
W:      https://github.com/akiradeveloper/dm-writeboost.git
F:      drivers/staging/dm-writeboost.git

(4) Next steps
- Fixed all source codes and documents as they are of
  dm-writeboost but not dm-lc.
- Github repo will be renamed.
- Complete the task said at (2).
- Generating a patch against linux-next
  and send it to Greg.

Thanks for your reading,
Akira

Re: [dm-devel] [RFC] dm-writeboost: plan to go to staging

[Akira Hayakawa]

Thanks, Alasdair.

I will reply to some of your comments
that is not answered up to now.

> The documentation file will eventually need rewriting to follow the same
> format as the other targets recently added to the kernel.  We document
> the kernel interface rather than any particular userspace tools, which
> just have the status of convenient examples.
dm-writeboost can be used by kicking kernel interfaces but
it is not recommended.
My userspace tools take care of all the
implications in using the kernel interfaces.
The fact dm-writeboost supports shared-caching and
write-back caching with auto-modulated migration
puts the interfaces and implementations under some constraints.
That's simply a tradeoff.

OK, I will document the kernel interfaces
and also mention the userspace tools.

> (Your code also needs many more comments inline to explain what it does
> and how it works.)
Comments are substantially noise is my philosophy.
I will add comments with given feedback clearly mentioning
what is clear and what is not clear about the source code.
Even in that case,
I'd better polish the code rather than write comments.

> Another little thing I noticed: look into using something like
> __dm_bless_for_disk() for your metadata and clearly segregate your
> on-disk structures and document the layout.
I couldn't understand what is __dm_bless_for_disk() for.
What does the word "bless" mean in this context?

By the way,
dm-cache and dm-writeboost are definitely different in a way that
dm-cache has an another disk for metadata while dm-writeboost doesn't.
dm-writeboost packs metadata and data within a log and
submit it to the cache device like log-structured filesystem.

So, the annotation checks can not be appropriated
to apply to dm-writeboost.

And what about moving these macros to
include/linux/device-mapper.h?

Akira

On 8/29/13 12:30 PM, Alasdair G Kergon wrote:
> On Wed, Aug 28, 2013 at 07:05:55PM -0700, Greg KH wrote:
>> For staging drivers, I need a TODO file that lists
>> what needs to be done to the code to get it into a mergable state for
>> the "real" part of the kernel,
> 
> Two simple requirements before putting your proof-of-concept into staging
> if you want to work that way:
> 
> 1) Drop the major version number to 0.  Version 1 is reserved for
> supported modules.
> 
> 2) Agree a new and meaningful target name with us so you don't have to
> change it later.  "lc" means nothing, I'm afraid.
> 
> Then in general terms, you should continue to compare your device-mapper
> target with the existing targets and where there are differences, either
> change your target to be like something that already exists, or be ready
> to explain why that can't or shouldn't be done.
> 
> In particular, the interface and architecture will need substantial
> changes and working these out should be your highest priority.
> 
> For example, you write:
> 
>   Be careful, you MUST create all the LVs as the destinations of
>   the dirty blocks on the cache device before this operation.  Otherwise,
>   the kernel may crash.
> 
> I read a statement like that as an indication of an interface or
> architectural problem.  The device-mapper approach is to 'design out'
> problems, rather than relying on users not doing bad things.
> Study the existing interfaces used by other targets to understand
> some approaches that proved successful, then decide which ones
> come closest to your needs.
> 
> (Your code also needs many more comments inline to explain what it does
> and how it works.)
> 
> The documentation file will eventually need rewriting to follow the same
> format as the other targets recently added to the kernel.  We document
> the kernel interface rather than any particular userspace tools, which
> just have the status of convenient examples.
> 
> Another little thing I noticed: look into using something like
> __dm_bless_for_disk() for your metadata and clearly segregate your
> on-disk structures and document the layout.
> 
> Alasdair
>