Re: [LSF/MM TOPIC] mm documentation

Re: [LSF/MM TOPIC] mm documentation

[Mike Rapoport]

(forgot to CC linux-mm)

On Tue, Jan 30, 2018 at 12:52:37PM +0200, Mike Rapoport wrote:
> Hello,
> 
> The mm kernel-doc documentation is not in a great shape. 
> 
> Some of the existing kernel-doc annotations were not reformatted during
> transition from dockbook to sphix. Sometimes the parameter descriptions
> do not match actual code. But aside these rather mechanical issues there
> are several points it'd like to discuss:
> 
> * Currently, only 14 files are linked to kernel-api.rst under "Memory
> Management in Linux" section. We have more than hundred files only in mm.
> Even the existing documentation is not generated when running "make
> htmldocs"
> * Do we want to keep "Memory Management in Linux" under kernel-api.rst or
> maybe it's worth adding, say, mm.rst?
> * What is the desired layout of the documentation, what sections we'd like
> to have, how the documentation should be ordered?
> 
> -- 
> Sincerely yours,
> Mike.

-- 
Sincerely yours,
Mike.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Michal Hocko]

On Tue 30-01-18 12:54:50, Mike Rapoport wrote:
> (forgot to CC linux-mm)
> 
> On Tue, Jan 30, 2018 at 12:52:37PM +0200, Mike Rapoport wrote:
> > Hello,
> > 
> > The mm kernel-doc documentation is not in a great shape. 
> > 
> > Some of the existing kernel-doc annotations were not reformatted during
> > transition from dockbook to sphix. Sometimes the parameter descriptions
> > do not match actual code. But aside these rather mechanical issues there
> > are several points it'd like to discuss:
> > 
> > * Currently, only 14 files are linked to kernel-api.rst under "Memory
> > Management in Linux" section. We have more than hundred files only in mm.
> > Even the existing documentation is not generated when running "make
> > htmldocs"

Is this documentation anywhere close to be actually useful?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Mike Rapoport]

On Tue, Jan 30, 2018 at 12:50:55PM +0100, Michal Hocko wrote:
> On Tue 30-01-18 12:54:50, Mike Rapoport wrote:
> > (forgot to CC linux-mm)
> > 
> > On Tue, Jan 30, 2018 at 12:52:37PM +0200, Mike Rapoport wrote:
> > > Hello,
> > > 
> > > The mm kernel-doc documentation is not in a great shape. 
> > > 
> > > Some of the existing kernel-doc annotations were not reformatted during
> > > transition from dockbook to sphix. Sometimes the parameter descriptions
> > > do not match actual code. But aside these rather mechanical issues there
> > > are several points it'd like to discuss:
> > > 
> > > * Currently, only 14 files are linked to kernel-api.rst under "Memory
> > > Management in Linux" section. We have more than hundred files only in mm.
> > > Even the existing documentation is not generated when running "make
> > > htmldocs"
> 
> Is this documentation anywhere close to be actually useful?

Some parts are documented better, some worse. For instance, bootmem and
z3fold are covered not bad at all, but, say, huge_memory has no structured
comments at all. Roughly half of the files in mm/ have some documentation,
but I didn't yet read that all to say how much of it is actually useful.

And maybe having some skeleton for MM API in htmldocs with at least some
documentation will encourage people to look at the documentation.

> -- 
> Michal Hocko
> SUSE Labs
> 

-- 
Sincerely yours,
Mike.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Michal Hocko]

On Tue 30-01-18 14:54:44, Mike Rapoport wrote:
> On Tue, Jan 30, 2018 at 12:50:55PM +0100, Michal Hocko wrote:
> > On Tue 30-01-18 12:54:50, Mike Rapoport wrote:
> > > (forgot to CC linux-mm)
> > > 
> > > On Tue, Jan 30, 2018 at 12:52:37PM +0200, Mike Rapoport wrote:
> > > > Hello,
> > > > 
> > > > The mm kernel-doc documentation is not in a great shape. 
> > > > 
> > > > Some of the existing kernel-doc annotations were not reformatted during
> > > > transition from dockbook to sphix. Sometimes the parameter descriptions
> > > > do not match actual code. But aside these rather mechanical issues there
> > > > are several points it'd like to discuss:
> > > > 
> > > > * Currently, only 14 files are linked to kernel-api.rst under "Memory
> > > > Management in Linux" section. We have more than hundred files only in mm.
> > > > Even the existing documentation is not generated when running "make
> > > > htmldocs"
> > 
> > Is this documentation anywhere close to be actually useful?
> 
> Some parts are documented better, some worse. For instance, bootmem and
> z3fold are covered not bad at all, but, say, huge_memory has no structured
> comments at all. Roughly half of the files in mm/ have some documentation,
> but I didn't yet read that all to say how much of it is actually useful.

It is good to hear that at least something has a documentation coverage.
I was asking mostly because I _think_ that the API documentation is far
from the top priority. We are seriously lacking any highlevel one which
describes the design and subsytems interaction. Well, we have missed
that train years ago. It will be really hard to catch up.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Mike Rapoport]

On Tue, Jan 30, 2018 at 02:41:41PM +0100, Michal Hocko wrote:
> On Tue 30-01-18 14:54:44, Mike Rapoport wrote:
> > On Tue, Jan 30, 2018 at 12:50:55PM +0100, Michal Hocko wrote:
> > > On Tue 30-01-18 12:54:50, Mike Rapoport wrote:
> > > > (forgot to CC linux-mm)
> > > > 
> > > > On Tue, Jan 30, 2018 at 12:52:37PM +0200, Mike Rapoport wrote:
> > > > > Hello,
> > > > > 
> > > > > The mm kernel-doc documentation is not in a great shape. 
> > > > > 
> > > > > Some of the existing kernel-doc annotations were not reformatted during
> > > > > transition from dockbook to sphix. Sometimes the parameter descriptions
> > > > > do not match actual code. But aside these rather mechanical issues there
> > > > > are several points it'd like to discuss:
> > > > > 
> > > > > * Currently, only 14 files are linked to kernel-api.rst under "Memory
> > > > > Management in Linux" section. We have more than hundred files only in mm.
> > > > > Even the existing documentation is not generated when running "make
> > > > > htmldocs"
> > > 
> > > Is this documentation anywhere close to be actually useful?
> > 
> > Some parts are documented better, some worse. For instance, bootmem and
> > z3fold are covered not bad at all, but, say, huge_memory has no structured
> > comments at all. Roughly half of the files in mm/ have some documentation,
> > but I didn't yet read that all to say how much of it is actually useful.
> 
> It is good to hear that at least something has a documentation coverage.
> I was asking mostly because I _think_ that the API documentation is far
> from the top priority. 

API documentations is important for kernel developers who are not deeply
involved with mm. When one develops a device driver, knowing how to
allocate and free memory is essential. And, while *malloc are included in
kernel-api.rst, CMA and HMM documentation is not visible.

> We are seriously lacking any highlevel one which describes the design and
> subsytems interaction.

I should have describe it better, but by "creating a new structure for mm
documentation" I've also meant adding high level description.

> Well, we have missed that train years ago. It will be really hard to catch up.

At least we can try.

> -- 
> Michal Hocko
> SUSE Labs
> 

-- 
Sincerely yours,
Mike.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Randy Dunlap]

On 01/30/2018 06:28 AM, Mike Rapoport wrote:
> On Tue, Jan 30, 2018 at 02:41:41PM +0100, Michal Hocko wrote:
>> On Tue 30-01-18 14:54:44, Mike Rapoport wrote:
>>> On Tue, Jan 30, 2018 at 12:50:55PM +0100, Michal Hocko wrote:
>>>> On Tue 30-01-18 12:54:50, Mike Rapoport wrote:
>>>>> (forgot to CC linux-mm)
>>>>>
>>>>> On Tue, Jan 30, 2018 at 12:52:37PM +0200, Mike Rapoport wrote:
>>>>>> Hello,
>>>>>>
>>>>>> The mm kernel-doc documentation is not in a great shape. 
>>>>>>
>>>>>> Some of the existing kernel-doc annotations were not reformatted during
>>>>>> transition from dockbook to sphix. Sometimes the parameter descriptions
>>>>>> do not match actual code. But aside these rather mechanical issues there
>>>>>> are several points it'd like to discuss:
>>>>>>
>>>>>> * Currently, only 14 files are linked to kernel-api.rst under "Memory
>>>>>> Management in Linux" section. We have more than hundred files only in mm.
>>>>>> Even the existing documentation is not generated when running "make
>>>>>> htmldocs"
>>>>
>>>> Is this documentation anywhere close to be actually useful?
>>>
>>> Some parts are documented better, some worse. For instance, bootmem and
>>> z3fold are covered not bad at all, but, say, huge_memory has no structured
>>> comments at all. Roughly half of the files in mm/ have some documentation,
>>> but I didn't yet read that all to say how much of it is actually useful.
>>
>> It is good to hear that at least something has a documentation coverage.
>> I was asking mostly because I _think_ that the API documentation is far
>> from the top priority. 
> 
> API documentations is important for kernel developers who are not deeply
> involved with mm. When one develops a device driver, knowing how to
> allocate and free memory is essential. And, while *malloc are included in
> kernel-api.rst, CMA and HMM documentation is not visible.
> 
>> We are seriously lacking any highlevel one which describes the design and
>> subsytems interaction.
> 
> I should have describe it better, but by "creating a new structure for mm
> documentation" I've also meant adding high level description.
> 
>> Well, we have missed that train years ago. It will be really hard to catch up.
> 
> At least we can try.

Hi,
I would move it all to a new mm.rst file.  That would be easier to maintain
and also allow parallel building.

-- 
~Randy

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Mike Rapoport]

On Tue, Jan 30, 2018 at 09:32:04AM -0800, Randy Dunlap wrote:
> On 01/30/2018 06:28 AM, Mike Rapoport wrote:
> > On Tue, Jan 30, 2018 at 02:41:41PM +0100, Michal Hocko wrote:
> >> On Tue 30-01-18 14:54:44, Mike Rapoport wrote:
> >>> On Tue, Jan 30, 2018 at 12:50:55PM +0100, Michal Hocko wrote:
> >>>> On Tue 30-01-18 12:54:50, Mike Rapoport wrote:
> >>>>> (forgot to CC linux-mm)
> >>>>>
> >>>>> On Tue, Jan 30, 2018 at 12:52:37PM +0200, Mike Rapoport wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> The mm kernel-doc documentation is not in a great shape. 
> >>>>>>
> >>>>>> Some of the existing kernel-doc annotations were not reformatted during
> >>>>>> transition from dockbook to sphix. Sometimes the parameter descriptions
> >>>>>> do not match actual code. But aside these rather mechanical issues there
> >>>>>> are several points it'd like to discuss:
> >>>>>>
> >>>>>> * Currently, only 14 files are linked to kernel-api.rst under "Memory
> >>>>>> Management in Linux" section. We have more than hundred files only in mm.
> >>>>>> Even the existing documentation is not generated when running "make
> >>>>>> htmldocs"
> >>>>
> >>>> Is this documentation anywhere close to be actually useful?
> >>>
> >>> Some parts are documented better, some worse. For instance, bootmem and
> >>> z3fold are covered not bad at all, but, say, huge_memory has no structured
> >>> comments at all. Roughly half of the files in mm/ have some documentation,
> >>> but I didn't yet read that all to say how much of it is actually useful.
> >>
> >> It is good to hear that at least something has a documentation coverage.
> >> I was asking mostly because I _think_ that the API documentation is far
> >> from the top priority. 
> > 
> > API documentations is important for kernel developers who are not deeply
> > involved with mm. When one develops a device driver, knowing how to
> > allocate and free memory is essential. And, while *malloc are included in
> > kernel-api.rst, CMA and HMM documentation is not visible.
> > 
> >> We are seriously lacking any highlevel one which describes the design and
> >> subsytems interaction.
> > 
> > I should have describe it better, but by "creating a new structure for mm
> > documentation" I've also meant adding high level description.
> > 
> >> Well, we have missed that train years ago. It will be really hard to catch up.
> > 
> > At least we can try.
> 
> Hi,
> I would move it all to a new mm.rst file.  That would be easier to maintain
> and also allow parallel building.

Agree. Could also be Documentation/vm/index.rst.
 
> -- 
> ~Randy
> 

-- 
Sincerely yours,
Mike.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[James Bottomley]

On Tue, 2018-01-30 at 16:28 +0200, Mike Rapoport wrote:
> On Tue, Jan 30, 2018 at 02:41:41PM +0100, Michal Hocko wrote:
> > 
> > On Tue 30-01-18 14:54:44, Mike Rapoport wrote:
> > > 
> > > On Tue, Jan 30, 2018 at 12:50:55PM +0100, Michal Hocko wrote:
> > > > 
> > > > On Tue 30-01-18 12:54:50, Mike Rapoport wrote:
> > > > > 
> > > > > (forgot to CC linux-mm)
> > > > > 
> > > > > On Tue, Jan 30, 2018 at 12:52:37PM +0200, Mike Rapoport
> > > > > wrote:
> > > > > > 
> > > > > > Hello,
> > > > > > 
> > > > > > The mm kernel-doc documentation is not in a great shape.A 
> > > > > > 
> > > > > > Some of the existing kernel-doc annotations were not
> > > > > > reformatted during transition from dockbook to sphix.
> > > > > > Sometimes the parameter descriptions do not match actual
> > > > > > code. But aside these rather mechanical issues there
> > > > > > are several points it'd like to discuss:
> > > > > > 
> > > > > > * Currently, only 14 files are linked to kernel-api.rst
> > > > > > under "Memory Management in Linux" section. We have more
> > > > > > than hundred files only in mm. Even the existing
> > > > > > documentation is not generated when running "make
> > > > > > htmldocs"
> > > > 
> > > > Is this documentation anywhere close to be actually useful?
> > > 
> > > Some parts are documented better, some worse. For instance,
> > > bootmem and z3fold are covered not bad at all, but, say,
> > > huge_memory has no structured comments at all. Roughly half of
> > > the files in mm/ have some documentation, but I didn't yet read
> > > that all to say how much of it is actually useful.
> > 
> > It is good to hear that at least something has a documentation
> > coverage. I was asking mostly because I _think_ that the API
> > documentation is far from the top priority.A 
> 
> API documentations is important for kernel developers who are not
> deeply involved with mm. When one develops a device driver, knowing
> how to allocate and free memory is essential. And, while *malloc are
> included in kernel-api.rst, CMA and HMM documentation is not visible.

Documentation is also one way new people get into the project.A A Not
being top priority is fine, but "far from" top priority implies not
worth doing, which gives the wrong impression.

> > We are seriously lacking any highlevel one which describes the
> > design and subsytems interaction.
> 
> I should have describe it better, but by "creating a new structure
> for mm documentation" I've also meant adding high level description.
> 
> > 
> > Well, we have missed that train years ago. It will be really hard
> > to catch up.
> 
> At least we can try.

How about simply insisting on adequately documenting new stuff and
asking submitters to add documentation when they change something.A A The
latter, at least, is fairly essential: there's nothing worse than
documentation that's actively wrong. A The former is useful to
reviewers. A I'm not saying this alone will ever get you to 100%, but at
least it's an incremental change which isn't too burdensome and which
moves the needle in the right direction.

James

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Matthew Wilcox]

On Tue, Jan 30, 2018 at 04:28:50PM +0200, Mike Rapoport wrote:
> On Tue, Jan 30, 2018 at 02:41:41PM +0100, Michal Hocko wrote:
> > It is good to hear that at least something has a documentation coverage.
> > I was asking mostly because I _think_ that the API documentation is far
> > from the top priority. 
> 
> API documentations is important for kernel developers who are not deeply
> involved with mm. When one develops a device driver, knowing how to
> allocate and free memory is essential. And, while *malloc are included in
> kernel-api.rst, CMA and HMM documentation is not visible.
> 
> > We are seriously lacking any highlevel one which describes the design and
> > subsytems interaction.
> 
> I should have describe it better, but by "creating a new structure for mm
> documentation" I've also meant adding high level description.

We should be really clear what kind of documentation we're trying to create.

There are four distinct types of documentation which would be useful:

 - How, when and why to use the various function calls and their
   parameters from the perspective of a user outside the mm/ hierarchy.
   Device driver authors, filesystem authors and others of their ilk.
 - The overall philosophy and structure of the mm directory, what it does,
   why it does it, perhaps even outlines of abandoned approaches.
 - What functionality the mm subsystem requires from others.  For example,
   what does the mm rely on from the CPU architectures (and maybe it would
   make sense to also include services the mm layer provides to arches in
   this section, like setting up sparsemem).
 - How to tweak the various knobs that the mm subsystem provides.
   Maybe this is all adequately documented elsewhere already.

Perhaps others can think of other types of documentation which would
be useful.  That shouldn't detract from my main point, which is that
saying "Now we have mm documentation" is laudable, but not enough.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Michal Hocko]

On Tue 30-01-18 18:38:38, Matthew Wilcox wrote:
> On Tue, Jan 30, 2018 at 04:28:50PM +0200, Mike Rapoport wrote:
> > On Tue, Jan 30, 2018 at 02:41:41PM +0100, Michal Hocko wrote:
> > > It is good to hear that at least something has a documentation coverage.
> > > I was asking mostly because I _think_ that the API documentation is far
> > > from the top priority. 
> > 
> > API documentations is important for kernel developers who are not deeply
> > involved with mm. When one develops a device driver, knowing how to
> > allocate and free memory is essential. And, while *malloc are included in
> > kernel-api.rst, CMA and HMM documentation is not visible.
> > 
> > > We are seriously lacking any highlevel one which describes the design and
> > > subsytems interaction.
> > 
> > I should have describe it better, but by "creating a new structure for mm
> > documentation" I've also meant adding high level description.
> 
> We should be really clear what kind of documentation we're trying to create.
> 
> There are four distinct types of documentation which would be useful:
> 
>  - How, when and why to use the various function calls and their
>    parameters from the perspective of a user outside the mm/ hierarchy.
>    Device driver authors, filesystem authors and others of their ilk.
>  - The overall philosophy and structure of the mm directory, what it does,
>    why it does it, perhaps even outlines of abandoned approaches.
>  - What functionality the mm subsystem requires from others.  For example,
>    what does the mm rely on from the CPU architectures (and maybe it would
>    make sense to also include services the mm layer provides to arches in
>    this section, like setting up sparsemem).

yes

>  - How to tweak the various knobs that the mm subsystem provides.
>    Maybe this is all adequately documented elsewhere already.

This would be Documentation/sysctl/vm.txt which is one that is at least
close to be complete.
 
> Perhaps others can think of other types of documentation which would
> be useful.

- design documentation of various parts of the MM - reclaim, memory
  hotplug, memcg, page allocator, memory models, THP, rmap code (you
  name it)

> That shouldn't detract from my main point, which is that
> saying "Now we have mm documentation" is laudable, but not enough.

Absolutely agreed.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] mm documentation

[Mike Rapoport]

On Wed, Jan 31, 2018 at 10:00:37AM +0100, Michal Hocko wrote:
> On Tue 30-01-18 18:38:38, Matthew Wilcox wrote:
> > On Tue, Jan 30, 2018 at 04:28:50PM +0200, Mike Rapoport wrote:
> > > On Tue, Jan 30, 2018 at 02:41:41PM +0100, Michal Hocko wrote:
> > > > It is good to hear that at least something has a documentation coverage.
> > > > I was asking mostly because I _think_ that the API documentation is far
> > > > from the top priority. 
> > > 
> > > API documentations is important for kernel developers who are not deeply
> > > involved with mm. When one develops a device driver, knowing how to
> > > allocate and free memory is essential. And, while *malloc are included in
> > > kernel-api.rst, CMA and HMM documentation is not visible.
> > > 
> > > > We are seriously lacking any highlevel one which describes the design and
> > > > subsytems interaction.
> > > 
> > > I should have describe it better, but by "creating a new structure for mm
> > > documentation" I've also meant adding high level description.
> > 
> > We should be really clear what kind of documentation we're trying to create.
> > 
> > There are four distinct types of documentation which would be useful:
> > 
> >  - How, when and why to use the various function calls and their
> >    parameters from the perspective of a user outside the mm/ hierarchy.
> >    Device driver authors, filesystem authors and others of their ilk.
> >  - The overall philosophy and structure of the mm directory, what it does,
> >    why it does it, perhaps even outlines of abandoned approaches.
> >  - What functionality the mm subsystem requires from others.  For example,
> >    what does the mm rely on from the CPU architectures (and maybe it would
> >    make sense to also include services the mm layer provides to arches in
> >    this section, like setting up sparsemem).
> 
> yes
> 
> >  - How to tweak the various knobs that the mm subsystem provides.
> >    Maybe this is all adequately documented elsewhere already.
> 
> This would be Documentation/sysctl/vm.txt which is one that is at least
> close to be complete.
> 
> > Perhaps others can think of other types of documentation which would
> > be useful.
> 
> - design documentation of various parts of the MM - reclaim, memory
>   hotplug, memcg, page allocator, memory models, THP, rmap code (you
>   name it)
> 
> > That shouldn't detract from my main point, which is that
> > saying "Now we have mm documentation" is laudable, but not enough.
> 
> Absolutely agreed.

I don't think anybody is saying "we have mm documentation", at least in the
sense "mm is well documented".

One of my points was that bringing some order to the existing bits of the
documentation is an important step forward and it does not contradict
necessity to add documentation you and Matthew described here.

> -- 
> Michal Hocko
> SUSE Labs
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

-- 
Sincerely yours,
Mike.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[LSF/MM TOPIC]: mm documentation

[Mike Rapoport]

Hi,

At the last Plumbers plenary there was a discussion about the
documentation and one of the questions to the panel was "Is it better
to have outdated documentation or no documentation at all?" And, not
surprisingly, they've answered, "No documentation is better than
outdated".

The mm documentation is, well, not entirely up to date. We can opt for
dropping the outdated parts, which would generate a nice negative
diffstat, but identifying the outdated documentation requires nearly
as much effort as updating it, so I think that making and keeping
the docs up to date would be a better option.

I'd like to discuss what can be done process-wise to improve the
situation.

Some points I had in mind:

* Pay more attention to docs during review
* Set an expectation level for docs accompanying a changeset
* Add automation to aid spotting inconsistencies between the code and
  the docs
* Spend some cycles to review and update the existing docs
* Spend some more cycles to add new documentation

I'd appreciate a discussion about how we can get to the second edition
of "Understanding the Linux Virtual Memory Manager", what are the gaps
(although they are too many), and what would be the best way to close
these gaps.

-- 
Sincerely yours,
Mike.

Re: [LSF/MM TOPIC]: mm documentation

[Jonathan Cameron]

On Mon, 28 Jan 2019 09:04:22 +0200
Mike Rapoport <rppt@linux.ibm.com> wrote:

> Hi,
> 
> At the last Plumbers plenary there was a discussion about the
> documentation and one of the questions to the panel was "Is it better
> to have outdated documentation or no documentation at all?" And, not
> surprisingly, they've answered, "No documentation is better than
> outdated".
> 
> The mm documentation is, well, not entirely up to date. We can opt for
> dropping the outdated parts, which would generate a nice negative
> diffstat, but identifying the outdated documentation requires nearly
> as much effort as updating it, so I think that making and keeping
> the docs up to date would be a better option.
> 
> I'd like to discuss what can be done process-wise to improve the
> situation.
> 
> Some points I had in mind:
> 
> * Pay more attention to docs during review
> * Set an expectation level for docs accompanying a changeset
> * Add automation to aid spotting inconsistencies between the code and
>   the docs
> * Spend some cycles to review and update the existing docs
> * Spend some more cycles to add new documentation
> 
> I'd appreciate a discussion about how we can get to the second edition
> of "Understanding the Linux Virtual Memory Manager", what are the gaps
> (although they are too many), and what would be the best way to close
> these gaps.
> 

As a recent newbie in mm code...

Even though it is perhaps in need of a refresh the existence of that
book is still useful and a great deal better than many other areas
of the kernel.  I would love to see a new version, but can fully
appreciate the immense effort involved.

Jonathan

[LSF/MM TOPIC] mm documentation

[Mike Rapoport]

Hi,

The mm documentation is, well, not entirely up to date. We can opt for
dropping the outdated parts, which would generate a nice negative
diffstat, but identifying the outdated documentation requires nearly
as much effort as updating it, so I think that making and keeping
the docs up to date would be a better option.

I'd like to discuss what can be done process-wise to improve the
situation.

Some points I had in mind:

* Pay more attention to docs during review
* Set an expectation level for docs accompanying a changeset
* Spend some cycles to review and update the existing docs
* Spend some more cycles to add new documentation
* Participate in prorams like Google Season of Docs

I'd appreciate a discussion about how we can improve the existing memory
management documentation so that a reader can get a coherent view of it,
what are the gaps (although they are too many), and what would be the best
way to close these gaps.

-- 
Sincerely yours,
Mike.

Re: [LSF/MM TOPIC] mm documentation

[Matthew Wilcox]

On Thu, May 20, 2021 at 11:56:09AM +0300, Mike Rapoport wrote:
> The mm documentation is, well, not entirely up to date. We can opt for
> dropping the outdated parts, which would generate a nice negative
> diffstat, but identifying the outdated documentation requires nearly
> as much effort as updating it, so I think that making and keeping
> the docs up to date would be a better option.
> 
> I'd like to discuss what can be done process-wise to improve the
> situation.
> 
> Some points I had in mind:
> 
> * Pay more attention to docs during review
> * Set an expectation level for docs accompanying a changeset
> * Spend some cycles to review and update the existing docs
> * Spend some more cycles to add new documentation
> * Participate in prorams like Google Season of Docs
> 
> I'd appreciate a discussion about how we can improve the existing memory
> management documentation so that a reader can get a coherent view of it,
> what are the gaps (although they are too many), and what would be the best
> way to close these gaps.

I think we have four target audiences for mm documentation,

 - Sysadmins/user space programmers who are trying to make their system
   perform well.  They need documentation of the
   proc/sys/cmdline/... parameters that the MM pays attention to.
 - Kernel programmers who are not (and do not wish to be) MM developers.
   Filesystem developers, device driver developers, networking
   developers.  They need kernel-doc for our exported functions and
   advice for when to use and not use particular functions/flags.
 - Programmers who want to "get started" in the MM area.  They may or
   may not be familiar with Linux internals (perhaps they're moving from
   another kernel, perhaps their experience is with some other part
   of Linux; perhaps they have no experience at all).  I think these
   people are probably well served by Mel's book, even if it is a few
   years old now.
 - Each other.  The MM is huge these days, and I certainly don't
   understand how it all interacts with itself.  I think we actually do
   a pretty good job of talking to each other and writing commit messages.

I think it's really the second group where we do the worst job of
documentation, but I may be biased.  It's certainly where I'm focusing
my documentation efforts.

Re: [LSF/MM TOPIC] mm documentation

[Mike Rapoport]

On Thu, May 20, 2021 at 03:19:08PM +0100, Matthew Wilcox wrote:
> On Thu, May 20, 2021 at 11:56:09AM +0300, Mike Rapoport wrote:
> > The mm documentation is, well, not entirely up to date. We can opt for
> > dropping the outdated parts, which would generate a nice negative
> > diffstat, but identifying the outdated documentation requires nearly
> > as much effort as updating it, so I think that making and keeping
> > the docs up to date would be a better option.
> > 
> > I'd like to discuss what can be done process-wise to improve the
> > situation.
> > 
> > Some points I had in mind:
> > 
> > * Pay more attention to docs during review
> > * Set an expectation level for docs accompanying a changeset
> > * Spend some cycles to review and update the existing docs
> > * Spend some more cycles to add new documentation
> > * Participate in prorams like Google Season of Docs
> > 
> > I'd appreciate a discussion about how we can improve the existing memory
> > management documentation so that a reader can get a coherent view of it,
> > what are the gaps (although they are too many), and what would be the best
> > way to close these gaps.
> 
> I think we have four target audiences for mm documentation,
> 
>  - Sysadmins/user space programmers who are trying to make their system
>    perform well.  They need documentation of the
>    proc/sys/cmdline/... parameters that the MM pays attention to.

I'd say this part needs some love. Just a few weeks ago we've discussed how
/proc/meminfo description is outdated and there are more examples.
Besides, this is probably the most important part to keep up to date.

>  - Kernel programmers who are not (and do not wish to be) MM developers.
>    Filesystem developers, device driver developers, networking
>    developers.  They need kernel-doc for our exported functions and
>    advice for when to use and not use particular functions/flags.
>  - Programmers who want to "get started" in the MM area.  They may or
>    may not be familiar with Linux internals (perhaps they're moving from
>    another kernel, perhaps their experience is with some other part
>    of Linux; perhaps they have no experience at all).  I think these
>    people are probably well served by Mel's book, even if it is a few
>    years old now.

Mel's book is definitely an excellent starting point, but I afraid its age
is starting to show. As it was written mostly about 2.4 with some bits
about 2.6, there are lot of details that are missing in the book. Some are
probably less important because the concept didn't change much (e.g. page
table management), but others have considerable effect on our code (e.g
migration, compaction, THP).

>  - Each other.  The MM is huge these days, and I certainly don't
>    understand how it all interacts with itself.  I think we actually do
>    a pretty good job of talking to each other and writing commit messages.

IMHO, there is a fifth group: arch developers. I think it is important to
have documentation of what core mm expects from an architecture and what is
the expected semantics of various low level functions architectures supply.
 
> I think it's really the second group where we do the worst job of
> documentation, but I may be biased.  It's certainly where I'm focusing
> my documentation efforts.

I'm not sure it's the worst, but certainly along with the first group it's
the most important part.

-- 
Sincerely yours,
Mike.

Re: [LSF/MM TOPIC] mm documentation

[Souptick Joarder]

On Fri, May 21, 2021 at 2:06 PM Mike Rapoport <rppt@kernel.org> wrote:
>
> On Thu, May 20, 2021 at 03:19:08PM +0100, Matthew Wilcox wrote:
> > On Thu, May 20, 2021 at 11:56:09AM +0300, Mike Rapoport wrote:
> > > The mm documentation is, well, not entirely up to date. We can opt for
> > > dropping the outdated parts, which would generate a nice negative
> > > diffstat, but identifying the outdated documentation requires nearly
> > > as much effort as updating it, so I think that making and keeping
> > > the docs up to date would be a better option.
> > >
> > > I'd like to discuss what can be done process-wise to improve the
> > > situation.
> > >
> > > Some points I had in mind:
> > >
> > > * Pay more attention to docs during review
> > > * Set an expectation level for docs accompanying a changeset
> > > * Spend some cycles to review and update the existing docs
> > > * Spend some more cycles to add new documentation
> > > * Participate in prorams like Google Season of Docs
> > >
> > > I'd appreciate a discussion about how we can improve the existing memory
> > > management documentation so that a reader can get a coherent view of it,
> > > what are the gaps (although they are too many), and what would be the best
> > > way to close these gaps.
> >
> > I think we have four target audiences for mm documentation,
> >
> >  - Sysadmins/user space programmers who are trying to make their system
> >    perform well.  They need documentation of the
> >    proc/sys/cmdline/... parameters that the MM pays attention to.
>
> I'd say this part needs some love. Just a few weeks ago we've discussed how
> /proc/meminfo description is outdated and there are more examples.
> Besides, this is probably the most important part to keep up to date.
>
> >  - Kernel programmers who are not (and do not wish to be) MM developers.
> >    Filesystem developers, device driver developers, networking
> >    developers.  They need kernel-doc for our exported functions and
> >    advice for when to use and not use particular functions/flags.
> >  - Programmers who want to "get started" in the MM area.  They may or
> >    may not be familiar with Linux internals (perhaps they're moving from
> >    another kernel, perhaps their experience is with some other part
> >    of Linux; perhaps they have no experience at all).  I think these
> >    people are probably well served by Mel's book, even if it is a few
> >    years old now.
>
> Mel's book is definitely an excellent starting point, but I afraid its age
> is starting to show. As it was written mostly about 2.4 with some bits
> about 2.6, there are lot of details that are missing in the book. Some are
> probably less important because the concept didn't change much (e.g. page
> table management), but others have considerable effect on our code (e.g
> migration, compaction, THP).

IMO, updating Mel's book will be helpful. I thought about it but realized that
without guidance I can't make good progress.

>
> >  - Each other.  The MM is huge these days, and I certainly don't
> >    understand how it all interacts with itself.  I think we actually do
> >    a pretty good job of talking to each other and writing commit messages.
>
> IMHO, there is a fifth group: arch developers. I think it is important to
> have documentation of what core mm expects from an architecture and what is
> the expected semantics of various low level functions architectures supply.
>
> > I think it's really the second group where we do the worst job of
> > documentation, but I may be biased.  It's certainly where I'm focusing
> > my documentation efforts.
>
> I'm not sure it's the worst, but certainly along with the first group it's
> the most important part.

I am interested to contribute to this if agree to take it forward.