Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Konstantin Khlebnikov]

On 23.10.2018 7:15, Joe Perches wrote:> On Mon, 2018-10-22 at 22:53 +0530, Arun KS wrote:
 >> Remove managed_page_count_lock spinlock and instead use atomic
 >> variables.
 >
 > Perhaps better to define and use macros for the accesses
 > instead of specific uses of atomic_long_<inc/dec/read>
 >
 > Something like:
 >
 > #define totalram_pages()	(unsigned long)atomic_long_read(&_totalram_pages)

or proper static inline
this code isn't so low level for breaking include dependencies with macro

 > #define totalram_pages_inc()	(unsigned long)atomic_long_inc(&_totalram_pages)
 > #define totalram_pages_dec()	(unsigned long)atomic_long_dec(&_totalram_pages)

these are void


conversion zone->managed_pages should be split into separate patch


[dropped bloated cc - my server rejects this mess]

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Kees Cook]

On Mon, Oct 22, 2018 at 10:11 PM, Konstantin Khlebnikov
<khlebnikov@yandex-team.ru> wrote:
> On 23.10.2018 7:15, Joe Perches wrote:> On Mon, 2018-10-22 at 22:53 +0530,
> Arun KS wrote:
>>> Remove managed_page_count_lock spinlock and instead use atomic
>>> variables.
>>
>> Perhaps better to define and use macros for the accesses
>> instead of specific uses of atomic_long_<inc/dec/read>
>>
>> Something like:
>>
>> #define totalram_pages()      (unsigned
>> long)atomic_long_read(&_totalram_pages)
>
> or proper static inline
> this code isn't so low level for breaking include dependencies with macro

BTW, I noticed a few places in the patch that did multiple evaluations
of totalram_pages. It might be worth fixing those prior to doing the
conversion, too. e.g.:

if (totalram_pages > something)
   foobar(totalram_pages); <- value may have changed here

should, instead, be:

var = totalram_pages; <- get stable view of the value
if (var > something)
    foobar(var);

-Kees

> [dropped bloated cc - my server rejects this mess]

Thank you -- I was struggling to figure out the best way to reply to this. :)

-Kees

-- 
Kees Cook

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Arun KS]

On 2018-10-24 01:34, Kees Cook wrote:
> On Mon, Oct 22, 2018 at 10:11 PM, Konstantin Khlebnikov
> <khlebnikov@yandex-team.ru> wrote:
>> On 23.10.2018 7:15, Joe Perches wrote:> On Mon, 2018-10-22 at 22:53 
>> +0530,
>> Arun KS wrote:
>>>> Remove managed_page_count_lock spinlock and instead use atomic
>>>> variables.
>>> 
>>> Perhaps better to define and use macros for the accesses
>>> instead of specific uses of atomic_long_<inc/dec/read>
>>> 
>>> Something like:
>>> 
>>> #define totalram_pages()      (unsigned
>>> long)atomic_long_read(&_totalram_pages)
>> 
>> or proper static inline
>> this code isn't so low level for breaking include dependencies with 
>> macro
> 
> BTW, I noticed a few places in the patch that did multiple evaluations
> of totalram_pages. It might be worth fixing those prior to doing the
> conversion, too. e.g.:
> 
> if (totalram_pages > something)
>    foobar(totalram_pages); <- value may have changed here
> 
> should, instead, be:
> 
> var = totalram_pages; <- get stable view of the value
> if (var > something)
>     foobar(var);

Thanks for reviewing. Point taken.
> 
> -Kees
> 
>> [dropped bloated cc - my server rejects this mess]
> 
> Thank you -- I was struggling to figure out the best way to reply to 
> this. :)
I'm sorry for the trouble caused. Sent the email using,
git send-email  --to-cmd="scripts/get_maintainer.pl -i" 
0001-convert-totalram_pages-totalhigh_pages-and-managed_p.patch

Is this not a recommended approach?

Regards,
Arun

> 
> -Kees

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Michal Hocko]

On Wed 24-10-18 10:47:52, Arun KS wrote:
> On 2018-10-24 01:34, Kees Cook wrote:
[...]
> > Thank you -- I was struggling to figure out the best way to reply to
> > this. :)
> I'm sorry for the trouble caused. Sent the email using,
> git send-email  --to-cmd="scripts/get_maintainer.pl -i"
> 0001-convert-totalram_pages-totalhigh_pages-and-managed_p.patch
> 
> Is this not a recommended approach?

Not really for tree wide mechanical changes. It is much more preferrable
IMHO to only CC people who should review the intention of the change
rather than each and every maintainer whose code is going to be changed.
This is a case by case thing of course but as soon as you see a giant CC
list from get_maintainer.pl then you should try to think twice to use
it. If not sure, just ask on the mailing list.

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Joe Perches]

On Wed, 2018-10-24 at 08:15 +0200, Michal Hocko wrote:
> On Wed 24-10-18 10:47:52, Arun KS wrote:
> > On 2018-10-24 01:34, Kees Cook wrote:
> [...]
> > > Thank you -- I was struggling to figure out the best way to reply to
> > > this. :)
> > I'm sorry for the trouble caused. Sent the email using,
> > git send-email  --to-cmd="scripts/get_maintainer.pl -i"
> > 0001-convert-totalram_pages-totalhigh_pages-and-managed_p.patch
> > 
> > Is this not a recommended approach?
> 
> Not really for tree wide mechanical changes. It is much more preferrable
> IMHO to only CC people who should review the intention of the change
> rather than each and every maintainer whose code is going to be changed.
> This is a case by case thing of course but as soon as you see a giant CC
> list from get_maintainer.pl then you should try to think twice to use
> it. If not sure, just ask on the mailing list.

Generally, it's better to use scripts to control
the --to-cmd and --cc-cmd options.

Something like what I detailed here:

https://lkml.org/lkml/2016/9/14/482

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Michal Hocko]

On Tue 23-10-18 23:26:16, Joe Perches wrote:
> On Wed, 2018-10-24 at 08:15 +0200, Michal Hocko wrote:
> > On Wed 24-10-18 10:47:52, Arun KS wrote:
> > > On 2018-10-24 01:34, Kees Cook wrote:
> > [...]
> > > > Thank you -- I was struggling to figure out the best way to reply to
> > > > this. :)
> > > I'm sorry for the trouble caused. Sent the email using,
> > > git send-email  --to-cmd="scripts/get_maintainer.pl -i"
> > > 0001-convert-totalram_pages-totalhigh_pages-and-managed_p.patch
> > > 
> > > Is this not a recommended approach?
> > 
> > Not really for tree wide mechanical changes. It is much more preferrable
> > IMHO to only CC people who should review the intention of the change
> > rather than each and every maintainer whose code is going to be changed.
> > This is a case by case thing of course but as soon as you see a giant CC
> > list from get_maintainer.pl then you should try to think twice to use
> > it. If not sure, just ask on the mailing list.
> 
> Generally, it's better to use scripts to control
> the --to-cmd and --cc-cmd options.

I would argue that it is better to use a common sense much more than
scripts.

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Joe Perches]

On Wed, 2018-10-24 at 10:23 +0200, Michal Hocko wrote:
> On Tue 23-10-18 23:26:16, Joe Perches wrote:
> > On Wed, 2018-10-24 at 08:15 +0200, Michal Hocko wrote:
> > > On Wed 24-10-18 10:47:52, Arun KS wrote:
> > > > On 2018-10-24 01:34, Kees Cook wrote:
> > > [...]
> > > > > Thank you -- I was struggling to figure out the best way to reply to
> > > > > this. :)
> > > > I'm sorry for the trouble caused. Sent the email using,
> > > > git send-email  --to-cmd="scripts/get_maintainer.pl -i"
> > > > 0001-convert-totalram_pages-totalhigh_pages-and-managed_p.patch
> > > > 
> > > > Is this not a recommended approach?
> > > 
> > > Not really for tree wide mechanical changes. It is much more preferrable
> > > IMHO to only CC people who should review the intention of the change
> > > rather than each and every maintainer whose code is going to be changed.
> > > This is a case by case thing of course but as soon as you see a giant CC
> > > list from get_maintainer.pl then you should try to think twice to use
> > > it. If not sure, just ask on the mailing list.
> > 
> > Generally, it's better to use scripts to control
> > the --to-cmd and --cc-cmd options.
> 
> I would argue that it is better to use a common sense much more than
> scripts.

Common sense isn't common.

Perhaps you could describe some guidelines you
use to determine what you think is sensible to
send patches to maintainers and reviewers and
appropriate mailing lists.

Then compare those to the rules in the scripts
I suggested.

My suggestions:

o send to all top-level listed maintainers and
  reviewers in MAINTAINERS for the specific files
  in a patch
o cc all maintainers and reviewers that are upstream
  paths for the files in a patch
o cc all the upstream mailing lists for the files
  in the patch.
o do not generally use git-history to exclude authors
  of patches like drive-by/whitespace cleanups

Other advanced possibilities for patches that
modify specific and perhaps complex logic blocks:

o cc the people that are not maintainers that
  have modified the specific blocks or functions

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Michal Hocko]

On Wed 24-10-18 01:39:18, Joe Perches wrote:
> On Wed, 2018-10-24 at 10:23 +0200, Michal Hocko wrote:
> > On Tue 23-10-18 23:26:16, Joe Perches wrote:
> > > On Wed, 2018-10-24 at 08:15 +0200, Michal Hocko wrote:
> > > > On Wed 24-10-18 10:47:52, Arun KS wrote:
> > > > > On 2018-10-24 01:34, Kees Cook wrote:
> > > > [...]
> > > > > > Thank you -- I was struggling to figure out the best way to reply to
> > > > > > this. :)
> > > > > I'm sorry for the trouble caused. Sent the email using,
> > > > > git send-email  --to-cmd="scripts/get_maintainer.pl -i"
> > > > > 0001-convert-totalram_pages-totalhigh_pages-and-managed_p.patch
> > > > > 
> > > > > Is this not a recommended approach?
> > > > 
> > > > Not really for tree wide mechanical changes. It is much more preferrable
> > > > IMHO to only CC people who should review the intention of the change
> > > > rather than each and every maintainer whose code is going to be changed.
> > > > This is a case by case thing of course but as soon as you see a giant CC
> > > > list from get_maintainer.pl then you should try to think twice to use
> > > > it. If not sure, just ask on the mailing list.
> > > 
> > > Generally, it's better to use scripts to control
> > > the --to-cmd and --cc-cmd options.
> > 
> > I would argue that it is better to use a common sense much more than
> > scripts.
> 
> Common sense isn't common.

But you cannot replace brain by a script.

Again, this is mostly a mechanical change quite internal to the MM
proper. Involving all the maintainers which happen to use totalram_pages
etc. are unlikely to need to know about this change. Sure there is some
chance of merge conflicts and that is where linux-next comes really
handy.

This is the best advice I can give here.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Michal Hocko]

[Trimmed CC list + Julia - there is indeed no need to CC everybody maintain a
file you are updating for the change like this]

On Tue 23-10-18 10:16:51, Arun Sudhilal wrote:
> On Mon, Oct 22, 2018 at 11:41 PM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Mon 22-10-18 22:53:22, Arun KS wrote:
> > > Remove managed_page_count_lock spinlock and instead use atomic
> > > variables.
> >
> 
> Hello Michal,
> > I assume this has been auto-generated. If yes, it would be better to
> > mention the script so that people can review it and regenerate for
> > comparision. Such a large change is hard to review manually.
> 
> Changes were made partially with script.  For totalram_pages and
> totalhigh_pages,
> 
> find dir -type f -exec sed -i
> 's/totalram_pages/atomic_long_read(\&totalram_pages)/g' {} \;
> find dir -type f -exec sed -i
> 's/totalhigh_pages/atomic_long_read(\&totalhigh_pages)/g' {} \;
> 
> For managed_pages it was mostly manual edits after using,
> find mm/ -type f -exec sed -i
> 's/zone->managed_pages/atomic_long_read(\&zone->managed_pages)/g' {}
> \;

I guess we should be able to use coccinelle for this kind of change and
reduce the amount of manual intervention to absolute minimum.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Julia Lawall]

On Tue, 23 Oct 2018, Michal Hocko wrote:

> [Trimmed CC list + Julia - there is indeed no need to CC everybody maintain a
> file you are updating for the change like this]
>
> On Tue 23-10-18 10:16:51, Arun Sudhilal wrote:
> > On Mon, Oct 22, 2018 at 11:41 PM Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > On Mon 22-10-18 22:53:22, Arun KS wrote:
> > > > Remove managed_page_count_lock spinlock and instead use atomic
> > > > variables.
> > >
> >
> > Hello Michal,
> > > I assume this has been auto-generated. If yes, it would be better to
> > > mention the script so that people can review it and regenerate for
> > > comparision. Such a large change is hard to review manually.
> >
> > Changes were made partially with script.  For totalram_pages and
> > totalhigh_pages,
> >
> > find dir -type f -exec sed -i
> > 's/totalram_pages/atomic_long_read(\&totalram_pages)/g' {} \;
> > find dir -type f -exec sed -i
> > 's/totalhigh_pages/atomic_long_read(\&totalhigh_pages)/g' {} \;
> >
> > For managed_pages it was mostly manual edits after using,
> > find mm/ -type f -exec sed -i
> > 's/zone->managed_pages/atomic_long_read(\&zone->managed_pages)/g' {}
> > \;
>
> I guess we should be able to use coccinelle for this kind of change and
> reduce the amount of manual intervention to absolute minimum.

Coccinelle looks like it would be desirable, especially in case the word
zone is not always used.

Arun, please feel free to contact me if you want to try it and need help.

julia

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Arun Sudhilal]

On Tue, Oct 23, 2018 at 12:11 PM Julia Lawall <julia.lawall@lip6.fr> wrote:
>
>
>
> On Tue, 23 Oct 2018, Michal Hocko wrote:
>
> > [Trimmed CC list + Julia - there is indeed no need to CC everybody maintain a
> > file you are updating for the change like this]
> >
> > On Tue 23-10-18 10:16:51, Arun Sudhilal wrote:
> > > On Mon, Oct 22, 2018 at 11:41 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > >
> > > > On Mon 22-10-18 22:53:22, Arun KS wrote:
> > > > > Remove managed_page_count_lock spinlock and instead use atomic
> > > > > variables.
> > > >
> > >
> > > Hello Michal,
> > > > I assume this has been auto-generated. If yes, it would be better to
> > > > mention the script so that people can review it and regenerate for
> > > > comparision. Such a large change is hard to review manually.
> > >
> > > Changes were made partially with script.  For totalram_pages and
> > > totalhigh_pages,
> > >
> > > find dir -type f -exec sed -i
> > > 's/totalram_pages/atomic_long_read(\&totalram_pages)/g' {} \;
> > > find dir -type f -exec sed -i
> > > 's/totalhigh_pages/atomic_long_read(\&totalhigh_pages)/g' {} \;
> > >
> > > For managed_pages it was mostly manual edits after using,
> > > find mm/ -type f -exec sed -i
> > > 's/zone->managed_pages/atomic_long_read(\&zone->managed_pages)/g' {}
> > > \;
> >
> > I guess we should be able to use coccinelle for this kind of change and
> > reduce the amount of manual intervention to absolute minimum.
>
> Coccinelle looks like it would be desirable, especially in case the word
> zone is not always used.
>
> Arun, please feel free to contact me if you want to try it and need help.

Thanks Julia. I m starting with,
http://coccinelle.lip6.fr/papers/backport_edcc15.pdf

Regards,
Arun
>
> julia

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Arun KS]

On 2018-10-23 11:43, Julia Lawall wrote:
> On Tue, 23 Oct 2018, Michal Hocko wrote:
> 
>> [Trimmed CC list + Julia - there is indeed no need to CC everybody 
>> maintain a
>> file you are updating for the change like this]
>> 
>> On Tue 23-10-18 10:16:51, Arun Sudhilal wrote:
>> > On Mon, Oct 22, 2018 at 11:41 PM Michal Hocko <mhocko@kernel.org> wrote:
>> > >
>> > > On Mon 22-10-18 22:53:22, Arun KS wrote:
>> > > > Remove managed_page_count_lock spinlock and instead use atomic
>> > > > variables.
>> > >
>> >
>> > Hello Michal,
>> > > I assume this has been auto-generated. If yes, it would be better to
>> > > mention the script so that people can review it and regenerate for
>> > > comparision. Such a large change is hard to review manually.
>> >
>> > Changes were made partially with script.  For totalram_pages and
>> > totalhigh_pages,
>> >
>> > find dir -type f -exec sed -i
>> > 's/totalram_pages/atomic_long_read(\&totalram_pages)/g' {} \;
>> > find dir -type f -exec sed -i
>> > 's/totalhigh_pages/atomic_long_read(\&totalhigh_pages)/g' {} \;
>> >
>> > For managed_pages it was mostly manual edits after using,
>> > find mm/ -type f -exec sed -i
>> > 's/zone->managed_pages/atomic_long_read(\&zone->managed_pages)/g' {}
>> > \;
>> 
>> I guess we should be able to use coccinelle for this kind of change 
>> and
>> reduce the amount of manual intervention to absolute minimum.
> 
> Coccinelle looks like it would be desirable, especially in case the 
> word
> zone is not always used.
> 
> Arun, please feel free to contact me if you want to try it and need 
> help.
Hello Julia,

I was able to come up .cocci for replacing managed_pages,

@@
struct zone *z;
@@
(
z->managed_pages = ...
|
- z->managed_pages
+ atomic_long_read(&z->managed_pages)
)

@@
struct zone *z;
expression e1;
@@
- z->managed_pages = e1
+ atomic_long_set(&z->managed_pages, e1)

@@
expression e,e1;
@@
- e->managed_pages += e1
+ atomic_long_add(e1, &e->managed_pages)

@@
expression z;
@@
- z.managed_pages
+ atomic_long_read(&z.managed_pages)

But I m not able to use same method for unsigned long 
variables(totalram_pages)

@@
unsigned long totalram_pages;
@@
(
totalram_pages = ...
|
-totalram_pages
+atomic_long_read(&totalram_pages)
)

This throws error,

spatch test1.cocci mm/page_alloc.c
init_defs_builtins: /usr/lib/coccinelle/standard.h
HANDLING: mm/page_alloc.c

previous modification:
MINUS
   >>> atomic_long_read(&rule starting on line 1:totalram_pages)

According to environment 1:
    rule starting on line 1.totalram_pages -> page_idx ^ (1 <<
order)

current modification:
MINUS
   >>> atomic_long_read(&rule starting on line 1:totalram_pages)

According to environment 1:
    rule starting on line 1.totalram_pages -> page_idx

Fatal error: exception Failure("rule starting on line 1: already tagged 
token:
C code context
File "mm/internal.h", line 175, column 8,  charpos = 5368
     around = 'page_idx', whole content =        return page_idx ^ (1 << 
order);")

Regards,
Arun


> 
> julia