Coccinelle archive on lore.kernel.org
 help / Atom feed
* [Cocci] 0079-netdev-destructor.cocci very slow
@ 2018-09-17 21:55 hauke
  2018-09-18  9:22 ` johannes
  0 siblings, 1 reply; 6+ messages in thread
From: hauke @ 2018-09-17 21:55 UTC (permalink / raw)
  To: cocci

The 0079-netdev-destructor.cocci spatch in backports is very slow for
me. For bigger files I get a warning that it takes over to 15 seconds to
apply it to just one file, for the complete backports tree it takes over
an hour to apply.

This is the patch:
https://git.kernel.org/pub/scm/linux/kernel/git/backports/backports.git/tree/patches/0079-netdev-destructor.cocci

When I remove the <-- --> in the first rule, it is applied in some
seconds on the complete tree, so an speed improvement of about 100
times, but it is not working correctly any more. ;-)

Is this normal or how can I improve the spatch to be faster? I am using
coccinelle 1.0.7 build with default configure arguments against the
libraries from Debian stable.

If this is normal I should probably try to reduce the number of files it
tries to apply this against in gentree.py before spatch gets started.

Hauke

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20180917/1cbf5f50/attachment.asc>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [Cocci] 0079-netdev-destructor.cocci very slow
  2018-09-17 21:55 [Cocci] 0079-netdev-destructor.cocci very slow hauke
@ 2018-09-18  9:22 ` johannes
  2018-09-18 21:52   ` hauke
  0 siblings, 1 reply; 6+ messages in thread
From: johannes @ 2018-09-18  9:22 UTC (permalink / raw)
  To: cocci

On Mon, 2018-09-17 at 23:55 +0200, Hauke Mehrtens wrote:
> The 0079-netdev-destructor.cocci spatch in backports is very slow for
> me. For bigger files I get a warning that it takes over to 15 seconds to
> apply it to just one file, for the complete backports tree it takes over
> an hour to apply.
> 
> This is the patch:
> https://git.kernel.org/pub/scm/linux/kernel/git/backports/backports.git/tree/patches/0079-netdev-destructor.cocci
> 
> When I remove the <-- --> in the first rule, it is applied in some
> seconds on the complete tree, so an speed improvement of about 100
> times, but it is not working correctly any more. ;-)
> 
> Is this normal or how can I improve the spatch to be faster? I am using
> coccinelle 1.0.7 build with default configure arguments against the
> libraries from Debian stable.

We've had this discussion before :-)
I think we determined that it was normal.

> If this is normal I should probably try to reduce the number of files it
> tries to apply this against in gentree.py before spatch gets started.

spatch should already try that internally, but perhaps with some extra
knowledge we can do a better job ...

johannes

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [Cocci] 0079-netdev-destructor.cocci very slow
  2018-09-18  9:22 ` johannes
@ 2018-09-18 21:52   ` hauke
  2018-09-19  8:43     ` johannes
  0 siblings, 1 reply; 6+ messages in thread
From: hauke @ 2018-09-18 21:52 UTC (permalink / raw)
  To: cocci

On 09/18/2018 11:22 AM, Johannes Berg wrote:
> On Mon, 2018-09-17 at 23:55 +0200, Hauke Mehrtens wrote:
>> The 0079-netdev-destructor.cocci spatch in backports is very slow for
>> me. For bigger files I get a warning that it takes over to 15 seconds to
>> apply it to just one file, for the complete backports tree it takes over
>> an hour to apply.
>>
>> This is the patch:
>> https://git.kernel.org/pub/scm/linux/kernel/git/backports/backports.git/tree/patches/0079-netdev-destructor.cocci
>>
>> When I remove the <-- --> in the first rule, it is applied in some
>> seconds on the complete tree, so an speed improvement of about 100
>> times, but it is not working correctly any more. ;-)
>>
>> Is this normal or how can I improve the spatch to be faster? I am using
>> coccinelle 1.0.7 build with default configure arguments against the
>> libraries from Debian stable.
> 
> We've had this discussion before :-)
> I think we determined that it was normal.
> 
>> If this is normal I should probably try to reduce the number of files it
>> tries to apply this against in gentree.py before spatch gets started.
> 
> spatch should already try that internally, but perhaps with some extra
> knowledge we can do a better job ...
> 
> johannes
> 
Hi Johannes,

Yes we talked about this topic some months ago in IRC.
If there is really no better solution, then I will grep in all files for
needs_free_netdev and priv_destructor and only apply this to the files
which are matching. This list should be pretty short.

It looks like coccinelle already does such a grep when I remove the <--
--> from the patch, because this is about 100 times faster.

Hauke

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20180918/07796208/attachment.asc>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [Cocci] 0079-netdev-destructor.cocci very slow
  2018-09-18 21:52   ` hauke
@ 2018-09-19  8:43     ` johannes
  2018-09-19  8:49       ` julia.lawall
  0 siblings, 1 reply; 6+ messages in thread
From: johannes @ 2018-09-19  8:43 UTC (permalink / raw)
  To: cocci

On Tue, 2018-09-18 at 23:52 +0200, Hauke Mehrtens wrote:

> > spatch should already try that internally, but perhaps with some extra
> > knowledge we can do a better job ...

> Yes we talked about this topic some months ago in IRC.
> If there is really no better solution, then I will grep in all files for
> needs_free_netdev and priv_destructor and only apply this to the files
> which are matching. This list should be pretty short.

Right. No objection to that. Perhaps we should have some sort of special
comment header for our spatches that the script can consume?

Something like

// restrict-files: grep -qE 'needs_free_netdev|priv_destructor'

and we'd run that on all files? Or perhaps the API should be more a la
"grep -lE" so we can run it on many files and get a list of matching
files out?

> It looks like coccinelle already does such a grep when I remove the <--
> --> from the patch, because this is about 100 times faster.

Good point, not sure why it doesn't do that with the <... ...>?

johannes

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [Cocci] 0079-netdev-destructor.cocci very slow
  2018-09-19  8:43     ` johannes
@ 2018-09-19  8:49       ` julia.lawall
  2018-09-19  9:02         ` johannes
  0 siblings, 1 reply; 6+ messages in thread
From: julia.lawall @ 2018-09-19  8:49 UTC (permalink / raw)
  To: cocci



On Wed, 19 Sep 2018, Johannes Berg wrote:

> On Tue, 2018-09-18 at 23:52 +0200, Hauke Mehrtens wrote:
>
> > > spatch should already try that internally, but perhaps with some extra
> > > knowledge we can do a better job ...
>
> > Yes we talked about this topic some months ago in IRC.
> > If there is really no better solution, then I will grep in all files for
> > needs_free_netdev and priv_destructor and only apply this to the files
> > which are matching. This list should be pretty short.
>
> Right. No objection to that. Perhaps we should have some sort of special
> comment header for our spatches that the script can consume?
>
> Something like
>
> // restrict-files: grep -qE 'needs_free_netdev|priv_destructor'
>
> and we'd run that on all files? Or perhaps the API should be more a la
> "grep -lE" so we can run it on many files and get a list of matching
> files out?
>
> > It looks like coccinelle already does such a grep when I remove the <--
> > --> from the patch, because this is about 100 times faster.
>
> Good point, not sure why it doesn't do that with the <... ...>?

Because <... ...> means 0 or more of what is inside.  <+... ...+> looks
for one or more and may be faster.  On the other hand, it ensures that
there is one or more, which can also be expensive.

It could be better to just have a rule:

@worthwhile@
@@

(
functions(...)
|
you(...)
|
like(...)
)

and then have the <... ...> rule depend on worthwhile.

julia

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [Cocci] 0079-netdev-destructor.cocci very slow
  2018-09-19  8:49       ` julia.lawall
@ 2018-09-19  9:02         ` johannes
  0 siblings, 0 replies; 6+ messages in thread
From: johannes @ 2018-09-19  9:02 UTC (permalink / raw)
  To: cocci

On Wed, 2018-09-19 at 10:49 +0200, Julia Lawall wrote:

> > > It looks like coccinelle already does such a grep when I remove the <--
> > > --> from the patch, because this is about 100 times faster.
> > 
> > Good point, not sure why it doesn't do that with the <... ...>?
> 
> Because <... ...> means 0 or more of what is inside.

Oops, right.

> <+... ...+> looks for one or more and may be faster.

Indeed, it's two orders of magnitude faster (running it on just
drivers/net/wireless goes from ~500 to ~2s for me) as it can throw away
almost all files immediately.

> On the other hand, it ensures that
> there is one or more, which can also be expensive.

That doesn't really matter all that much for us - the (really) expensive
part is running it on all files that don't even contain it at all.

> It could be better to just have a rule:
> 
> @worthwhile@
> @@
> 
> (
> functions(...)
> > 
> 
> you(...)
> > 
> 
> like(...)
> )
> 
> and then have the <... ...> rule depend on worthwhile.

Good idea too.

Thanks!

johannes

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-09-17 21:55 [Cocci] 0079-netdev-destructor.cocci very slow hauke
2018-09-18  9:22 ` johannes
2018-09-18 21:52   ` hauke
2018-09-19  8:43     ` johannes
2018-09-19  8:49       ` julia.lawall
2018-09-19  9:02         ` johannes

Coccinelle archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/cocci/0 cocci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 cocci cocci/ https://lore.kernel.org/cocci \
		cocci@systeme.lip6.fr cocci@archiver.kernel.org
	public-inbox-index cocci


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/fr.lip6.systeme.cocci


AGPL code for this site: git clone https://public-inbox.org/ public-inbox