* [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions
@ 2021-04-09 11:01 Peter Korsgaard
2021-04-17 7:19 ` Yann E. MORIN
2021-04-26 7:09 ` Peter Korsgaard
0 siblings, 2 replies; 3+ messages in thread
From: Peter Korsgaard @ 2021-04-09 11:01 UTC (permalink / raw)
To: buildroot
ijson < 2.5 (as available in Debian 10) use the slow python backend by
default instead of the most efficient one available like modern ijson
versions, significantly slowing down cve checking. E.G.:
time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html
Goes from
174,44s user 2,11s system 99% cpu 2:58,04 total
To
93,53s user 2,00s system 98% cpu 1:36,65 total
E.G. almost 2x as fast.
As a workaround, detect when the python backend is used and try to use a
more efficient one instead. Use the yajl2_cffi backend as recommended by
upstream, as it is most likely to work, and print a warning (and continue)
if we fail to load it.
The detection is slightly complicated by the fact that ijson.backends used
to be a reference to a backend module, but is nowadays a string (without the
ijson.backends prefix).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
support/scripts/cve.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index 6e97ea193f..965fc2a466 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -28,6 +28,12 @@ import operator
try:
import ijson
+ # backend is a module in < 2.5, a string in >= 2.5
+ if 'python' in getattr(ijson.backend, '__name__', ijson.backend):
+ try:
+ import ijson.backends.yajl2_cffi as ijson
+ except ImportError:
+ sys.stderr.write('Warning: Using slow ijson python backend\n')
except ImportError:
sys.stderr.write("You need ijson to parse NVD for CVE check\n")
exit(1)
--
2.20.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions
2021-04-09 11:01 [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions Peter Korsgaard
@ 2021-04-17 7:19 ` Yann E. MORIN
2021-04-26 7:09 ` Peter Korsgaard
1 sibling, 0 replies; 3+ messages in thread
From: Yann E. MORIN @ 2021-04-17 7:19 UTC (permalink / raw)
To: buildroot
Peter, All,
On 2021-04-09 13:01 +0200, Peter Korsgaard spake thusly:
> ijson < 2.5 (as available in Debian 10) use the slow python backend by
> default instead of the most efficient one available like modern ijson
> versions, significantly slowing down cve checking. E.G.:
>
> time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html
>
> Goes from
> 174,44s user 2,11s system 99% cpu 2:58,04 total
>
> To
> 93,53s user 2,00s system 98% cpu 1:36,65 total
>
> E.G. almost 2x as fast.
>
> As a workaround, detect when the python backend is used and try to use a
> more efficient one instead. Use the yajl2_cffi backend as recommended by
> upstream, as it is most likely to work, and print a warning (and continue)
> if we fail to load it.
>
> The detection is slightly complicated by the fact that ijson.backends used
> to be a reference to a backend module, but is nowadays a string (without the
> ijson.backends prefix).
>
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Applied to master, thanks.
Regards,
Yann E. MORIN.
> ---
> support/scripts/cve.py | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/support/scripts/cve.py b/support/scripts/cve.py
> index 6e97ea193f..965fc2a466 100755
> --- a/support/scripts/cve.py
> +++ b/support/scripts/cve.py
> @@ -28,6 +28,12 @@ import operator
>
> try:
> import ijson
> + # backend is a module in < 2.5, a string in >= 2.5
> + if 'python' in getattr(ijson.backend, '__name__', ijson.backend):
> + try:
> + import ijson.backends.yajl2_cffi as ijson
> + except ImportError:
> + sys.stderr.write('Warning: Using slow ijson python backend\n')
> except ImportError:
> sys.stderr.write("You need ijson to parse NVD for CVE check\n")
> exit(1)
> --
> 2.20.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 3+ messages in thread
* [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions
2021-04-09 11:01 [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions Peter Korsgaard
2021-04-17 7:19 ` Yann E. MORIN
@ 2021-04-26 7:09 ` Peter Korsgaard
1 sibling, 0 replies; 3+ messages in thread
From: Peter Korsgaard @ 2021-04-26 7:09 UTC (permalink / raw)
To: buildroot
>>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:
> ijson < 2.5 (as available in Debian 10) use the slow python backend by
> default instead of the most efficient one available like modern ijson
> versions, significantly slowing down cve checking. E.G.:
> time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html
> Goes from
> 174,44s user 2,11s system 99% cpu 2:58,04 total
> To
> 93,53s user 2,00s system 98% cpu 1:36,65 total
> E.G. almost 2x as fast.
> As a workaround, detect when the python backend is used and try to use a
> more efficient one instead. Use the yajl2_cffi backend as recommended by
> upstream, as it is most likely to work, and print a warning (and continue)
> if we fail to load it.
> The detection is slightly complicated by the fact that ijson.backends used
> to be a reference to a backend module, but is nowadays a string (without the
> ijson.backends prefix).
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Committed to 2021.02.x, thanks.
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-04-26 7:09 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-09 11:01 [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions Peter Korsgaard
2021-04-17 7:19 ` Yann E. MORIN
2021-04-26 7:09 ` Peter Korsgaard
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.