[Buildroot] [PATCH] package/plg-utils: escape \ in generated legal-info

[Buildroot] [PATCH] package/plg-utils: escape \ in generated legal-info

[Yann E. MORIN]

In the output of legal-info, which is JSON-formatted, we include the
CPI_ID (when it is valid).

For xerces, the CPE_IS contains two sequences aof \+ (which is exactly
what is present in the NIST DB, [0]).

However, in JSON, like in C, \ escapes the following character; only a
very limited set of characters are valid to escape: " \ / b f n r t u.
Escaping any other character is invalid. Conformant JSON parser will
choke on invalid sequences, and so does not the json python module:

      File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
        obj, end = self.scan_once(s, idx)
    ValueError: Invalid \escape: line 1 column 608554 (char 608553)

We fix that be globally escaping \ in our json output, in the generic
sanitsing macro.

[0] https://nvd.nist.gov/products/cpe/detail/645?namingFormat=2.3&orderBy=CPEURI&keyword=xerces&status=FINAL

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-utils.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index b50d459075..ae3c7f9da9 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -171,8 +171,9 @@ endef
 clean-json = $(strip \
 	$(subst $(comma)},}, $(subst $(comma)$(space)},$(space)}, \
 	$(subst $(comma)],], $(subst $(comma)$(space)],$(space)], \
+	$(subst \,\\, \
 		$(strip $(1)) \
-	)))) \
+	))))) \
 )
 
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-- 
2.25.1

[Buildroot] [PATCH] package/plg-utils: escape \ in generated legal-info

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > In the output of legal-info, which is JSON-formatted, we include the
 > CPI_ID (when it is valid).

 > For xerces, the CPE_IS contains two sequences aof \+ (which is exactly
 > what is present in the NIST DB, [0]).

 > However, in JSON, like in C, \ escapes the following character; only a
 > very limited set of characters are valid to escape: " \ / b f n r t u.
 > Escaping any other character is invalid. Conformant JSON parser will
 > choke on invalid sequences, and so does not the json python module:

 >       File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
 >         obj, end = self.scan_once(s, idx)
 >     ValueError: Invalid \escape: line 1 column 608554 (char 608553)

 > We fix that be globally escaping \ in our json output, in the generic
 > sanitsing macro.

 > [0] https://nvd.nist.gov/products/cpe/detail/645?namingFormat=2.3&orderBy=CPEURI&keyword=xerces&status=FINAL

I still wonder if it wouldn't be better to not have the backslashes in
the variable and do whatever escaping is needed inside the CVE logic,
but OK - We need a quick fix and this solves it.

Perhaps we should add a gitlab test to verify that we generate valid
json, E.G. by piping it to jq (or similar).

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] package/plg-utils: escape \ in generated legal-info

[Yann E. MORIN]

Peter, All,

On 2021-02-07 10:35 +0100, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> I still wonder if it wouldn't be better to not have the backslashes in
> the variable and do whatever escaping is needed inside the CVE logic,

I think quite the opposite, in fact: we want the CPE_ID value to be
exactly what is in the NVD database without any mangling on our side.

The rules to encode the CPE stuff are non-trivial at least to me),
requiring some escaping/de-escaping in the various formats, and with
different rules for the attributes and their representation

All is defined in NISTIR 7695 [0], in the following chapters:
    5.3.2 - Restrictions on attribute-value strings
    6.2.1 - Syntax for Formatted String Binding

[0] https://doi.org/10.6028/NIST.IR.7695

> but OK - We need a quick fix and this solves it.
> 
> Perhaps we should add a gitlab test to verify that we generate valid
> json, E.G. by piping it to jq (or similar).

Yeah, I'm working on it... But we can't do that in gitlab, because the
output of show-info depends on the selected packages, so it would have
to be done in the autobuilders.

> Committed, thanks.

Thanks.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] package/plg-utils: escape \ in generated legal-info

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Peter, All,
 > On 2021-02-07 10:35 +0100, Peter Korsgaard spake thusly:
 >> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
 >> I still wonder if it wouldn't be better to not have the backslashes in
 >> the variable and do whatever escaping is needed inside the CVE logic,

 > I think quite the opposite, in fact: we want the CPE_ID value to be
 > exactly what is in the NVD database without any mangling on our side.

 > The rules to encode the CPE stuff are non-trivial at least to me),
 > requiring some escaping/de-escaping in the various formats, and with
 > different rules for the attributes and their representation

 > All is defined in NISTIR 7695 [0], in the following chapters:
 >     5.3.2 - Restrictions on attribute-value strings
 >     6.2.1 - Syntax for Formatted String Binding

 > [0] https://doi.org/10.6028/NIST.IR.7695

Ok.

 >> but OK - We need a quick fix and this solves it.
 >> 
 >> Perhaps we should add a gitlab test to verify that we generate valid
 >> json, E.G. by piping it to jq (or similar).

 > Yeah, I'm working on it... But we can't do that in gitlab, because the
 > output of show-info depends on the selected packages, so it would have
 > to be done in the autobuilders.

But that then requires jq on the autobuilder. Can't we just do a 'make
allyespackageconfig' in gitlab?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] package/plg-utils: escape \ in generated legal-info

[Yann E. MORIN]

Peter, All,

On 2021-02-07 13:18 +0100, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
>  >> Perhaps we should add a gitlab test to verify that we generate valid
>  >> json, E.G. by piping it to jq (or similar).
>  > Yeah, I'm working on it... But we can't do that in gitlab, because the
>  > output of show-info depends on the selected packages, so it would have
>  > to be done in the autobuilders.
> But that then requires jq on the autobuilder. Can't we just do a 'make
> allyespackageconfig' in gitlab?

We don't have jq in our current base image either... We can push a new
image, though.

As for the autobuilders, we can build host-jq...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] package/plg-utils: escape \ in generated legal-info

[Yann E. MORIN]

Peter, All,

On 2021-02-07 13:18 +0100, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
>  > Peter, All,
>  > On 2021-02-07 10:35 +0100, Peter Korsgaard spake thusly:
>  >> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
>  >> Perhaps we should add a gitlab test to verify that we generate valid
>  >> json, E.G. by piping it to jq (or similar).
>  > Yeah, I'm working on it... But we can't do that in gitlab, because the
>  > output of show-info depends on the selected packages, so it would have
>  > to be done in the autobuilders.
> But that then requires jq on the autobuilder. Can't we just do a 'make
> allyespackageconfig' in gitlab?

So, allyespackageconfig is not buildable:

    package/luajit/luajit.mk:77: *** Configuration error: both "luajit" and
    "lua" are selected as providers for virtual package "luainterpreter".
    Only one provider can be selected at a time. Please fix your
    configuration.  Stop.

Solving this means reimplementing parts of the logic in the
autobuilders... So I still think it would be better to augment the
autobuilder script to run and validate show-info against a valid
configuration...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] package/plg-utils: escape \ in generated legal-info

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Peter, All,
 > On 2021-02-07 13:18 +0100, Peter Korsgaard spake thusly:
 >> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
 >> > Peter, All,
 >> > On 2021-02-07 10:35 +0100, Peter Korsgaard spake thusly:
 >> >> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
 >> >> Perhaps we should add a gitlab test to verify that we generate valid
 >> >> json, E.G. by piping it to jq (or similar).
 >> > Yeah, I'm working on it... But we can't do that in gitlab, because the
 >> > output of show-info depends on the selected packages, so it would have
 >> > to be done in the autobuilders.
 >> But that then requires jq on the autobuilder. Can't we just do a 'make
 >> allyespackageconfig' in gitlab?

 > So, allyespackageconfig is not buildable:

 >     package/luajit/luajit.mk:77: *** Configuration error: both "luajit" and
 >     "lua" are selected as providers for virtual package "luainterpreter".
 >     Only one provider can be selected at a time. Please fix your
 >     configuration.  Stop.

:/

 > Solving this means reimplementing parts of the logic in the
 > autobuilders... So I still think it would be better to augment the
 > autobuilder script to run and validate show-info against a valid
 > configuration...

That is also fine by me.

-- 
Bye, Peter Korsgaard