CVE-check failing on world with meta-openembedded: diff.gz file

CVE-check failing on world with meta-openembedded: diff.gz file

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

Dear all,
We're running cve-check on a world build containing oe-core, meta-oe and
more. We have an issue with the lockdev recipe
(meta-openembedded/meta-oe/recipes-support/lockdev/lockdev_1.0.3.bb), which
causes a fail like below:

$ bitbake world --runonly=do_cve_check
ERROR: lockdev-1_1.0.3-r0 do_cve_check: File Not found:
<path>lockdev/1_1.0.3-r0/lockdev_1.0.3-1.6.diff
ERROR: lockdev-1_1.0.3-r0 do_cve_check: Failure in searching patches
ERROR: Logfile of failure stored in:
<path>/lockdev/1_1.0.3-r0/temp/log.do_cve_check.8709
ERROR: Task
(<path>/meta-openembedded/meta-oe/recipes-support/lockdev/lockdev_1.0.3.bb:do_cve_check)
failed with exit code '1'

The issue is caused by the fact that lockdev_1.0.3-1.6.diff is missing.
When we look into the recipe, it is downloading lockdev_1.0.3-1.6.diff.gz
file Please note the additional extension.

Stripping the extension comes from oe-core/meta/oe/patch.py, from the
patch_path function, which is figuring out if a file is a patch, and
returning the local path if it is so. However, at the moment when we
do_cve_check, the .gz file is not uncompressed.

I'm wondering how to solve it.
1. Add a dependency to make sure eventual patch files are decompressed
first?
2. Do not consider this as a patch file in the scope of cve-check ? (this
is more a part of the source then an actual patch that might be fixing a
CVE)

This is the only case like that we have in the build. Please note that
removing ".diff" from the extension list in patch_path() is solving the
issue.

Any comments or suggestions?

Kind regards,
Marta

[-- Attachment #2: Type: text/html, Size: 1957 bytes --]

Re: CVE-check failing on world with meta-openembedded: diff.gz file

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 1853 bytes --]

On Fri, Apr 22, 2022 at 11:07 AM Marta Rybczynska <rybczynska@gmail.com>
wrote:

> Dear all,
> We're running cve-check on a world build containing oe-core, meta-oe and
> more. We have an issue with the lockdev recipe
> (meta-openembedded/meta-oe/recipes-support/lockdev/lockdev_1.0.3.bb),
> which causes a fail like below:
>
> $ bitbake world --runonly=do_cve_check
> ERROR: lockdev-1_1.0.3-r0 do_cve_check: File Not found:
> <path>lockdev/1_1.0.3-r0/lockdev_1.0.3-1.6.diff
> ERROR: lockdev-1_1.0.3-r0 do_cve_check: Failure in searching patches
> ERROR: Logfile of failure stored in:
> <path>/lockdev/1_1.0.3-r0/temp/log.do_cve_check.8709
> ERROR: Task
> (<path>/meta-openembedded/meta-oe/recipes-support/lockdev/lockdev_1.0.3.bb:do_cve_check)
> failed with exit code '1'
>
> The issue is caused by the fact that lockdev_1.0.3-1.6.diff is missing.
> When we look into the recipe, it is downloading lockdev_1.0.3-1.6.diff.gz
> file Please note the additional extension.
>
> Stripping the extension comes from oe-core/meta/oe/patch.py, from the
> patch_path function, which is figuring out if a file is a patch, and
> returning the local path if it is so. However, at the moment when we
> do_cve_check, the .gz file is not uncompressed.
>
> I'm wondering how to solve it.
> 1. Add a dependency to make sure eventual patch files are decompressed
> first?
> 2. Do not consider this as a patch file in the scope of cve-check ? (this
> is more a part of the source then an actual patch that might be fixing a
> CVE)
>
> This is the only case like that we have in the build. Please note that
> removing ".diff" from the extension list in patch_path() is solving the
> issue.
>
> Any comments or suggestions?
>
>
Khem,
Any feedback about this? From the history it looks like it was you who
changed the recipe to use the .diff.gz file.

Thanks in advance,
Marta

[-- Attachment #2: Type: text/html, Size: 2510 bytes --]

Re: CVE-check failing on world with meta-openembedded: diff.gz file

[Khem Raj]

Hi Marta

On 4/26/22 5:20 AM, Marta Rybczynska wrote:
> 
> 
> On Fri, Apr 22, 2022 at 11:07 AM Marta Rybczynska <rybczynska@gmail.com 
> <mailto:rybczynska@gmail.com>> wrote:
> 
>     Dear all,
>     We're running cve-check on a world build containing oe-core, meta-oe
>     and more. We have an issue with the lockdev recipe
>     (meta-openembedded/meta-oe/recipes-support/lockdev/lockdev_1.0.3.bb
>     <http://lockdev_1.0.3.bb>), which causes a fail like below:
> 
>     $ bitbake world --runonly=do_cve_check
>     ERROR: lockdev-1_1.0.3-r0 do_cve_check: File Not found:
>     <path>lockdev/1_1.0.3-r0/lockdev_1.0.3-1.6.diff
>     ERROR: lockdev-1_1.0.3-r0 do_cve_check: Failure in searching patches
>     ERROR: Logfile of failure stored in:
>     <path>/lockdev/1_1.0.3-r0/temp/log.do_cve_check.8709
>     ERROR: Task
>     (<path>/meta-openembedded/meta-oe/recipes-support/lockdev/lockdev_1.0.3.bb:do_cve_check)
>     failed with exit code '1'
> 
>     The issue is caused by the fact that lockdev_1.0.3-1.6.diff is
>     missing. When we look into the recipe, it is downloading
>     lockdev_1.0.3-1.6.diff.gz file Please note the additional extension.
> 
>     Stripping the extension comes from oe-core/meta/oe/patch.py, from
>     the patch_path function, which is figuring out if a file is a patch,
>     and returning the local path if it is so. However, at the moment
>     when we do_cve_check, the .gz file is not uncompressed.
> 
>     I'm wondering how to solve it.
>     1. Add a dependency to make sure eventual patch files are
>     decompressed first?


I think this option looks best or perhaps we should drop applying debian 
diff entirely since debian seems to have dropped this package

I think regardless of what we do with this package it seems to be a 
limitation of cve-check process which perhaps should either be fixed or 
documented.

>     2. Do not consider this as a patch file in the scope of cve-check ?
>     (this is more a part of the source then an actual patch that might
>     be fixing a CVE)
> 
>     This is the only case like that we have in the build. Please note
>     that removing ".diff" from the extension list in patch_path() is
>     solving the issue.
> 
>     Any comments or suggestions?
> 
> 
> Khem,
> Any feedback about this? From the history it looks like it was you who 
> changed the recipe to use the .diff.gz file.
> 
> Thanks in advance,
> Marta

Re: CVE-check failing on world with meta-openembedded: diff.gz file

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 2989 bytes --]

On Tue, Apr 26, 2022 at 6:09 PM Khem Raj <raj.khem@gmail.com> wrote:

> Hi Marta
>
> On 4/26/22 5:20 AM, Marta Rybczynska wrote:
> >
> >
> > On Fri, Apr 22, 2022 at 11:07 AM Marta Rybczynska <rybczynska@gmail.com
> > <mailto:rybczynska@gmail.com>> wrote:
> >
> >     Dear all,
> >     We're running cve-check on a world build containing oe-core, meta-oe
> >     and more. We have an issue with the lockdev recipe
> >     (meta-openembedded/meta-oe/recipes-support/lockdev/lockdev_1.0.3.bb
> >     <http://lockdev_1.0.3.bb>), which causes a fail like below:
> >
> >     $ bitbake world --runonly=do_cve_check
> >     ERROR: lockdev-1_1.0.3-r0 do_cve_check: File Not found:
> >     <path>lockdev/1_1.0.3-r0/lockdev_1.0.3-1.6.diff
> >     ERROR: lockdev-1_1.0.3-r0 do_cve_check: Failure in searching patches
> >     ERROR: Logfile of failure stored in:
> >     <path>/lockdev/1_1.0.3-r0/temp/log.do_cve_check.8709
> >     ERROR: Task
> >
>  (<path>/meta-openembedded/meta-oe/recipes-support/lockdev/lockdev_1.0.3.bb:
> do_cve_check)
> >     failed with exit code '1'
> >
> >     The issue is caused by the fact that lockdev_1.0.3-1.6.diff is
> >     missing. When we look into the recipe, it is downloading
> >     lockdev_1.0.3-1.6.diff.gz file Please note the additional extension.
> >
> >     Stripping the extension comes from oe-core/meta/oe/patch.py, from
> >     the patch_path function, which is figuring out if a file is a patch,
> >     and returning the local path if it is so. However, at the moment
> >     when we do_cve_check, the .gz file is not uncompressed.
> >
> >     I'm wondering how to solve it.
> >     1. Add a dependency to make sure eventual patch files are
> >     decompressed first?
>
>
> I think this option looks best or perhaps we should drop applying debian
> diff entirely since debian seems to have dropped this package
>
> I think regardless of what we do with this package it seems to be a
> limitation of cve-check process which perhaps should either be fixed or
> documented.
>
> >     2. Do not consider this as a patch file in the scope of cve-check ?
> >     (this is more a part of the source then an actual patch that might
> >     be fixing a CVE)
> >
> >     This is the only case like that we have in the build. Please note
> >     that removing ".diff" from the extension list in patch_path() is
> >     solving the issue.
> >
> >     Any comments or suggestions?
> >
> >

Adding Ross, Richard and Steve.

I'm wondering if it makes sense to consider .diff.gz (or .patch.gz) files
as patches for
cve-check. They basically come directly from 3rd parties and it is quite
unlikely to expect
them to keep the CVE: tag. All the pieces of documentation I can find
mention also only
.patch files for CVEs, and not .patch.gz.

This is tempting to remove the .gz handling here (for the cve-check) in my
opinion.

Also, since the commit f5f97d33a1703d75b9fd9760f2c7767081538e00, cve-check
depends only on do_fetch.

Any further opinions?

Regards,
Marta

[-- Attachment #2: Type: text/html, Size: 4618 bytes --]

Re: [oe] CVE-check failing on world with meta-openembedded: diff.gz file

[Ross Burton]

On 27 Apr 2022, at 08:14, Marta Rybczynska via lists.openembedded.org <rybczynska=gmail.com@lists.openembedded.org> wrote:
> I'm wondering if it makes sense to consider .diff.gz (or .patch.gz) files as patches for
> cve-check. They basically come directly from 3rd parties and it is quite unlikely to expect
> them to keep the CVE: tag. All the pieces of documentation I can find mention also only
> .patch files for CVEs, and not .patch.gz.
>
> This is tempting to remove the .gz handling here (for the cve-check) in my opinion.
>
> Also, since the commit f5f97d33a1703d75b9fd9760f2c7767081538e00, cve-check
> depends only on do_fetch.

The patch being a .patch.gz isn’t entirely relevant, it’s the fact that it’s a remote http: patch not a local file: patch which is causing the problem. The code uses the localpath, which only exists for remote URL after do_unpack.

There are three alternatives here:
1) Only consider local patches.  Any remote patches won’t be scanned, but they don’t work anyway right now. This might mean the dependency on do_fetch can be dropped to speed up checking even further.
2) Change the task dependency to be on do_unpack instead of do_fetch. This will slow down processing if a build hasn’t already happened as tarballs will be unpacked, but remote files will be present for scanning then.
3) Try to be clever and manually call unpack on remote files.  More complicated but preserves the speed.

I’m actually undecided over what the best solution is. Clearly we need some test cases for this code too.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [oe] CVE-check failing on world with meta-openembedded: diff.gz file

[Richard Purdie]

On Wed, 2022-04-27 at 10:32 +0000, Ross Burton wrote:
> On 27 Apr 2022, at 08:14, Marta Rybczynska via lists.openembedded.org
> <rybczynska=gmail.com@lists.openembedded.org> wrote:
> > I'm wondering if it makes sense to consider .diff.gz (or .patch.gz) files as
> > patches for
> > cve-check. They basically come directly from 3rd parties and it is quite
> > unlikely to expect
> > them to keep the CVE: tag. All the pieces of documentation I can find
> > mention also only
> > .patch files for CVEs, and not .patch.gz.
> > 
> > This is tempting to remove the .gz handling here (for the cve-check) in my
> > opinion.
> > 
> > Also, since the commit f5f97d33a1703d75b9fd9760f2c7767081538e00, cve-check
> > depends only on do_fetch.
> 
> The patch being a .patch.gz isn’t entirely relevant, it’s the fact that it’s a
> remote http: patch not a local file: patch which is causing the problem. The
> code uses the localpath, which only exists for remote URL after do_unpack.
> 
> There are three alternatives here:
> 1) Only consider local patches.  Any remote patches won’t be scanned, but they
> don’t work anyway right now. This might mean the dependency on do_fetch can be
> dropped to speed up checking even further.
> 2) Change the task dependency to be on do_unpack instead of do_fetch. This
> will slow down processing if a build hasn’t already happened as tarballs will
> be unpacked, but remote files will be present for scanning then.
> 3) Try to be clever and manually call unpack on remote files.  More
> complicated but preserves the speed.
> 
> I’m actually undecided over what the best solution is. Clearly we need some
> test cases for this code too.

I think the deciding factor may be that most remote patches probably don't have
the information we're looking for in them anyway?

Cheers,

Richard

Re: [oe] CVE-check failing on world with meta-openembedded: diff.gz file

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]

On Wed, Apr 27, 2022 at 3:33 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2022-04-27 at 10:32 +0000, Ross Burton wrote:
> > On 27 Apr 2022, at 08:14, Marta Rybczynska via lists.openembedded.org
> > <rybczynska=gmail.com@lists.openembedded.org> wrote:
> > > I'm wondering if it makes sense to consider .diff.gz (or .patch.gz)
> files as
> > > patches for
> > > cve-check. They basically come directly from 3rd parties and it is
> quite
> > > unlikely to expect
> > > them to keep the CVE: tag. All the pieces of documentation I can find
> > > mention also only
> > > .patch files for CVEs, and not .patch.gz.
> > >
> > > This is tempting to remove the .gz handling here (for the cve-check)
> in my
> > > opinion.
> > >
> > > Also, since the commit f5f97d33a1703d75b9fd9760f2c7767081538e00,
> cve-check
> > > depends only on do_fetch.
> >
> > The patch being a .patch.gz isn’t entirely relevant, it’s the fact that
> it’s a
> > remote http: patch not a local file: patch which is causing the problem.
> The
> > code uses the localpath, which only exists for remote URL after
> do_unpack.
> >
> > There are three alternatives here:
> > 1) Only consider local patches.  Any remote patches won’t be scanned,
> but they
> > don’t work anyway right now. This might mean the dependency on do_fetch
> can be
> > dropped to speed up checking even further.
> > 2) Change the task dependency to be on do_unpack instead of do_fetch.
> This
> > will slow down processing if a build hasn’t already happened as tarballs
> will
> > be unpacked, but remote files will be present for scanning then.
> > 3) Try to be clever and manually call unpack on remote files.  More
> > complicated but preserves the speed.
> >
> > I’m actually undecided over what the best solution is. Clearly we need
> some
> > test cases for this code too.
>
> I think the deciding factor may be that most remote patches probably don't
> have
> the information we're looking for in them anyway?


I think covering more cases has value if it can be done in a reasonable
manner without punishing the system too much otherwise a diagnostic message
should definitely be an improvement which will nudge the user to perhaps
change the patch mechanism like we did for lockdev

>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 3265 bytes --]