[opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[Stefan Agner]

From: Stefan Agner <stefan.agner@toradex.com>

When using sstate, two parallel builds can produce two packages
with the same mtime but different checksums. When later one of
those two builds fetches the others ipk, the package index does
not get udpated properly (since mtime matches). This ends up with
messages such as:
  Downloading file:/../tmp/work/../image/...ipk.
  Removing corrupt package file /../sysroot/../var/cache/opkg/volatile/...ipk

However, in that case, ctime is different. Use ctime instead of
mtime to prevent failures like this.

Suggested-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
---
This addresses the issue discussed here:
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156348.html

 opkg-make-index | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index 3227fc0..db7bf64 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -115,12 +115,12 @@ for abspath in files:
      pkg = None
      fnameStat = os.stat(abspath)
      if filename in old_pkg_hash:
-          if filename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[filename]:
+          if filename in pkgsStamps and int(fnameStat.st_ctime) == pkgsStamps[filename]:
             if (verbose):
                sys.stderr.write("Found %s in Packages\n" % (filename,))
             pkg = old_pkg_hash[filename]
           else:
-               sys.stderr.write("Found %s in Packages, but mtime differs - re-reading\n" % (filename,))
+               sys.stderr.write("Found %s in Packages, but ctime differs - re-reading\n" % (filename,))
 
      if not pkg:
           if (verbose):
@@ -137,7 +137,7 @@ for abspath in files:
      else:
           old_filename = ""
      s = packages.add_package(pkg, opt_a)
-     pkgsStamps[filename] = fnameStat.st_mtime
+     pkgsStamps[filename] = fnameStat.st_ctime
      if s == 0:
           if old_filename:
                # old package was displaced by newer
-- 
2.13.6

[opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[Stefan Agner]

From: Stefan Agner <stefan.agner@toradex.com>

When using sstate, two parallel builds can produce two packages
with the same mtime but different checksums. When later one of
those two builds fetches the others ipk, the package index does
not get udpated properly (since mtime matches). This ends up with
messages such as:
  Downloading file:/../tmp/work/../image/...ipk.
  Removing corrupt package file /../sysroot/../var/cache/opkg/volatile/...ipk

However, in that case, ctime is different. Use ctime instead of
mtime to prevent failures like this.

Suggested-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
---
This addresses the issue discussed here:
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156348.html

 opkg-make-index | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index 3227fc0..db7bf64 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -115,12 +115,12 @@ for abspath in files:
      pkg = None
      fnameStat = os.stat(abspath)
      if filename in old_pkg_hash:
-          if filename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[filename]:
+          if filename in pkgsStamps and int(fnameStat.st_ctime) == pkgsStamps[filename]:
             if (verbose):
                sys.stderr.write("Found %s in Packages\n" % (filename,))
             pkg = old_pkg_hash[filename]
           else:
-               sys.stderr.write("Found %s in Packages, but mtime differs - re-reading\n" % (filename,))
+               sys.stderr.write("Found %s in Packages, but ctime differs - re-reading\n" % (filename,))
 
      if not pkg:
           if (verbose):
@@ -137,7 +137,7 @@ for abspath in files:
      else:
           old_filename = ""
      s = packages.add_package(pkg, opt_a)
-     pkgsStamps[filename] = fnameStat.st_mtime
+     pkgsStamps[filename] = fnameStat.st_ctime
      if s == 0:
           if old_filename:
                # old package was displaced by newer
-- 
2.13.6

Re: [opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[richard.purdie]

On Fri, 2018-10-19 at 17:38 +0200, Stefan Agner wrote:
> From: Stefan Agner <stefan.agner@toradex.com>
> 
> When using sstate, two parallel builds can produce two packages
> with the same mtime but different checksums. When later one of
> those two builds fetches the others ipk, the package index does
> not get udpated properly (since mtime matches). This ends up with
> messages such as:
>   Downloading file:/../tmp/work/../image/...ipk.
>   Removing corrupt package file
> /../sysroot/../var/cache/opkg/volatile/...ipk
> 
> However, in that case, ctime is different. Use ctime instead of
> mtime to prevent failures like this.
> 
> Suggested-by: Khem Raj <raj.khem@gmail.com>
> Signed-off-by: Stefan Agner <stefan.agner@toradex.com>

Acked-by: Richard Purdie <richard.purdie@linuxfoundation.org>

(I've been following this problem and this makes sense to me)

> ---
> This addresses the issue discussed here:
> 
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156348.html
> 
>  opkg-make-index | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/opkg-make-index b/opkg-make-index
> index 3227fc0..db7bf64 100755
> --- a/opkg-make-index
> +++ b/opkg-make-index
> @@ -115,12 +115,12 @@ for abspath in files:
>       pkg = None
>       fnameStat = os.stat(abspath)
>       if filename in old_pkg_hash:
> -          if filename in pkgsStamps and int(fnameStat.st_mtime) ==
> pkgsStamps[filename]:
> +          if filename in pkgsStamps and int(fnameStat.st_ctime) ==
> pkgsStamps[filename]:
>              if (verbose):
>                 sys.stderr.write("Found %s in Packages\n" %
> (filename,))
>              pkg = old_pkg_hash[filename]
>            else:
> -               sys.stderr.write("Found %s in Packages, but mtime
> differs - re-reading\n" % (filename,))
> +               sys.stderr.write("Found %s in Packages, but ctime
> differs - re-reading\n" % (filename,))
>  
>       if not pkg:
>            if (verbose):
> @@ -137,7 +137,7 @@ for abspath in files:
>       else:
>            old_filename = ""
>       s = packages.add_package(pkg, opt_a)
> -     pkgsStamps[filename] = fnameStat.st_mtime
> +     pkgsStamps[filename] = fnameStat.st_ctime
>       if s == 0:
>            if old_filename:
>                 # old package was displaced by newer

Re: [opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[Khem Raj]

On Sat, Oct 20, 2018 at 10:16 AM <richard.purdie@linuxfoundation.org> wrote:
>
> On Fri, 2018-10-19 at 17:38 +0200, Stefan Agner wrote:
> > From: Stefan Agner <stefan.agner@toradex.com>
> >
> > When using sstate, two parallel builds can produce two packages
> > with the same mtime but different checksums. When later one of
> > those two builds fetches the others ipk, the package index does
> > not get udpated properly (since mtime matches). This ends up with
> > messages such as:
> >   Downloading file:/../tmp/work/../image/...ipk.
> >   Removing corrupt package file
> > /../sysroot/../var/cache/opkg/volatile/...ipk
> >
> > However, in that case, ctime is different. Use ctime instead of
> > mtime to prevent failures like this.
> >
> > Suggested-by: Khem Raj <raj.khem@gmail.com>
> > Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
>
> Acked-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>

Yep, thats for following up with implementing it.

Acked-by: Khem Raj <raj.khem@gmail.com>

> (I've been following this problem and this makes sense to me)
>
> > ---
> > This addresses the issue discussed here:
> >
> http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156348.html
> >
> >  opkg-make-index | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/opkg-make-index b/opkg-make-index
> > index 3227fc0..db7bf64 100755
> > --- a/opkg-make-index
> > +++ b/opkg-make-index
> > @@ -115,12 +115,12 @@ for abspath in files:
> >       pkg = None
> >       fnameStat = os.stat(abspath)
> >       if filename in old_pkg_hash:
> > -          if filename in pkgsStamps and int(fnameStat.st_mtime) ==
> > pkgsStamps[filename]:
> > +          if filename in pkgsStamps and int(fnameStat.st_ctime) ==
> > pkgsStamps[filename]:
> >              if (verbose):
> >                 sys.stderr.write("Found %s in Packages\n" %
> > (filename,))
> >              pkg = old_pkg_hash[filename]
> >            else:
> > -               sys.stderr.write("Found %s in Packages, but mtime
> > differs - re-reading\n" % (filename,))
> > +               sys.stderr.write("Found %s in Packages, but ctime
> > differs - re-reading\n" % (filename,))
> >
> >       if not pkg:
> >            if (verbose):
> > @@ -137,7 +137,7 @@ for abspath in files:
> >       else:
> >            old_filename = ""
> >       s = packages.add_package(pkg, opt_a)
> > -     pkgsStamps[filename] = fnameStat.st_mtime
> > +     pkgsStamps[filename] = fnameStat.st_ctime
> >       if s == 0:
> >            if old_filename:
> >                 # old package was displaced by newer
>

Re: [opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[Khem Raj]

On Sat, Oct 20, 2018 at 10:16 AM <richard.purdie@linuxfoundation.org> wrote:
>
> On Fri, 2018-10-19 at 17:38 +0200, Stefan Agner wrote:
> > From: Stefan Agner <stefan.agner@toradex.com>
> >
> > When using sstate, two parallel builds can produce two packages
> > with the same mtime but different checksums. When later one of
> > those two builds fetches the others ipk, the package index does
> > not get udpated properly (since mtime matches). This ends up with
> > messages such as:
> >   Downloading file:/../tmp/work/../image/...ipk.
> >   Removing corrupt package file
> > /../sysroot/../var/cache/opkg/volatile/...ipk
> >
> > However, in that case, ctime is different. Use ctime instead of
> > mtime to prevent failures like this.
> >
> > Suggested-by: Khem Raj <raj.khem@gmail.com>
> > Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
>
> Acked-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>

Yep, thats for following up with implementing it.

Acked-by: Khem Raj <raj.khem@gmail.com>

> (I've been following this problem and this makes sense to me)
>
> > ---
> > This addresses the issue discussed here:
> >
> http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156348.html
> >
> >  opkg-make-index | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/opkg-make-index b/opkg-make-index
> > index 3227fc0..db7bf64 100755
> > --- a/opkg-make-index
> > +++ b/opkg-make-index
> > @@ -115,12 +115,12 @@ for abspath in files:
> >       pkg = None
> >       fnameStat = os.stat(abspath)
> >       if filename in old_pkg_hash:
> > -          if filename in pkgsStamps and int(fnameStat.st_mtime) ==
> > pkgsStamps[filename]:
> > +          if filename in pkgsStamps and int(fnameStat.st_ctime) ==
> > pkgsStamps[filename]:
> >              if (verbose):
> >                 sys.stderr.write("Found %s in Packages\n" %
> > (filename,))
> >              pkg = old_pkg_hash[filename]
> >            else:
> > -               sys.stderr.write("Found %s in Packages, but mtime
> > differs - re-reading\n" % (filename,))
> > +               sys.stderr.write("Found %s in Packages, but ctime
> > differs - re-reading\n" % (filename,))
> >
> >       if not pkg:
> >            if (verbose):
> > @@ -137,7 +137,7 @@ for abspath in files:
> >       else:
> >            old_filename = ""
> >       s = packages.add_package(pkg, opt_a)
> > -     pkgsStamps[filename] = fnameStat.st_mtime
> > +     pkgsStamps[filename] = fnameStat.st_ctime
> >       if s == 0:
> >            if old_filename:
> >                 # old package was displaced by newer
>

Re: [opkg-devel] [opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[Alejandro Del Castillo]

makes sense, sounds like this is going to fix a bunch of nasty 
intermittent failures, thanks!

merged

On 10/19/18 10:38 AM, Stefan Agner wrote:
> From: Stefan Agner <stefan.agner@toradex.com>
> 
> When using sstate, two parallel builds can produce two packages
> with the same mtime but different checksums. When later one of
> those two builds fetches the others ipk, the package index does
> not get udpated properly (since mtime matches). This ends up with
> messages such as:
>    Downloading file:/../tmp/work/../image/...ipk.
>    Removing corrupt package file /../sysroot/../var/cache/opkg/volatile/...ipk
> 
> However, in that case, ctime is different. Use ctime instead of
> mtime to prevent failures like this.
> 
> Suggested-by: Khem Raj <raj.khem@gmail.com>
> Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
> ---
> This addresses the issue discussed here:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openembedded.org_pipermail_openembedded-2Dcore_2018-2DOctober_156348.html&d=DwIBaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=wNcrL2akRn6jfxhHaKavUrJB_C9JAMXtynjLd8ZzgXQ&m=Innit37H69hUyZPGuuhwO6R5CbUNNTfXQwxbqsEA2NE&s=oFvqASrFTgasDqZ901HeIBFSsf6Cn4FcBieOOBU4MdI&e=
> 
>   opkg-make-index | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/opkg-make-index b/opkg-make-index
> index 3227fc0..db7bf64 100755
> --- a/opkg-make-index
> +++ b/opkg-make-index
> @@ -115,12 +115,12 @@ for abspath in files:
>        pkg = None
>        fnameStat = os.stat(abspath)
>        if filename in old_pkg_hash:
> -          if filename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[filename]:
> +          if filename in pkgsStamps and int(fnameStat.st_ctime) == pkgsStamps[filename]:
>               if (verbose):
>                  sys.stderr.write("Found %s in Packages\n" % (filename,))
>               pkg = old_pkg_hash[filename]
>             else:
> -               sys.stderr.write("Found %s in Packages, but mtime differs - re-reading\n" % (filename,))
> +               sys.stderr.write("Found %s in Packages, but ctime differs - re-reading\n" % (filename,))
>   
>        if not pkg:
>             if (verbose):
> @@ -137,7 +137,7 @@ for abspath in files:
>        else:
>             old_filename = ""
>        s = packages.add_package(pkg, opt_a)
> -     pkgsStamps[filename] = fnameStat.st_mtime
> +     pkgsStamps[filename] = fnameStat.st_ctime
>        if s == 0:
>             if old_filename:
>                  # old package was displaced by newer
> 

-- 
Cheers,

Alejandro

Re: [opkg-devel] [opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[Stefan Agner]

Hi Alejandro,

On 22.10.2018 16:45, Alejandro Del Castillo wrote:
> makes sense, sounds like this is going to fix a bunch of nasty 
> intermittent failures, thanks!
> 
> merged

Thanks for merging!

With the merge in opkg-utils this is not yet actively used in OE of
course. So my question: Is there a new release of opkg-utils planned
anytime soon? Or should that be added as a patch in the OE recipe for
now?

--
Stefan

> 
> On 10/19/18 10:38 AM, Stefan Agner wrote:
>> From: Stefan Agner <stefan.agner@toradex.com>
>>
>> When using sstate, two parallel builds can produce two packages
>> with the same mtime but different checksums. When later one of
>> those two builds fetches the others ipk, the package index does
>> not get udpated properly (since mtime matches). This ends up with
>> messages such as:
>>    Downloading file:/../tmp/work/../image/...ipk.
>>    Removing corrupt package file /../sysroot/../var/cache/opkg/volatile/...ipk
>>
>> However, in that case, ctime is different. Use ctime instead of
>> mtime to prevent failures like this.
>>
>> Suggested-by: Khem Raj <raj.khem@gmail.com>
>> Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
>> ---
>> This addresses the issue discussed here:
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openembedded.org_pipermail_openembedded-2Dcore_2018-2DOctober_156348.html&d=DwIBaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=wNcrL2akRn6jfxhHaKavUrJB_C9JAMXtynjLd8ZzgXQ&m=Innit37H69hUyZPGuuhwO6R5CbUNNTfXQwxbqsEA2NE&s=oFvqASrFTgasDqZ901HeIBFSsf6Cn4FcBieOOBU4MdI&e=
>>
>>   opkg-make-index | 6 +++---
>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/opkg-make-index b/opkg-make-index
>> index 3227fc0..db7bf64 100755
>> --- a/opkg-make-index
>> +++ b/opkg-make-index
>> @@ -115,12 +115,12 @@ for abspath in files:
>>        pkg = None
>>        fnameStat = os.stat(abspath)
>>        if filename in old_pkg_hash:
>> -          if filename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[filename]:
>> +          if filename in pkgsStamps and int(fnameStat.st_ctime) == pkgsStamps[filename]:
>>               if (verbose):
>>                  sys.stderr.write("Found %s in Packages\n" % (filename,))
>>               pkg = old_pkg_hash[filename]
>>             else:
>> -               sys.stderr.write("Found %s in Packages, but mtime differs - re-reading\n" % (filename,))
>> +               sys.stderr.write("Found %s in Packages, but ctime differs - re-reading\n" % (filename,))
>>
>>        if not pkg:
>>             if (verbose):
>> @@ -137,7 +137,7 @@ for abspath in files:
>>        else:
>>             old_filename = ""
>>        s = packages.add_package(pkg, opt_a)
>> -     pkgsStamps[filename] = fnameStat.st_mtime
>> +     pkgsStamps[filename] = fnameStat.st_ctime
>>        if s == 0:
>>             if old_filename:
>>                  # old package was displaced by newer
>>
> 
> -- 
> Cheers,
> 
> Alejandro

Re: [opkg-devel] [opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[Alejandro Del Castillo]

On 11/7/18 6:07 AM, Stefan Agner wrote:
> Hi Alejandro,
> 
> On 22.10.2018 16:45, Alejandro Del Castillo wrote:
>> makes sense, sounds like this is going to fix a bunch of nasty
>> intermittent failures, thanks!
>>
>> merged
> 
> Thanks for merging!

np

> With the merge in opkg-utils this is not yet actively used in OE of
> course. So my question: Is there a new release of opkg-utils planned
> anytime soon? Or should that be added as a patch in the OE recipe for
> now?

I plan to release opkg & opkg-utils, version 0.4.0 on mid-December. 
Maybe its worth adding the patch now?

-- 
Cheers,

Alejandro