Problems with sstate-cache in Artifactory

Problems with sstate-cache in Artifactory

[Ankur Tyagi]

Hi,

I am trying to setup sstate-cache in Artifactory so that multiple
Linux host systems can share it.

Once build finished in one Linux machine, I uploaded all of
sstate-cache directory to Artifactory

Then I configured sstate mirrors in local.conf
SSTATE_MIRRORS ?= "\
file://.* https://artifactory.com/yocto/sstate-cache/PATH;downloadfilename=PATH
\
"
But when the build started, it didn't used anything from the sstate
mirror and ended up building everything

Am I missing something?

Thanks
Ankur

Re: Problems with sstate-cache in Artifactory

[Burton, Ross]

Check the log for the a particular setscene task and it will show what
requests it made, and if they failed.

Ross

On Wed, 3 Jul 2019 at 11:09, Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
>
> Hi,
>
> I am trying to setup sstate-cache in Artifactory so that multiple
> Linux host systems can share it.
>
> Once build finished in one Linux machine, I uploaded all of
> sstate-cache directory to Artifactory
>
> Then I configured sstate mirrors in local.conf
> SSTATE_MIRRORS ?= "\
> file://.* https://artifactory.com/yocto/sstate-cache/PATH;downloadfilename=PATH
> \
> "
> But when the build started, it didn't used anything from the sstate
> mirror and ended up building everything
>
> Am I missing something?
>
> Thanks
> Ankur
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Problems with sstate-cache in Artifactory

[Ankur Tyagi]

Hi Ross,

I can't see any "log.do_setscene" inside the tmp directory.
When I start an image build, it does say executing setscene task but
then no setscene task log for any recipe

Initialising tasks: 100%
|############################################################################################################################################|
Time: 0:00:01
Checking sstate mirror object availability: 100%
|####################################################################################################################|
Time: 0:00:02
Sstate summary: Wanted 484 Found 0 Missed 968 Current 0 (0% match, 0% complete)
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks

Regards
Ankur



On Thu, Jul 4, 2019 at 12:04 AM Burton, Ross <ross.burton@intel.com> wrote:
>
> Check the log for the a particular setscene task and it will show what
> requests it made, and if they failed.
>
> Ross
>
> On Wed, 3 Jul 2019 at 11:09, Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
> >
> > Hi,
> >
> > I am trying to setup sstate-cache in Artifactory so that multiple
> > Linux host systems can share it.
> >
> > Once build finished in one Linux machine, I uploaded all of
> > sstate-cache directory to Artifactory
> >
> > Then I configured sstate mirrors in local.conf
> > SSTATE_MIRRORS ?= "\
> > file://.* https://artifactory.com/yocto/sstate-cache/PATH;downloadfilename=PATH
> > \
> > "
> > But when the build started, it didn't used anything from the sstate
> > mirror and ended up building everything
> >
> > Am I missing something?
> >
> > Thanks
> > Ankur
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Problems with sstate-cache in Artifactory

[Ankur Tyagi]

Hi Chris,

Thanks for your pointers, I will try them.

Regarding using curl for sstate fetch, how to set SSTATE_MIRRORS ?
Below is my current configuration

SSTATE_MIRRORS ?= "\
file://.* https://artifactory.com/yocto/sstate-cache/PATH;downloadfilename=PATH
\
"
Thanks
Ankur

On Tue, Jul 9, 2019 at 3:37 AM chris.laplante@agilent.com
<chris.laplante@agilent.com> wrote:
>
> > I can't see any "log.do_setscene" inside the tmp directory.
> > When I start an image build, it does say executing setscene task but
> > then no setscene task log for any recipe
> >
> > Initialising tasks: 100%
> > |##############################################################################################################
> > ##############################|
> > Time: 0:00:01
> > Checking sstate mirror object availability: 100%
> > |##############################################################################################################
> > ######|
> > Time: 0:00:02
> > Sstate summary: Wanted 484 Found 0 Missed 968 Current 0 (0% match, 0% complete)
> > NOTE: Executing SetScene Tasks
> > NOTE: Executing RunQueue Tasks
>
> We have Artifactory serving our sstate as well, so here's some tips.
>
> 1. Try seeing if you can fetch some sstate using curl directly. If you are authenticating to the Artifactory server using a .netrc server, make sure to pass "--netrc" to curl.
> 2. You can diagnose the sstate mirror by passing -DDD to bitbake. It will spit out a ton of debug messages, so you might want to redirect it all to a file. E.g.:
>
> bitbake something -DDD >out.txt 2>&1
>
> Start looking after the line that begins "DEBUG: SState using premirror of". It will dump out the URLs that it tries to fetch.
>
> 3. Within Artifactory, you can check the access log by going to: Admin > System Logs > access.log
>
> Look for "DENIED LOGIN" errors.
>
> Chris

Re: Problems with sstate-cache in Artifactory

[Ankur Tyagi]

Hi Chris,

1) I am able to fetch sstate using curl, wget

2) Looking in bitbake -DDD output, url is correct as I can use that
url to download it using curl, wget
DEBUG: SState: Attempting to fetch
file://5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
DEBUG: Searching for
5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
in paths:
    /home/nws/wsl/arago/downstream/ggl-oe-layersetup/build/sstate-cache
DEBUG: Defaulting to
/home/nws/wsl/arago/downstream/ggl-oe-layersetup/build/sstate-cache/5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
for 5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
DEBUG: Testing URL
file://5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
DEBUG: For url ['file', '',
'5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz',
'', '', OrderedDict()] comparing ['file', '', '.*', '', '',
OrderedDict()] to ['https', 'artifactory.local.gallagher.io',
'/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/PATH', '', '',
OrderedDict([('downloadfilename', 'PATH')])]
DEBUG: For url file://5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
returning https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti-staging%3Awsl-linux-gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz;downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found

Regards
Ankur

On Tue, Jul 9, 2019 at 9:16 AM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
>
> Hi Chris,
>
> Thanks for your pointers, I will try them.
>
> Regarding using curl for sstate fetch, how to set SSTATE_MIRRORS ?
> Below is my current configuration
>
> SSTATE_MIRRORS ?= "\
> file://.* https://artifactory.com/yocto/sstate-cache/PATH;downloadfilename=PATH
> \
> "
> Thanks
> Ankur
>
> On Tue, Jul 9, 2019 at 3:37 AM chris.laplante@agilent.com
> <chris.laplante@agilent.com> wrote:
> >
> > > I can't see any "log.do_setscene" inside the tmp directory.
> > > When I start an image build, it does say executing setscene task but
> > > then no setscene task log for any recipe
> > >
> > > Initialising tasks: 100%
> > > |##############################################################################################################
> > > ##############################|
> > > Time: 0:00:01
> > > Checking sstate mirror object availability: 100%
> > > |##############################################################################################################
> > > ######|
> > > Time: 0:00:02
> > > Sstate summary: Wanted 484 Found 0 Missed 968 Current 0 (0% match, 0% complete)
> > > NOTE: Executing SetScene Tasks
> > > NOTE: Executing RunQueue Tasks
> >
> > We have Artifactory serving our sstate as well, so here's some tips.
> >
> > 1. Try seeing if you can fetch some sstate using curl directly. If you are authenticating to the Artifactory server using a .netrc server, make sure to pass "--netrc" to curl.
> > 2. You can diagnose the sstate mirror by passing -DDD to bitbake. It will spit out a ton of debug messages, so you might want to redirect it all to a file. E.g.:
> >
> > bitbake something -DDD >out.txt 2>&1
> >
> > Start looking after the line that begins "DEBUG: SState using premirror of". It will dump out the URLs that it tries to fetch.
> >
> > 3. Within Artifactory, you can check the access log by going to: Admin > System Logs > access.log
> >
> > Look for "DENIED LOGIN" errors.
> >
> > Chris

Re: Problems with sstate-cache in Artifactory

[chris.laplante]

> DEBUG: For url file://5c/sstate:u-boot-ti-staging:wsl-linux-
> gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
> returning https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti-
> staging%3Awsl-linux-
> gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz;
> downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux-
> gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
> DEBUG: checkstatus: trying again
> DEBUG: checkstatus: trying again
> DEBUG: checkstatus: trying again
> DEBUG: checkstatus: trying again
> DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found

Is this mapping correct? Does using curl to download https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti-staging%3Awsl-linux-gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz;downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz work?

Chris

Re: Problems with sstate-cache in Artifactory

[Ankur Tyagi]

On Wed, Jul 10, 2019 at 2:28 AM chris.laplante@agilent.com
<chris.laplante@agilent.com> wrote:
>
> > DEBUG: For url file://5c/sstate:u-boot-ti-staging:wsl-linux-
> > gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
> > returning https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti-
> > staging%3Awsl-linux-
> > gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz;
> > downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux-
> > gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
> > DEBUG: checkstatus: trying again
> > DEBUG: checkstatus: trying again
> > DEBUG: checkstatus: trying again
> > DEBUG: checkstatus: trying again
> > DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found
>
> Is this mapping correct? Does using curl to download https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti-staging%3Awsl-linux-gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz;downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz work?
>

Yes, curl/wget can download it. What wrong do you suspect here?

Thanks
Ankur

> Chris

Re: Problems with sstate-cache in Artifactory

[Ankur Tyagi]

Hi Chris,

I tried uploading sstate-cache again in Artifactory but this time used
jfrog-cli (https://www.jfrog.com/confluence/display/CLI/CLI+for+JFrog+Artifactory)
instead of curl and now it is working fine.

regards
Ankur

On Wed, Jul 10, 2019 at 1:39 PM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
>
> On Wed, Jul 10, 2019 at 2:28 AM chris.laplante@agilent.com
> <chris.laplante@agilent.com> wrote:
> >
> > > DEBUG: For url file://5c/sstate:u-boot-ti-staging:wsl-linux-
> > > gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
> > > returning https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti-
> > > staging%3Awsl-linux-
> > > gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz;
> > > downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux-
> > > gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz
> > > DEBUG: checkstatus: trying again
> > > DEBUG: checkstatus: trying again
> > > DEBUG: checkstatus: trying again
> > > DEBUG: checkstatus: trying again
> > > DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found
> >
> > Is this mapping correct? Does using curl to download https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti-staging%3Awsl-linux-gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz;downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz work?
> >
>
> Yes, curl/wget can download it. What wrong do you suspect here?
>
> Thanks
> Ankur
>
> > Chris