externalsrc + sstate why is not compatible?

externalsrc + sstate why is not compatible?

[Koehler, Yannick]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

In our situation, we have many packages that are in-house, as such we use a local folder for the several in-house modules.  To do so, we need to use externalsrc to point to the local code so that the git repo contains both the code + recipe, instead of 20 repos (1 per packages) and 1 more for yocto recipe which complicates things when you want to submit a change.


Yet, the externalsrc disable the setscene tasks and set the BB_DONTCACHE variable.  I altered the script to remove those but then a change to the externalsrc folder is not detected. I wonder if it is because the S variable is set past the sstate algorithm and is then unable to consider the externalsrc folder as the real source location.  Any expert on this matter that can guide me to either make sstate works for external src or teach me how to have code + recipe in a single git repo.  I am certainly not the only case using such a pattern.


--

Yannick Koehler

[-- Attachment #2: Type: text/html, Size: 1615 bytes --]

Re: externalsrc + sstate why is not compatible?

[Alexander Kanavin]

On 12/06/2017 04:15 AM, Koehler, Yannick wrote:
> In our situation, we have many packages that are in-house, as such we
>  use a local folder for the several in-house modules.  To do so, we
> need to use externalsrc to point to the local code so that the git
> repo contains both the code + recipe, instead of 20 repos (1 per
> packages) and 1 more for yocto recipe which complicates things when
> you want to submit a change.

You don't have to have 20 repos. You can place the modules into a single
repo with subdirs.

> Yet, the externalsrc disable the setscene tasks and set the
> BB_DONTCACHE variable.  I altered the script to remove those but then
> a change to the externalsrc folder is not detected. I wonder if it is
> because the S variable is set past the sstate algorithm and is then
> unable to consider the externalsrc folder as the real source
> location.  Any expert on this matter that can guide me to either make
> sstate works for external src or teach me how to have code + recipe
> in a single git repo.  I am certainly not the only case using such a
> pattern.

The bitbake documentation claims you can specify a directory in the 
file: fetcher, and then all of it will be unpacked to workdir. I've 
never seen or tried it myself, but maybe you can investigate in that 
direction, and make it work if it doesn't.

Alex

Re: externalsrc + sstate why is not compatible?

[Koehler, Yannick]

When you say "subdirs", are you referring to submodules or something along those line?  We have used attempted to use submodules and it still required 20 + 1 repo and the overhead is similar to yocto + 20 repo but then you have yocto + 20 repos + 1 main repos, as such, this is not acceptable for us.  We also looked at other combination of multiple repos supported or in work for git and none are satisfying or supported in a way I could propose to use it in our project.

If you mean 1 repo, which subdirs representing each package, I would be interested, but how would it work with yocto SRC_URI?

In regards to file fetcher, I will go check the code, I thought the unpack would only occurs for tarball, not subdir.

--
Yannick Koehler

-----Message d'origine-----
De : Alexander Kanavin [mailto:alexander.kanavin@linux.intel.com] 
Envoyé : 6 décembre 2017 03:47
À : Koehler, Yannick <yannick.koehler@hpe.com>; yocto@yoctoproject.org
Objet : Re: [yocto] externalsrc + sstate why is not compatible?

On 12/06/2017 04:15 AM, Koehler, Yannick wrote:
> In our situation, we have many packages that are in-house, as such we  
> use a local folder for the several in-house modules.  To do so, we 
> need to use externalsrc to point to the local code so that the git 
> repo contains both the code + recipe, instead of 20 repos (1 per
> packages) and 1 more for yocto recipe which complicates things when 
> you want to submit a change.

You don't have to have 20 repos. You can place the modules into a single repo with subdirs.

> Yet, the externalsrc disable the setscene tasks and set the 
> BB_DONTCACHE variable.  I altered the script to remove those but then 
> a change to the externalsrc folder is not detected. I wonder if it is 
> because the S variable is set past the sstate algorithm and is then 
> unable to consider the externalsrc folder as the real source location.  
> Any expert on this matter that can guide me to either make sstate 
> works for external src or teach me how to have code + recipe in a 
> single git repo.  I am certainly not the only case using such a 
> pattern.

The bitbake documentation claims you can specify a directory in the
file: fetcher, and then all of it will be unpacked to workdir. I've never seen or tried it myself, but maybe you can investigate in that direction, and make it work if it doesn't.

Alex

Re: externalsrc + sstate why is not compatible?

[Marcelo E. Magallon]

On Wed, Dec 06, 2017 at 02:13:26PM +0000, Koehler, Yannick wrote:

> In regards to file fetcher, I will go check the code, I thought the
> unpack would only occurs for tarball, not subdir.

If you have:

	SRC_URI := "file://some-dir/"

and your tree looks somewhat like this:

	.git
	poky
	src/some-dir

and you set things up so that bitbake will look in src/ (set FILESPATH),
then it will copy "some-dir" to $WORKDIR/some-dir, and you can point S
there.

This satisfies your requirement as I understand it (single repo, all the
source code is there, including poky).

The huge downside of this is that some-dir is copied and this confuses
people.

I know the bitbake version in 2.4 has some differences that might help
here, but I haven't had the chance to investigate further.

Marcelo

Re: externalsrc + sstate why is not compatible?

[Koehler, Yannick]

Ok, will try that.  If that works, I may see if I can alter the file fetcher to use symlinks, not sure if sstate subsystem will like that or not.

If we do so, and someone change the file in /src/somedir will yocto redo the fetch/unpack pattern to recopy over the original content by itself?

--
Yannick Koehler

-----Message d'origine-----
De : Magallon, Marcelo 
Envoyé : 6 décembre 2017 09:47
À : Koehler, Yannick <yannick.koehler@hpe.com>
Cc : Alexander Kanavin <alexander.kanavin@linux.intel.com>; yocto@yoctoproject.org
Objet : Re: [yocto] externalsrc + sstate why is not compatible?

On Wed, Dec 06, 2017 at 02:13:26PM +0000, Koehler, Yannick wrote:

> In regards to file fetcher, I will go check the code, I thought the 
> unpack would only occurs for tarball, not subdir.

If you have:

	SRC_URI := "file://some-dir/"

and your tree looks somewhat like this:

	.git
	poky
	src/some-dir

and you set things up so that bitbake will look in src/ (set FILESPATH), then it will copy "some-dir" to $WORKDIR/some-dir, and you can point S there.

This satisfies your requirement as I understand it (single repo, all the source code is there, including poky).

The huge downside of this is that some-dir is copied and this confuses people.

I know the bitbake version in 2.4 has some differences that might help here, but I haven't had the chance to investigate further.

Marcelo

Re: externalsrc + sstate why is not compatible?

[Alexander Kanavin]

On 12/06/2017 04:13 PM, Koehler, Yannick wrote:
> When you say "subdirs", are you referring to submodules or something along those line?  We have used attempted to use submodules and it still required 20 + 1 repo and the overhead is similar to yocto + 20 repo but then you have yocto + 20 repos + 1 main repos, as such, this is not acceptable for us.  We also looked at other combination of multiple repos supported or in work for git and none are satisfying or supported in a way I could propose to use it in our project.
> 
> If you mean 1 repo, which subdirs representing each package, I would be interested, but how would it work with yocto SRC_URI?

The latter. It's simple: you set SRC_URI in each recipe to the same 
value, but set S differently:

S = "${WORKDIR}/git/module-1"

S = "${WORKDIR}/git/module-2"

etc.

Alex

Re: externalsrc + sstate why is not compatible?

[Marcelo E. Magallon]

On Wed, Dec 06, 2017 at 10:59:37AM -0600, Koehler, Yannick wrote:
> Ok, will try that.  If that works, I may see if I can alter the file
> fetcher to use symlinks, not sure if sstate subsystem will like that
> or not.

That was my idea, but I've never tried it. I'm sure the devil is in the
details. If you can elide the copy and simply replace it with a symlink,
you'd get faster builds.

> If we do so, and someone change the file in /src/somedir will yocto
> redo the fetch/unpack pattern to recopy over the original content by
> itself?

Yes, it does. You change a file, and the fetcher runs again. This means
S and B are deleted (if split), do_fetch runs again and S is populated
fresh. do_configure and do_compile populate B again.

Once you get that working, people will complain as to why the system is
rebuilding everything instead of just the file that changed :-) It might
work "as expected" if S = B, and your build system in the module is
smart enough to produce the correct dependencies (e.g. cmake or bazel).

Marcelo

Re: externalsrc + sstate why is not compatible?

[Koehler, Yannick]

Ok, will try this...

In our case, we are using CMake for most user space app, and when developing purely app related stuff we can use cmake directly thru command-line or eclipse without actually having a need for yocto.  Yocto in this context only serve to build the final image artefact.  As such, CMake ensure that only the correct files gets generated and use its own build folder as well.  We also get the nice benefit of Live Debug with eclipse using the yocto sdk.

Based on that, the file fetcher copy (if recursive) may be enough for us, even more so when combining with ccache, which we also uses.

--
Yannick Koehler

-----Message d'origine-----
De : Magallon, Marcelo 
Envoyé : 6 décembre 2017 13:46
À : Koehler, Yannick <yannick.koehler@hpe.com>
Cc : Alexander Kanavin <alexander.kanavin@linux.intel.com>; yocto@yoctoproject.org
Objet : Re: [yocto] externalsrc + sstate why is not compatible?

On Wed, Dec 06, 2017 at 10:59:37AM -0600, Koehler, Yannick wrote:
> Ok, will try that.  If that works, I may see if I can alter the file 
> fetcher to use symlinks, not sure if sstate subsystem will like that 
> or not.

That was my idea, but I've never tried it. I'm sure the devil is in the details. If you can elide the copy and simply replace it with a symlink, you'd get faster builds.

> If we do so, and someone change the file in /src/somedir will yocto 
> redo the fetch/unpack pattern to recopy over the original content by 
> itself?

Yes, it does. You change a file, and the fetcher runs again. This means S and B are deleted (if split), do_fetch runs again and S is populated fresh. do_configure and do_compile populate B again.

Once you get that working, people will complain as to why the system is rebuilding everything instead of just the file that changed :-) It might work "as expected" if S = B, and your build system in the module is smart enough to produce the correct dependencies (e.g. cmake or bazel).

Marcelo