devtool hardcodes using 'git' patchtool which causes errors

devtool hardcodes using 'git' patchtool which causes errors

[Adrian Ambrożewicz]

In poky sources i can see that default PATCHTOOL used is 'quilt' 
(https://github.com/openembedded/openembedded-core/blob/master/meta/conf/bitbake.conf) 
and it works flawlessly. Devtool however still defaults to legacy 'git' 
PATCHTOOL, as specified here:
http://git.yoctoproject.org/cgit.cgi/poky/plain/scripts/lib/devtool/standard.py
  > f.write('PATCHTOOL = "git"\n')

Request:
I've tried to debug the problem but I'm not very familiar with Bitbake 
code, so I wonder if we could default to 'quilt' for devtool as well?

Rationale:
'git' PATCHER doesn't work well when he have nested WORKDIR and 
striplevel used. 'bitbake -c patch [pkg]' works fine, but 'devtool 
modify [pkg]' fails with weird behavior. For example - new files created 
by patches are created in wrong location (WORKDIR instead of nested 
directory) and package doesn't compile until manually fixed.

Regards,
Adrian

Re: devtool hardcodes using 'git' patchtool which causes errors

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

I am not entirely sure about this, but I believe devtool is using git to
keep track of patches when modifying or upgrading (e.g. adding, modifying
or rebasing custom recipe patches), so using quilt may not be an option.
Fixing the scenarios that go wrong for you is probably a better path.

Alex

On Tue, 21 Jan 2020 at 10:44, Adrian Ambrożewicz <
adrian.ambrozewicz@linux.intel.com> wrote:

> In poky sources i can see that default PATCHTOOL used is 'quilt'
> (
> https://github.com/openembedded/openembedded-core/blob/master/meta/conf/bitbake.conf)
>
> and it works flawlessly. Devtool however still defaults to legacy 'git'
> PATCHTOOL, as specified here:
>
> http://git.yoctoproject.org/cgit.cgi/poky/plain/scripts/lib/devtool/standard.py
>   > f.write('PATCHTOOL = "git"\n')
>
> Request:
> I've tried to debug the problem but I'm not very familiar with Bitbake
> code, so I wonder if we could default to 'quilt' for devtool as well?
>
> Rationale:
> 'git' PATCHER doesn't work well when he have nested WORKDIR and
> striplevel used. 'bitbake -c patch [pkg]' works fine, but 'devtool
> modify [pkg]' fails with weird behavior. For example - new files created
> by patches are created in wrong location (WORKDIR instead of nested
> directory) and package doesn't compile until manually fixed.
>
> Regards,
> Adrian
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>

[-- Attachment #2: Type: text/html, Size: 2444 bytes --]

Re: devtool hardcodes using 'git' patchtool which causes errors

[Adrian Ambrożewicz]

I see. However - this looks like quite a minefield :)

W dniu 1/21/2020 o 11:49, Alexander Kanavin pisze:
> I am not entirely sure about this, but I believe devtool is using git to 
> keep track of patches when modifying or upgrading (e.g. adding, 
> modifying or rebasing custom recipe patches), so using quilt may not be 
> an option. Fixing the scenarios that go wrong for you is probably a 
> better path.
> 
> Alex
> 
> On Tue, 21 Jan 2020 at 10:44, Adrian Ambrożewicz 
> <adrian.ambrozewicz@linux.intel.com 
> <mailto:adrian.ambrozewicz@linux.intel.com>> wrote:
> 
>     In poky sources i can see that default PATCHTOOL used is 'quilt'
>     (https://github.com/openembedded/openembedded-core/blob/master/meta/conf/bitbake.conf)
> 
>     and it works flawlessly. Devtool however still defaults to legacy 'git'
>     PATCHTOOL, as specified here:
>     http://git.yoctoproject.org/cgit.cgi/poky/plain/scripts/lib/devtool/standard.py
>        > f.write('PATCHTOOL = "git"\n')
> 
>     Request:
>     I've tried to debug the problem but I'm not very familiar with Bitbake
>     code, so I wonder if we could default to 'quilt' for devtool as well?
> 
>     Rationale:
>     'git' PATCHER doesn't work well when he have nested WORKDIR and
>     striplevel used. 'bitbake -c patch [pkg]' works fine, but 'devtool
>     modify [pkg]' fails with weird behavior. For example - new files
>     created
>     by patches are created in wrong location (WORKDIR instead of nested
>     directory) and package doesn't compile until manually fixed.
> 
>     Regards,
>     Adrian
>     -- 
>     _______________________________________________
>     bitbake-devel mailing list
>     bitbake-devel@lists.openembedded.org
>     <mailto:bitbake-devel@lists.openembedded.org>
>     http://lists.openembedded.org/mailman/listinfo/bitbake-devel
> 

Re: devtool hardcodes using 'git' patchtool which causes errors

[Adrian Ambrożewicz]

I actually gave the problem some thought and I believe i must take back 
my first assumption that 'git' patcher is wrong. In fact - is either 
quilt or the way we achieve certain thing described here:

We have repo with project in nested directory. Lets imagine repo with 
following structure
\ // repo root
  \ project_dir
  \ other_project_dir

We have a recipe, which is set to build project from this repo. We do 
that by modyfing S variable to:
S = "${WORKDIR}/git/project_dir/"

Our local patches aims the repository structure, so they modify paths 
like "+++ a/project_dir/src/file.cpp". They don't apply cleanly during 
'bitbake -c patch [project]', as quilt tries to apply them from inside S 
directory ('${WORKDIR}/git/project_dir'). To mitigate that - author of 
recipe used 'striplevel=2' to allow proper building in such case.

Now that I analyzed that I see why 'git' patcher has problems with that. 
Git patcher uses 'git am', which naturally knows where the root of 
repository is (even if it was called from subdirectory). In case of 
'git' patcher striplevel should be defaulted to 1.

My question is - is there a way to have both of these patchers working 
with such folder structure? Maybe 'S' overwriting was not a good idea 
after all?

Regards,
Adrian


W dniu 1/22/2020 o 09:49, Adrian Ambrożewicz pisze:
> I see. However - this looks like quite a minefield :)
> 
> W dniu 1/21/2020 o 11:49, Alexander Kanavin pisze:
>> I am not entirely sure about this, but I believe devtool is using git 
>> to keep track of patches when modifying or upgrading (e.g. adding, 
>> modifying or rebasing custom recipe patches), so using quilt may not 
>> be an option. Fixing the scenarios that go wrong for you is probably a 
>> better path.
>>
>> Alex
>>
>> On Tue, 21 Jan 2020 at 10:44, Adrian Ambrożewicz 
>> <adrian.ambrozewicz@linux.intel.com 
>> <mailto:adrian.ambrozewicz@linux.intel.com>> wrote:
>>
>>     In poky sources i can see that default PATCHTOOL used is 'quilt'
>>     
>> (https://github.com/openembedded/openembedded-core/blob/master/meta/conf/bitbake.conf) 
>>
>>
>>     and it works flawlessly. Devtool however still defaults to legacy 
>> 'git'
>>     PATCHTOOL, as specified here:
>>     
>> http://git.yoctoproject.org/cgit.cgi/poky/plain/scripts/lib/devtool/standard.py 
>>
>>        > f.write('PATCHTOOL = "git"\n')
>>
>>     Request:
>>     I've tried to debug the problem but I'm not very familiar with 
>> Bitbake
>>     code, so I wonder if we could default to 'quilt' for devtool as well?
>>
>>     Rationale:
>>     'git' PATCHER doesn't work well when he have nested WORKDIR and
>>     striplevel used. 'bitbake -c patch [pkg]' works fine, but 'devtool
>>     modify [pkg]' fails with weird behavior. For example - new files
>>     created
>>     by patches are created in wrong location (WORKDIR instead of nested
>>     directory) and package doesn't compile until manually fixed.
>>
>>     Regards,
>>     Adrian
>>     --     _______________________________________________
>>     bitbake-devel mailing list
>>     bitbake-devel@lists.openembedded.org
>>     <mailto:bitbake-devel@lists.openembedded.org>
>>     http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>>

Re: devtool hardcodes using 'git' patchtool which causes errors

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]

On Wed, 22 Jan 2020 at 16:06, Adrian Ambrożewicz <
adrian.ambrozewicz@linux.intel.com> wrote:

>
> My question is - is there a way to have both of these patchers working
> with such folder structure? Maybe 'S' overwriting was not a good idea
> after all?
>

I guess the problem here is that S is used for both patching and building,
and the 'root' for these may not necessarily match. Maybe there could be a
variable that, when set, overrides S for the purpose of patching.

Alex

[-- Attachment #2: Type: text/html, Size: 832 bytes --]

Re: devtool hardcodes using 'git' patchtool which causes errors

[Adrian Ambrożewicz]

It looks like git:// fetcher option 'subpath' might be good alternative 
designed to address those kind of issues.

Setting:
SRC_URI = '[repopath];subpath=project_dir'
S = '${WORKDIR}/project_dir'
works exactly the same as my previous approach when it comes to 'bitbake 
-c patch [project]'

Unfortunately it looks like devtool ignores that param and clones whole 
repository structure. I'll try fiddling around to implement that support 
in devtool.


W dniu 1/22/2020 o 16:10, Alexander Kanavin pisze:
> On Wed, 22 Jan 2020 at 16:06, Adrian Ambrożewicz 
> <adrian.ambrozewicz@linux.intel.com 
> <mailto:adrian.ambrozewicz@linux.intel.com>> wrote:
> 
> 
>     My question is - is there a way to have both of these patchers working
>     with such folder structure? Maybe 'S' overwriting was not a good idea
>     after all?
> 
> 
> I guess the problem here is that S is used for both patching and 
> building, and the 'root' for these may not necessarily match. Maybe 
> there could be a variable that, when set, overrides S for the purpose of 
> patching.
> 
> Alex

Re: devtool hardcodes using 'git' patchtool which causes errors

[Adrian Ambrożewicz]

I've tried yet another approach, leveraging that this project is built 
using cmake and it worked! I know it's not a generic fix, but it looks 
like in bitbake nothing is really generic...

Instead of having project set like this:
SRC_URI = '[path_to_repo]'
S = '${WORKDIR}/git/project_dir'

I've set it like this:
SRC_URI = "[path_to_repo]"
S = "${WORKDIR}/git"
OECMAKE_SOURCEPATH = "${S}/virtual-media"

In that way both quilt and git patcher uses proper 'S' directory, while 
Cmake perform compilation from desired subdirectory.

I just hope I won't stumble into this kind of problem with another build 
system :)

Thanks a lot for your input, it lead me to correct track.

Regards,
Adrian

W dniu 1/22/2020 o 16:49, Adrian Ambrożewicz pisze:
> It looks like git:// fetcher option 'subpath' might be good alternative 
> designed to address those kind of issues.
> 
> Setting:
> SRC_URI = '[repopath];subpath=project_dir'
> S = '${WORKDIR}/project_dir'
> works exactly the same as my previous approach when it comes to 'bitbake 
> -c patch [project]'
> 
> Unfortunately it looks like devtool ignores that param and clones whole 
> repository structure. I'll try fiddling around to implement that support 
> in devtool.
> 
> 
> W dniu 1/22/2020 o 16:10, Alexander Kanavin pisze:
>> On Wed, 22 Jan 2020 at 16:06, Adrian Ambrożewicz 
>> <adrian.ambrozewicz@linux.intel.com 
>> <mailto:adrian.ambrozewicz@linux.intel.com>> wrote:
>>
>>
>>     My question is - is there a way to have both of these patchers 
>> working
>>     with such folder structure? Maybe 'S' overwriting was not a good idea
>>     after all?
>>
>>
>> I guess the problem here is that S is used for both patching and 
>> building, and the 'root' for these may not necessarily match. Maybe 
>> there could be a variable that, when set, overrides S for the purpose 
>> of patching.
>>
>> Alex

Re: devtool hardcodes using 'git' patchtool which causes errors

[Andre McCurdy]

On Thu, Jan 23, 2020 at 2:03 AM Adrian Ambrożewicz
<adrian.ambrozewicz@linux.intel.com> wrote:
>
> I've tried yet another approach, leveraging that this project is built
> using cmake and it worked! I know it's not a generic fix, but it looks
> like in bitbake nothing is really generic...

An alternative approach (perhaps the official generic solution?) may
be to use S="${WORKDIR}/git/project_dir", (re)create your patches from
the top level of the git repo without any extra strip levels etc and
then specify "patchdir" for each patch in SRC_URI, e.g. in your case
patchdir would be set to ".." in order to apply patches from the
directory one level above ${S}.

SRC_URI += "file://mychanges.patch;patchdir=.."

Re: devtool hardcodes using 'git' patchtool which causes errors

[Adrian Ambrożewicz]

I was not aware of patchdir option, so kudos for you. Will keep that in 
mind, thanks!

W dniu 1/25/2020 o 07:34, Andre McCurdy pisze:
> On Thu, Jan 23, 2020 at 2:03 AM Adrian Ambrożewicz
> <adrian.ambrozewicz@linux.intel.com> wrote:
>>
>> I've tried yet another approach, leveraging that this project is built
>> using cmake and it worked! I know it's not a generic fix, but it looks
>> like in bitbake nothing is really generic...
> 
> An alternative approach (perhaps the official generic solution?) may
> be to use S="${WORKDIR}/git/project_dir", (re)create your patches from
> the top level of the git repo without any extra strip levels etc and
> then specify "patchdir" for each patch in SRC_URI, e.g. in your case
> patchdir would be set to ".." in order to apply patches from the
> directory one level above ${S}.
> 
> SRC_URI += "file://mychanges.patch;patchdir=.."
>