* [PATCH 0/5] devtool: create kernel config fragment
@ 2015-12-18 8:39 Markus Lehtonen
2015-12-18 8:39 ` [PATCH 1/5] devtool: extract: use the correct datastore for builddir Markus Lehtonen
` (4 more replies)
0 siblings, 5 replies; 10+ messages in thread
From: Markus Lehtonen @ 2015-12-18 8:39 UTC (permalink / raw)
To: openembedded-core
This patchset implements an initial support for creating kernel config
fragments on 'devtool update-recipe'. The feature relies on having an updated
.config file and an initial base config (.config.orig) in the source tree.
Devtool does a diff operation between the two and creates/upates a config
fragment which is added to SRC_URI of the recipe.
The first patch in the series is a bugfix and should be straightforward to
merge.
[YOCTO #6658]
The following changes since commit 6f822a9fd185f479ef86c584b6d91a51b3a24e44:
meta: more removals of redunant FILES_${PN}-dbg (2015-12-16 12:11:26 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib marquiz/devtool/kernel
for you to fetch changes up to 2adc35e3129be74afe28dc3087ea44665a80221c:
devtool: update-recipe: create kernel config fragment (2015-12-17 14:34:15 +0200)
Markus Lehtonen (5):
devtool: extract: use the correct datastore for builddir
kernel.bbclass: copy .config instead of moving
devtool: extract: cleanup srctree
cml1.bbclass: copy .config to S if externalsr is in use
devtool: update-recipe: create kernel config fragment
meta/classes/cml1.bbclass | 5 ++++
meta/classes/kernel.bbclass | 2 +-
scripts/lib/devtool/standard.py | 65 +++++++++++++++++++++++++++++++++++++++--
3 files changed, 68 insertions(+), 4 deletions(-)
--
2.1.4
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 1/5] devtool: extract: use the correct datastore for builddir 2015-12-18 8:39 [PATCH 0/5] devtool: create kernel config fragment Markus Lehtonen @ 2015-12-18 8:39 ` Markus Lehtonen 2015-12-18 8:39 ` [PATCH 2/5] kernel.bbclass: copy .config instead of moving Markus Lehtonen ` (3 subsequent siblings) 4 siblings, 0 replies; 10+ messages in thread From: Markus Lehtonen @ 2015-12-18 8:39 UTC (permalink / raw) To: openembedded-core Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> --- scripts/lib/devtool/standard.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index a5e81f3..43fce11 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -461,7 +461,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d): # Store generate and store kernel config logger.info('Generating kernel config') task_executor.exec_func('do_configure', False) - kconfig = os.path.join(d.getVar('B', True), '.config') + kconfig = os.path.join(crd.getVar('B', True), '.config') shutil.copy2(kconfig, srcsubdir) -- 2.1.4 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 2/5] kernel.bbclass: copy .config instead of moving 2015-12-18 8:39 [PATCH 0/5] devtool: create kernel config fragment Markus Lehtonen 2015-12-18 8:39 ` [PATCH 1/5] devtool: extract: use the correct datastore for builddir Markus Lehtonen @ 2015-12-18 8:39 ` Markus Lehtonen 2015-12-18 12:22 ` Richard Purdie 2015-12-18 8:39 ` [PATCH 3/5] devtool: extract: cleanup srctree Markus Lehtonen ` (2 subsequent siblings) 4 siblings, 1 reply; 10+ messages in thread From: Markus Lehtonen @ 2015-12-18 8:39 UTC (permalink / raw) To: openembedded-core Copy kernel .config from ${S} to ${B}, instead of moving it. This prevents mangling the source tree, which is undesirable e.g. when externalsrc is used. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> --- meta/classes/kernel.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass index 4ce1611..7de9f20 100644 --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -338,7 +338,7 @@ kernel_do_configure() { touch ${B}/.scmversion ${S}/.scmversion if [ "${S}" != "${B}" ] && [ -f "${S}/.config" ] && [ ! -f "${B}/.config" ]; then - mv "${S}/.config" "${B}/.config" + cp "${S}/.config" "${B}/.config" fi # Copy defconfig to .config if .config does not exist. This allows -- 2.1.4 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 2/5] kernel.bbclass: copy .config instead of moving 2015-12-18 8:39 ` [PATCH 2/5] kernel.bbclass: copy .config instead of moving Markus Lehtonen @ 2015-12-18 12:22 ` Richard Purdie 2015-12-18 12:39 ` Markus Lehtonen 0 siblings, 1 reply; 10+ messages in thread From: Richard Purdie @ 2015-12-18 12:22 UTC (permalink / raw) To: Markus Lehtonen, openembedded-core On Fri, 2015-12-18 at 10:39 +0200, Markus Lehtonen wrote: > Copy kernel .config from ${S} to ${B}, instead of moving it. This > prevents mangling the source tree, which is undesirable e.g. when > externalsrc is used. > > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> > --- > meta/classes/kernel.bbclass | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/classes/kernel.bbclass > b/meta/classes/kernel.bbclass > index 4ce1611..7de9f20 100644 > --- a/meta/classes/kernel.bbclass > +++ b/meta/classes/kernel.bbclass > @@ -338,7 +338,7 @@ kernel_do_configure() { > touch ${B}/.scmversion ${S}/.scmversion > > if [ "${S}" != "${B}" ] && [ -f "${S}/.config" ] && [ ! -f > "${B}/.config" ]; then > - mv "${S}/.config" "${B}/.config" > + cp "${S}/.config" "${B}/.config" > fi > > # Copy defconfig to .config if .config does not exist. This > allows I'm not sure about this, doesn't this trigger the kernel to see ${S} as being 'dirty' and cause other issues when you try and do out of tree builds with it? It also means we have two copies of "config" around which can end up being different and confuse users no end :(. Cheers, Richard ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/5] kernel.bbclass: copy .config instead of moving 2015-12-18 12:22 ` Richard Purdie @ 2015-12-18 12:39 ` Markus Lehtonen 2015-12-18 14:18 ` Richard Purdie 0 siblings, 1 reply; 10+ messages in thread From: Markus Lehtonen @ 2015-12-18 12:39 UTC (permalink / raw) To: Richard Purdie, openembedded-core Hi Richard, On 18/12/15 14:22, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote: >On Fri, 2015-12-18 at 10:39 +0200, Markus Lehtonen wrote: >> Copy kernel .config from ${S} to ${B}, instead of moving it. This >> prevents mangling the source tree, which is undesirable e.g. when >> externalsrc is used. >> >> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> >> --- >> meta/classes/kernel.bbclass | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/meta/classes/kernel.bbclass >> b/meta/classes/kernel.bbclass >> index 4ce1611..7de9f20 100644 >> --- a/meta/classes/kernel.bbclass >> +++ b/meta/classes/kernel.bbclass >> @@ -338,7 +338,7 @@ kernel_do_configure() { >> touch ${B}/.scmversion ${S}/.scmversion >> >> if [ "${S}" != "${B}" ] && [ -f "${S}/.config" ] && [ ! -f >> "${B}/.config" ]; then >> - mv "${S}/.config" "${B}/.config" >> + cp "${S}/.config" "${B}/.config" >> fi >> >> # Copy defconfig to .config if .config does not exist. This >> allows > >I'm not sure about this, doesn't this trigger the kernel to see ${S} as >being 'dirty' and cause other issues when you try and do out of tree >builds with it? That shouldn't be a problem as the kernel .gitignore ignores .config (or '.*' to be more specific). There are other tasks that make changes to the kernel source tree, as well, like do_kernel_metadata. >It also means we have two copies of "config" around which can end up >being different and confuse users no end :(. Yes, I must agree. What do you think if ${B}/.config would be a symlink to ${S}/.config? I.e. + ln -s "${S}/.config" "${B}/.config" Thanks, Markus ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/5] kernel.bbclass: copy .config instead of moving 2015-12-18 12:39 ` Markus Lehtonen @ 2015-12-18 14:18 ` Richard Purdie 2015-12-18 15:48 ` Markus Lehtonen 0 siblings, 1 reply; 10+ messages in thread From: Richard Purdie @ 2015-12-18 14:18 UTC (permalink / raw) To: Markus Lehtonen, openembedded-core On Fri, 2015-12-18 at 14:39 +0200, Markus Lehtonen wrote: > On 18/12/15 14:22, "Richard Purdie" < > richard.purdie@linuxfoundation.org> wrote: > > On Fri, 2015-12-18 at 10:39 +0200, Markus Lehtonen wrote: > > > if [ "${S}" != "${B}" ] && [ -f "${S}/.config" ] && [ ! > > > -f > > > "${B}/.config" ]; then > > > - mv "${S}/.config" "${B}/.config" > > > + cp "${S}/.config" "${B}/.config" > > > fi > > > > > > # Copy defconfig to .config if .config does not exist. > > > This > > > allows > > > > I'm not sure about this, doesn't this trigger the kernel to see > > ${S} as > > being 'dirty' and cause other issues when you try and do out of > > tree > > builds with it? > > That shouldn't be a problem as the kernel .gitignore ignores .config > (or '.*' to be more specific). There are other tasks that make > changes to the kernel source tree, as well, like do_kernel_metadata. > > > > It also means we have two copies of "config" around which can end > > up > > being different and confuse users no end :(. > > Yes, I must agree. What do you think if ${B}/.config would be a > symlink to ${S}/.config? I.e. > + ln -s "${S}/.config" "${B}/.config" > I think I'd prefer we move the file over the ${B} and then symlink from ${S} since that way if its modified, its mostly likely to be done from ${B} at least by the automated code? Cheers, Richard ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/5] kernel.bbclass: copy .config instead of moving 2015-12-18 14:18 ` Richard Purdie @ 2015-12-18 15:48 ` Markus Lehtonen 0 siblings, 0 replies; 10+ messages in thread From: Markus Lehtonen @ 2015-12-18 15:48 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 18/12/15 16:18, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote: >On Fri, 2015-12-18 at 14:39 +0200, Markus Lehtonen wrote: >> On 18/12/15 14:22, "Richard Purdie" < >> richard.purdie@linuxfoundation.org> wrote: >> > On Fri, 2015-12-18 at 10:39 +0200, Markus Lehtonen wrote: >> > > if [ "${S}" != "${B}" ] && [ -f "${S}/.config" ] && [ ! >> > > -f >> > > "${B}/.config" ]; then >> > > - mv "${S}/.config" "${B}/.config" >> > > + cp "${S}/.config" "${B}/.config" >> > > fi >> > > >> > > # Copy defconfig to .config if .config does not exist. >> > > This >> > > allows >> > >> > I'm not sure about this, doesn't this trigger the kernel to see >> > ${S} as >> > being 'dirty' and cause other issues when you try and do out of >> > tree >> > builds with it? >> >> That shouldn't be a problem as the kernel .gitignore ignores .config >> (or '.*' to be more specific). There are other tasks that make >> changes to the kernel source tree, as well, like do_kernel_metadata. >> >> >> > It also means we have two copies of "config" around which can end >> > up >> > being different and confuse users no end :(. >> >> Yes, I must agree. What do you think if ${B}/.config would be a >> symlink to ${S}/.config? I.e. >> + ln -s "${S}/.config" "${B}/.config" >> > >I think I'd prefer we move the file over the ${B} and then symlink from >${S} since that way if its modified, its mostly likely to be done from >${B} at least by the automated code? Hmm, what I suggested is not good (at least on its own) because menuconfig moves .config to .config.old. Symlinking from ${S} to ${B} is also problematic for the intended devtool use case because removing/cleaning ${B} also removes our .config. Back to the drawing board, I guess... Thanks, Markus ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 3/5] devtool: extract: cleanup srctree 2015-12-18 8:39 [PATCH 0/5] devtool: create kernel config fragment Markus Lehtonen 2015-12-18 8:39 ` [PATCH 1/5] devtool: extract: use the correct datastore for builddir Markus Lehtonen 2015-12-18 8:39 ` [PATCH 2/5] kernel.bbclass: copy .config instead of moving Markus Lehtonen @ 2015-12-18 8:39 ` Markus Lehtonen 2015-12-18 8:39 ` [PATCH 4/5] cml1.bbclass: copy .config to S if externalsr is in use Markus Lehtonen 2015-12-18 8:39 ` [PATCH 5/5] devtool: update-recipe: create kernel config fragment Markus Lehtonen 4 siblings, 0 replies; 10+ messages in thread From: Markus Lehtonen @ 2015-12-18 8:39 UTC (permalink / raw) To: openembedded-core Some bitbake tasks, notably do_kernel_metadata et al. dirty the sourcetree. Run git clean in order to get rid of the confusing and possibly outdated extra files. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> --- scripts/lib/devtool/standard.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index 43fce11..f817671 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -458,10 +458,21 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d): bb.process.run('git checkout patches', cwd=srcsubdir) if bb.data.inherits_class('kernel-yocto', d): - # Store generate and store kernel config logger.info('Generating kernel config') task_executor.exec_func('do_configure', False) kconfig = os.path.join(crd.getVar('B', True), '.config') + else: + kconfig = None + + # Clean source tree in case it has been dirtied by some bitbake task + stdout, _ = bb.process.run('git status --porcelain --ignored', + cwd=srcsubdir) + if stdout: + logger.info('Source tree is dirty, cleaning up') + bb.process.run('git clean -fdx', cwd=srcsubdir) + + if kconfig: + # Store kernel config in srctree shutil.copy2(kconfig, srcsubdir) -- 2.1.4 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 4/5] cml1.bbclass: copy .config to S if externalsr is in use 2015-12-18 8:39 [PATCH 0/5] devtool: create kernel config fragment Markus Lehtonen ` (2 preceding siblings ...) 2015-12-18 8:39 ` [PATCH 3/5] devtool: extract: cleanup srctree Markus Lehtonen @ 2015-12-18 8:39 ` Markus Lehtonen 2015-12-18 8:39 ` [PATCH 5/5] devtool: update-recipe: create kernel config fragment Markus Lehtonen 4 siblings, 0 replies; 10+ messages in thread From: Markus Lehtonen @ 2015-12-18 8:39 UTC (permalink / raw) To: openembedded-core This makes it easier to track changes made to config. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> --- meta/classes/cml1.bbclass | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/meta/classes/cml1.bbclass b/meta/classes/cml1.bbclass index 95cf584..a95a2bf 100644 --- a/meta/classes/cml1.bbclass +++ b/meta/classes/cml1.bbclass @@ -28,6 +28,11 @@ python do_menuconfig() { oe_terminal("${SHELL} -c \"make ${KCONFIG_CONFIG_COMMAND}; if [ \$? -ne 0 ]; then echo 'Command failed.'; printf 'Press any key to continue... '; read r; fi\"", '${PN} Configuration', d) + # Copy .config back to source tree if externalsrc is in use + if (d.getVar('EXTERNALSRC', True) and + os.path.exists(os.path.join(d.getVar('S', True), '.config'))): + shutil.copy2('.config', os.path.join(d.getVar('S', True), '.config')) + # FIXME this check can be removed when the minimum bitbake version has been bumped if hasattr(bb.build, 'write_taint'): try: -- 2.1.4 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 5/5] devtool: update-recipe: create kernel config fragment 2015-12-18 8:39 [PATCH 0/5] devtool: create kernel config fragment Markus Lehtonen ` (3 preceding siblings ...) 2015-12-18 8:39 ` [PATCH 4/5] cml1.bbclass: copy .config to S if externalsr is in use Markus Lehtonen @ 2015-12-18 8:39 ` Markus Lehtonen 4 siblings, 0 replies; 10+ messages in thread From: Markus Lehtonen @ 2015-12-18 8:39 UTC (permalink / raw) To: openembedded-core Create kernel config fragment if the user makes modifications to .config. User may change .config e.g. by directly editing it or by running the 'do_menuconfig' bitbake task which will copy the modified .config back to the source tree. Devtool generates one monolithic fragment by simply doing a diff between .config and .config.orig files in the source directory. If either of these files is missing, the config fragment is not gerenrated or updated. The output is a file, 'devtool-fragment.cfg' that gets added to SRC_URI in the recipe (as well as copied into the 'oe-local-files' directory if that is present in the source tree). This patch also changes the devtool 'extract' command to create the .config.orig file at the source tree creation time. [YOCTO #6658] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> --- scripts/lib/devtool/standard.py | 50 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index f817671..aa9414b 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -20,6 +20,7 @@ import os import sys import re import shutil +import subprocess import tempfile import logging import argparse @@ -474,6 +475,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d): if kconfig: # Store kernel config in srctree shutil.copy2(kconfig, srcsubdir) + shutil.copy2(kconfig, os.path.join(srcsubdir, '.config.orig')) tempdir_localdir = os.path.join(tempdir, 'oe-local-files') @@ -804,6 +806,30 @@ def _export_patches(srctree, rd, start_rev, destdir): return (updated, added, existing_patches) +def _create_kconfig_diff(srctree, rd, outfile): + """Create a kernel config fragment""" + # Only update config fragment if both config files exist + orig_config = os.path.join(srctree, '.config.orig') + new_config = os.path.join(srctree, '.config') + if os.path.exists(orig_config) and os.path.exists(new_config): + cmd = ['diff', '--new-line-format=%L', '--old-line-format=', + '--unchanged-line-format=', orig_config, new_config] + pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + stdout, stderr = pipe.communicate() + if pipe.returncode == 1: + with open(outfile, 'w') as fobj: + fobj.write(stdout) + elif pipe.returncode == 0: + if os.path.exists(outfile): + # Remove fragment file in case of empty diff + os.unlink(outfile) + else: + raise bb.process.ExecutionError(cmd, pipe.returncode, stdout, stderr) + return True + return False + + def _export_local_files(srctree, rd, destdir): """Copy local files from srctree to given location. Returns three-tuple of dicts: @@ -824,6 +850,7 @@ def _export_local_files(srctree, rd, destdir): updated = OrderedDict() added = OrderedDict() removed = OrderedDict() + local_files_dir = os.path.join(srctree, 'oe-local-files') git_files = _git_ls_tree(srctree) if 'oe-local-files' in git_files: # If tracked by Git, take the files from srctree HEAD. First get @@ -834,11 +861,32 @@ def _export_local_files(srctree, rd, destdir): env=dict(os.environ, GIT_WORK_TREE=destdir, GIT_INDEX_FILE=tmp_index)) new_set = _git_ls_tree(srctree, tree, True).keys() - elif os.path.isdir(os.path.join(srctree, 'oe-local-files')): + elif os.path.isdir(local_files_dir): # If not tracked by Git, just copy from working copy new_set = _ls_tree(os.path.join(srctree, 'oe-local-files')) bb.process.run(['cp', '-ax', os.path.join(srctree, 'oe-local-files', '.'), destdir]) + else: + new_set = [] + + # Special handling for kernel config + if bb.data.inherits_class('kernel-yocto', rd): + fragment_fn = 'devtool-fragment.cfg' + fragment_path = os.path.join(destdir, fragment_fn) + if _create_kconfig_diff(srctree, rd, fragment_path): + if os.path.exists(fragment_path): + if fragment_fn not in new_set: + new_set.append(fragment_fn) + # Copy fragment to local-files + if os.path.isdir(local_files_dir): + shutil.copy2(fragment_path, local_files_dir) + else: + if fragment_fn in new_set: + new_set.remove(fragment_fn) + # Remove fragment from local-files + if os.path.exists(os.path.join(local_files_dir, fragment_fn)): + os.unlink(os.path.join(local_files_dir, fragment_fn)) + if new_set is not None: for fname in new_set: if fname in existing_files: -- 2.1.4 ^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-12-18 15:48 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-12-18 8:39 [PATCH 0/5] devtool: create kernel config fragment Markus Lehtonen 2015-12-18 8:39 ` [PATCH 1/5] devtool: extract: use the correct datastore for builddir Markus Lehtonen 2015-12-18 8:39 ` [PATCH 2/5] kernel.bbclass: copy .config instead of moving Markus Lehtonen 2015-12-18 12:22 ` Richard Purdie 2015-12-18 12:39 ` Markus Lehtonen 2015-12-18 14:18 ` Richard Purdie 2015-12-18 15:48 ` Markus Lehtonen 2015-12-18 8:39 ` [PATCH 3/5] devtool: extract: cleanup srctree Markus Lehtonen 2015-12-18 8:39 ` [PATCH 4/5] cml1.bbclass: copy .config to S if externalsr is in use Markus Lehtonen 2015-12-18 8:39 ` [PATCH 5/5] devtool: update-recipe: create kernel config fragment Markus Lehtonen
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.