[PATCH v2 0/3] Devtool: provide easy means of reconfiguring

[PATCH v2 0/3] Devtool: provide easy means of reconfiguring

[Sai Hari Chandana Kalluri]

This patch series provides support for the user to run menuconfig command in the 
devtool flow. This would allow the user to modify the current configurations and 
generate a config fragment to update the recipe using devtool finish. Devtool
menuconfig command will work on all packages that contain menuconfig as a task.

1. The implementation checks if devtool menuconfig command is called for a valid
package.
2. It checks for oe-local-files dir within source and creates one if
needed, this directory is needed to store the final generated config fragment so
that devtool finish can update the recipe.
3. Menuconfig command is called for users to make necessary changes. After
saving the changes, diffconfig command is run to generate the fragment.

Currently, when the user runs devtool modify command, it checks out the entire    
source tree which is a bit of an over head in time and space. This patch series    
also provides a way to create a copy(hard links) of the kernel source, if    
present, from work-shared to workspace to be more efficient .    
    
Also, if the kernel source is not present in the staging kernel dir and the user    
fetches the source tree in workspace using devtool modify, then this patch    
series creates a copy of source from workspace to work-shared. This is    
necessary for packages that may use the kernel source.    
    
[YOCTO #10416]    
    
---                                                                                                                   
changelog v2    
1. Add a check to verify the set kernel version by user and kernel version                                            
present in work-shared before creating hard links to workspace.    
2. Add a check to verify the set kernel version by user and kernel version                                            
present in workspace do not match, when placing a copy of kernel-source    
from workspace to work-shared. 
Sai Hari Chandana Kalluri (3):
  devtool modify: Update devtool modify to copy source from work-shared
    if     its already downloaded.
  devtool modify: Create a copy of kernel source within work-shared if
    not     present
  devtool: provide support for devtool menuconfig command.

Sai Hari Chandana Kalluri (3):
  devtool modify: Update devtool modify to copy source from work-shared
    if     its already downloaded.
  devtool modify: Create a copy of kernel source within work-shared if
    not     present
  devtool: provide support for devtool menuconfig command.

 scripts/lib/devtool/menuconfig.py |  80 +++++++++++++++++++
 scripts/lib/devtool/standard.py   | 162 ++++++++++++++++++++++++++++++++++++--
 2 files changed, 237 insertions(+), 5 deletions(-)
 create mode 100644 scripts/lib/devtool/menuconfig.py

-- 
2.7.4

[PATCH v2 1/3] devtool modify: Update devtool modify to copy source from work-shared if its already downloaded.

[Sai Hari Chandana Kalluri]

In the regular devtool modify flow, the kernel source is fetched by running
do_fetch task. This is an overhead in time and space.

This patch updates modify command to check if the kernel source is already
downloaded. If so, then instead of calling do_fetch, copy the source
from work-shared to devtool workspace by creating hard links. Else run
the usual devtool modify flow and call do_fetch task.

[YOCTO #10416]

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
changelog v2: 
1. Add a check to verify the set kernel version by user and kernel version
present in work-shared before creating hard links. This is to ensure that right
kernel version is present in workspace.
---
 scripts/lib/devtool/standard.py | 148 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 144 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index b7d4d47..ba1afa4 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -717,6 +717,72 @@ def _check_preserve(config, recipename):
                     tf.write(line)
     os.rename(newfile, origfile)
 
+# Function links a file from src location to dest location
+def copy_file(c,dest):
+    import errno
+    destdir = os.path.dirname(dest)
+    if os.path.islink(c):
+       linkto = os.readlink(c)
+       if os.path.lexists(dest):
+            if not os.path.islink(dest):
+                raise OSError(errno.EEXIST, "Link %s already exists as a file" % dest, dest)
+            if os.readlink(dest) == linkto:
+                return dest
+            raise OSError(errno.EEXIST, "Link %s already exists to a different location? (%s vs %s)" % (dest, os.readlink(dest), linkto), dest)
+       os.symlink(linkto, dest)
+    else:
+       try:
+           os.link(c, dest)
+       except OSError as err:
+           if err.errno == errno.EXDEV:
+                bb.utils.copyfile(c, dest)
+           else:
+                raise
+
+# Function creates folders in a given target location
+def copy_dirs(root,dirs,target):
+    for d in dirs:
+        destdir =  os.path.join(target,d)
+        if os.path.islink(os.path.join(root,d)):
+             linkto = os.readlink(os.path.join(root,d))
+             os.symlink(linkto,destdir) 
+        else:
+             bb.utils.mkdirhier(target+d)
+
+# Function to link src dir to dest dir
+def copy_src_to_ws(srcdir,srctree):
+    target = srctree
+    if os.path.exists(target):
+        raise DevtoolError('source already in your workspace')
+
+    bb.utils.mkdirhier(target)
+    for root,dirs,files in os.walk(srcdir):
+        #convert abspath to relpath for root
+        destdir = root.replace(srcdir,"")
+        target = srctree+destdir+"/"
+        copy_dirs(root,dirs,target)  
+        for f in files:
+           copy_file(os.path.join(root,f),os.path.join(target,f))
+
+def get_staging_kver(srcdir):
+     #Kernel version from work-shared
+     kerver = []
+     staging_kerVer=""
+     if os.path.exists(srcdir) and os.listdir(srcdir):
+            with open(os.path.join(srcdir,"Makefile")) as f:
+                 version = [next(f) for x in range(5)][1:4]
+                 for word in version:
+                       kerver.append(word.split('= ')[1].split('\n')[0])
+                 staging_kerVer = ".".join(kerver)
+     return staging_kerVer
+
+def get_staging_kbranch(srcdir):
+      staging_kbranch = ""
+      if os.path.exists(srcdir) and os.listdir(srcdir):
+          (branch, _) = bb.process.run('git branch | grep \* | cut -d \' \' -f2', cwd=srcdir)
+          staging_kbranch = "".join(branch.split('\n')[0])
+      return staging_kbranch
+
 def modify(args, config, basepath, workspace):
     """Entry point for the devtool 'modify' subcommand"""
     import bb
@@ -763,6 +829,78 @@ def modify(args, config, basepath, workspace):
         initial_rev = None
         commits = []
         check_commits = False
+
+        if bb.data.inherits_class('kernel-yocto', rd):
+               #Current set kernel version
+               kernelVersion = rd.getVar('LINUX_VERSION')
+               srcdir = rd.getVar('STAGING_KERNEL_DIR')
+               kbranch = rd.getVar('KBRANCH')
+
+               staging_kerVer = get_staging_kver(srcdir)
+               staging_kbranch = get_staging_kbranch(srcdir)
+               if (os.path.exists(srcdir) and os.listdir(srcdir)) and (kernelVersion in staging_kerVer and staging_kbranch == kbranch):
+                  copy_src_to_ws(srcdir,srctree)
+                  workdir = rd.getVar('WORKDIR')
+                  srcsubdir = rd.getVar('S')
+                  localfilesdir = os.path.join(srctree,'oe-local-files') 
+                  # Move local source files into separate subdir
+                  recipe_patches = [os.path.basename(patch) for patch in oe.recipeutils.get_recipe_patches(rd)]
+                  local_files = oe.recipeutils.get_recipe_local_files(rd)
+
+                  for key in local_files.copy():
+                        if key.endswith('scc'):
+                              sccfile = open(local_files[key], 'r')
+                              for l in sccfile:
+                                  line = l.split()
+                                  if line and line[0] in ('kconf', 'patch'):
+                                        local_files[line[-1]] = os.path.join(os.path.dirname(local_files[key]), line[-1])
+                                        shutil.copy2(os.path.join(os.path.dirname(local_files[key]), line[-1]), workdir)
+                              sccfile.close()
+
+                  # Ignore local files with subdir={BP}
+                  srcabspath = os.path.abspath(srcsubdir)
+                  local_files = [fname for fname in local_files if os.path.exists(os.path.join(workdir, fname)) and  (srcabspath == workdir or not  os.path.join(workdir, fname).startswith(srcabspath + os.sep))]
+                  if local_files:
+                       for fname in local_files:
+                              copy_src_to_ws(os.path.join(workdir, fname), os.path.join(srctree, 'oe-local-files', fname))
+                       with open(os.path.join(srctree, 'oe-local-files', '.gitignore'), 'w') as f:
+                              f.write('# Ignore local files, by default. Remove this file ''if you want to commit the directory to Git\n*\n')
+
+                  if os.path.abspath(rd.getVar('S')) == os.path.abspath(rd.getVar('WORKDIR')):
+                  # If recipe extracts to ${WORKDIR}, symlink the files into the srctree
+                  # (otherwise the recipe won't build as expected)
+                        local_files_dir = os.path.join(srctree, 'oe-local-files')
+                        addfiles = []
+                        for root, _, files in os.walk(local_files_dir):
+                            relpth = os.path.relpath(root, local_files_dir)
+                            if relpth != '.':
+                                  bb.utils.mkdirhier(os.path.join(srctree, relpth))
+                            for fn in files:
+                                if fn == '.gitignore':
+                                    continue
+                                destpth = os.path.join(srctree, relpth, fn)
+                                if os.path.exists(destpth):
+                                    os.unlink(destpth)
+                                os.symlink('oe-local-files/%s' % fn, destpth)
+                                addfiles.append(os.path.join(relpth, fn))
+                        if addfiles:
+                           bb.process.run('git add %s' % ' '.join(addfiles), cwd=srctree)
+                        useroptions = []
+                        oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d)
+                        bb.process.run('git %s commit -a -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree)
+
+                  task = 'do_configure'
+                  res = tinfoil.build_targets(pn, task, handle_events=True)
+
+                  # Copy .config to workspace 
+                  kconfpath=rd.getVar('B')
+                  logger.info('Copying kernel config to workspace')
+                  shutil.copy2(os.path.join(kconfpath, '.config'),srctree)
+
+                  # Set this to true, we still need to get initial_rev
+                  # by parsing the git repo
+                  args.no_extract = True
+
         if not args.no_extract:
             initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
             if not initial_rev:
@@ -852,10 +990,12 @@ def modify(args, config, basepath, workspace):
                 f.write('\ndo_patch() {\n'
                         '    :\n'
                         '}\n')
-                f.write('\ndo_configure_append() {\n'
-                        '    cp ${B}/.config ${S}/.config.baseline\n'
-                        '    ln -sfT ${B}/.config ${S}/.config.new\n'
-                        '}\n')
+
+            if rd.getVarFlag('do_menuconfig','task'):
+                    f.write('\ndo_configure_append() {\n'
+                    '    cp ${B}/.config ${S}/.config.baseline\n'
+                    '    ln -sfT ${B}/.config ${S}/.config.new\n'
+                    '}\n')
             if initial_rev:
                 f.write('\n# initial_rev: %s\n' % initial_rev)
                 for commit in commits:
-- 
2.7.4

[PATCH v2 2/3] devtool modify: Create a copy of kernel source within work-shared if not present

[Sai Hari Chandana Kalluri]

If kernel source is not already downloaded i.e staging kernel dir is
empty, place a copy of the source when the user runs devtool modify linux-yocto.
This way the kernel source is available for other packages that use it.

[YOCTO #10416]

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
changelog v2: 

1. Add a check to verify the set kernel version by user and kernel version 
present in workspace do not match before placing a copy of kernel-source 
from workspace to work-shared.

---

 scripts/lib/devtool/standard.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ba1afa4..24a0168 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -572,7 +572,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
         with open(preservestampfile, 'w') as f:
             f.write(d.getVar('STAMP'))
         try:
-            if bb.data.inherits_class('kernel-yocto', d):
+            if is_kernel_yocto:
                 # We need to generate the kernel config
                 task = 'do_configure'
             else:
@@ -599,6 +599,18 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             raise DevtoolError('Something went wrong with source extraction - the devtool-source class was not active or did not function correctly:\n%s' % str(e))
         srcsubdir_rel = os.path.relpath(srcsubdir, os.path.join(tempdir, 'workdir'))
 
+        # Check if work-shared is empty, if yes 
+        # find source and copy to work-shared
+        if is_kernel_yocto:
+              workshareddir = d.getVar('STAGING_KERNEL_DIR')
+              staging_kerVer = get_staging_kver(workshareddir) 
+              kernelVersion = d.getVar('LINUX_VERSION')
+              if (os.path.exists(workshareddir) and (not os.listdir(workshareddir) or kernelVersion!=staging_kerVer)): 
+                   shutil.rmtree(workshareddir)
+                   copy_src_to_ws(srcsubdir,workshareddir)
+              elif (not os.path.exists(workshareddir)):
+                   copy_src_to_ws(srcsubdir,workshareddir)
+
         tempdir_localdir = os.path.join(tempdir, 'oe-local-files')
         srctree_localdir = os.path.join(srctree, 'oe-local-files')
 
-- 
2.7.4

[PATCH v2 3/3] devtool: provide support for devtool menuconfig command.

[Sai Hari Chandana Kalluri]

All packages that support the menuconfig task will be able to run devtool
menuconfig command. This would allow the user to modify the current
configure options and create a config fragment which can be added to a
recipe using devtool finish.

1. The patch checks if devtool menuconfig command is called for a valid
package.
2. It checks for oe-local-files dir within source and creates one
if needed, this directory is needed to store the final generated config
fragment so that devtool finish can update the recipe.
3. Menuconfig command is called for users to make necessary changes. After
saving the changes, diffconfig command is run to generate the fragment.

Syntax:
	devtool menuconfig <package name>
	 Ex: devtool menuconfig linux-yocto

The config fragment is saved as devtool-fragment.cfg within
oe-local-files dir.

	Ex: <workspace_path>/sources/linux-yocto/oe-local-files/devtool-fragment.cfg

Run devtool finish to update the recipe by appending the config fragment
to SRC_URI and place a copy of the fragment within the layer where the
recipe resides.
	Ex: devtool finish linux-yocto meta

[YOCTO #10416]

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 scripts/lib/devtool/menuconfig.py | 80 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)
 create mode 100644 scripts/lib/devtool/menuconfig.py

diff --git a/scripts/lib/devtool/menuconfig.py b/scripts/lib/devtool/menuconfig.py
new file mode 100644
index 0000000..38133db
--- /dev/null
+++ b/scripts/lib/devtool/menuconfig.py
@@ -0,0 +1,80 @@
+# OpenEmbedded Development tool - menuconfig command plugin
+#
+# Copyright (C) 2018 Xilinx
+# Written by: Chandana Kalluri <ckalluri@xilinx.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""Devtool menuconfig plugin"""
+
+import os
+import bb
+import logging
+import argparse
+import re
+import glob
+from devtool import setup_tinfoil, parse_recipe, DevtoolError, standard, exec_build_env_command
+
+logger = logging.getLogger('devtool')
+
+def menuconfig(args, config, basepath, workspace):
+    """Entry point for the devtool 'menuconfig' subcommand"""
+
+    rd = "" 
+    kconfigpath = ""
+    pn_src = ""
+    localfilesdir = ""
+    workspace_dir = ""
+    tinfoil = setup_tinfoil(basepath=basepath)
+    try:
+      rd = parse_recipe(config, tinfoil, args.component, appends=True, filter_workspace=False)
+      if not rd:
+         return 1
+
+      pn =  rd.getVar('PN', True)
+      if pn not in workspace:
+         raise DevtoolError("Run devtool modify before calling menuconfig for %s" %pn)
+
+      if not rd.getVarFlag('do_menuconfig','task'):
+         raise DevtoolError("This package does not support menuconfig option")
+
+      workspace_dir = os.path.join(basepath,'workspace/sources')
+      kconfigpath = rd.getVar('B')
+      pn_src = os.path.join(workspace_dir,pn)
+
+      #add check to see if oe_local_files exists or not
+      localfilesdir = os.path.join(pn_src,'oe-local-files') 
+      if not os.path.exists(localfilesdir):
+          bb.utils.mkdirhier(localfilesdir)
+          #Add gitignore to ensure source tree is clean
+          gitignorefile = os.path.join(localfilesdir,'.gitignore')
+          with open(gitignorefile, 'w') as f:
+                  f.write('# Ignore local files, by default. Remove this file if you want to commit the directory to Git\n')
+                  f.write('*\n')
+
+    finally:
+      tinfoil.shutdown()
+
+    logger.info('Launching menuconfig')
+    exec_build_env_command(config.init_path, basepath, 'bitbake -c menuconfig %s' % pn, watch=True) 
+    fragment = os.path.join(localfilesdir, 'devtool-fragment.cfg')
+    res = standard._create_kconfig_diff(pn_src,rd,fragment)
+
+    return 0
+
+def register_commands(subparsers, context):
+    """register devtool subcommands from this plugin"""
+    parser_menuconfig = subparsers.add_parser('menuconfig',help='allows altering the system component configuration', description='launches the make menuconfig command, allows user to make changes to configuration. creates a config fragment', group='advanced') 
+    parser_menuconfig.add_argument('component', help='compenent to alter config')
+    parser_menuconfig.set_defaults(func=menuconfig,fixed_setup=context.fixed_setup)
-- 
2.7.4

Re: [PATCH v2 3/3] devtool: provide support for devtool menuconfig command.

[Chandana Kalluri]

Ping

-----Original Message-----
From: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com> 
Sent: Friday, January 25, 2019 11:53 AM
To: openembedded-core@lists.openembedded.org
Cc: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>; Chandana Kalluri <ckalluri@xilinx.com>
Subject: [OE-Core][PATCH v2 3/3] devtool: provide support for devtool menuconfig command.

All packages that support the menuconfig task will be able to run devtool menuconfig command. This would allow the user to modify the current configure options and create a config fragment which can be added to a recipe using devtool finish.

1. The patch checks if devtool menuconfig command is called for a valid package.
2. It checks for oe-local-files dir within source and creates one if needed, this directory is needed to store the final generated config fragment so that devtool finish can update the recipe.
3. Menuconfig command is called for users to make necessary changes. After saving the changes, diffconfig command is run to generate the fragment.

Syntax:
	devtool menuconfig <package name>
	 Ex: devtool menuconfig linux-yocto

The config fragment is saved as devtool-fragment.cfg within oe-local-files dir.

	Ex: <workspace_path>/sources/linux-yocto/oe-local-files/devtool-fragment.cfg

Run devtool finish to update the recipe by appending the config fragment to SRC_URI and place a copy of the fragment within the layer where the recipe resides.
	Ex: devtool finish linux-yocto meta

[YOCTO #10416]

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 scripts/lib/devtool/menuconfig.py | 80 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)
 create mode 100644 scripts/lib/devtool/menuconfig.py

diff --git a/scripts/lib/devtool/menuconfig.py b/scripts/lib/devtool/menuconfig.py
new file mode 100644
index 0000000..38133db
--- /dev/null
+++ b/scripts/lib/devtool/menuconfig.py
@@ -0,0 +1,80 @@
+# OpenEmbedded Development tool - menuconfig command plugin # # 
+Copyright (C) 2018 Xilinx # Written by: Chandana Kalluri 
+<ckalluri@xilinx.com> # # This program is free software; you can 
+redistribute it and/or modify # it under the terms of the GNU General 
+Public License version 2 as # published by the Free Software 
+Foundation.
+#
+# This program is distributed in the hope that it will be useful, # but 
+WITHOUT ANY WARRANTY; without even the implied warranty of # 
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the # GNU 
+General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License 
+along # with this program; if not, write to the Free Software 
+Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""Devtool menuconfig plugin"""
+
+import os
+import bb
+import logging
+import argparse
+import re
+import glob
+from devtool import setup_tinfoil, parse_recipe, DevtoolError, 
+standard, exec_build_env_command
+
+logger = logging.getLogger('devtool')
+
+def menuconfig(args, config, basepath, workspace):
+    """Entry point for the devtool 'menuconfig' subcommand"""
+
+    rd = "" 
+    kconfigpath = ""
+    pn_src = ""
+    localfilesdir = ""
+    workspace_dir = ""
+    tinfoil = setup_tinfoil(basepath=basepath)
+    try:
+      rd = parse_recipe(config, tinfoil, args.component, appends=True, filter_workspace=False)
+      if not rd:
+         return 1
+
+      pn =  rd.getVar('PN', True)
+      if pn not in workspace:
+         raise DevtoolError("Run devtool modify before calling 
+ menuconfig for %s" %pn)
+
+      if not rd.getVarFlag('do_menuconfig','task'):
+         raise DevtoolError("This package does not support menuconfig 
+ option")
+
+      workspace_dir = os.path.join(basepath,'workspace/sources')
+      kconfigpath = rd.getVar('B')
+      pn_src = os.path.join(workspace_dir,pn)
+
+      #add check to see if oe_local_files exists or not
+      localfilesdir = os.path.join(pn_src,'oe-local-files') 
+      if not os.path.exists(localfilesdir):
+          bb.utils.mkdirhier(localfilesdir)
+          #Add gitignore to ensure source tree is clean
+          gitignorefile = os.path.join(localfilesdir,'.gitignore')
+          with open(gitignorefile, 'w') as f:
+                  f.write('# Ignore local files, by default. Remove this file if you want to commit the directory to Git\n')
+                  f.write('*\n')
+
+    finally:
+      tinfoil.shutdown()
+
+    logger.info('Launching menuconfig')
+    exec_build_env_command(config.init_path, basepath, 'bitbake -c menuconfig %s' % pn, watch=True) 
+    fragment = os.path.join(localfilesdir, 'devtool-fragment.cfg')
+    res = standard._create_kconfig_diff(pn_src,rd,fragment)
+
+    return 0
+
+def register_commands(subparsers, context):
+    """register devtool subcommands from this plugin"""
+    parser_menuconfig = subparsers.add_parser('menuconfig',help='allows altering the system component configuration', description='launches the make menuconfig command, allows user to make changes to configuration. creates a config fragment', group='advanced') 
+    parser_menuconfig.add_argument('component', help='compenent to alter config')
+    
+parser_menuconfig.set_defaults(func=menuconfig,fixed_setup=context.fixe
+d_setup)
--
2.7.4

Re: [PATCH v2 1/3] devtool modify: Update devtool modify to copy source from work-shared if its already downloaded.

[Paul Eggleton]

Thanks for looking into this problem - this is a tricky one. My apologies for the delay in reviewing. Comments below.

On Saturday, 26 January 2019 8:53:00 AM NZDT Sai Hari Chandana Kalluri wrote:
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -717,6 +717,72 @@ def _check_preserve(config, recipename):
>                      tf.write(line)
>      os.rename(newfile, origfile)
>  
> +# Function links a file from src location to dest location
> +def copy_file(c,dest):
> +    import errno
> +    destdir = os.path.dirname(dest)
> +    if os.path.islink(c):
> +       linkto = os.readlink(c)
> +       if os.path.lexists(dest):
> +            if not os.path.islink(dest):
> +                raise OSError(errno.EEXIST, "Link %s already exists as a file" % dest, dest)
> +            if os.readlink(dest) == linkto:
> +                return dest
> +            raise OSError(errno.EEXIST, "Link %s already exists to a different location? (%s vs %s)" % (dest, os.readlink(dest), linkto), dest)
> +       os.symlink(linkto, dest)
> +    else:
> +       try:
> +           os.link(c, dest)
> +       except OSError as err:
> +           if err.errno == errno.EXDEV:
> +                bb.utils.copyfile(c, dest)
> +           else:
> +                raise
> +
> +# Function creates folders in a given target location
> +def copy_dirs(root,dirs,target):
> +    for d in dirs:
> +        destdir =  os.path.join(target,d)
> +        if os.path.islink(os.path.join(root,d)):
> +             linkto = os.readlink(os.path.join(root,d))
> +             os.symlink(linkto,destdir) 
> +        else:
> +             bb.utils.mkdirhier(target+d)
> +
> +# Function to link src dir to dest dir
> +def copy_src_to_ws(srcdir,srctree):
> +    target = srctree
> +    if os.path.exists(target):
> +        raise DevtoolError('source already in your workspace')
> +
> +    bb.utils.mkdirhier(target)
> +    for root,dirs,files in os.walk(srcdir):
> +        #convert abspath to relpath for root
> +        destdir = root.replace(srcdir,"")
> +        target = srctree+destdir+"/"
> +        copy_dirs(root,dirs,target)  
> +        for f in files:
> +           copy_file(os.path.join(root,f),os.path.join(target,f))

We already have a function to do this - oe.path.copyhardlinktree(). I appreciate that does not symlink if it can't hard link as you're doing here, but given the side-effects if the kernel source goes away I think we'd prefer "hard link or copy" rather than "hard link or symlink".

> @@ -852,10 +990,12 @@ def modify(args, config, basepath, workspace):
>                  f.write('\ndo_patch() {\n'
>                          '    :\n'
>                          '}\n')
> -                f.write('\ndo_configure_append() {\n'
> -                        '    cp ${B}/.config ${S}/.config.baseline\n'
> -                        '    ln -sfT ${B}/.config ${S}/.config.new\n'
> -                        '}\n')
> +
> +            if rd.getVarFlag('do_menuconfig','task'):
> +                    f.write('\ndo_configure_append() {\n'
> +                    '    cp ${B}/.config ${S}/.config.baseline\n'
> +                    '    ln -sfT ${B}/.config ${S}/.config.new\n'
> +                    '}\n')
>              if initial_rev:
>                  f.write('\n# initial_rev: %s\n' % initial_rev)
>                  for commit in commits:

This change does not appear to be directly related to the rest of the commit, can it be split out and properly described?

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 3/3] devtool: provide support for devtool menuconfig command.

[Paul Eggleton]

On Saturday, 26 January 2019 8:53:02 AM NZDT Sai Hari Chandana Kalluri wrote:
> +def menuconfig(args, config, basepath, workspace):
> +    """Entry point for the devtool 'menuconfig' subcommand"""
> +
> +    rd = "" 
> +    kconfigpath = ""
> +    pn_src = ""
> +    localfilesdir = ""
> +    workspace_dir = ""
> +    tinfoil = setup_tinfoil(basepath=basepath)
> +    try:
> +      rd = parse_recipe(config, tinfoil, args.component, appends=True, filter_workspace=False)
> +      if not rd:
> +         return 1
> +
> +      pn =  rd.getVar('PN', True)
> +      if pn not in workspace:
> +         raise DevtoolError("Run devtool modify before calling menuconfig for %s" %pn)

Strictly speaking we have check_workspace_recipe() for this. It could be that we should extend the message printed by that to be a bit more friendly and suggest devtool add/modify/upgrade first, but I'd like to be consistent.

>...
> +      #add check to see if oe_local_files exists or not
> +      localfilesdir = os.path.join(pn_src,'oe-local-files') 
> +      if not os.path.exists(localfilesdir):
> +          bb.utils.mkdirhier(localfilesdir)
> +          #Add gitignore to ensure source tree is clean
> +          gitignorefile = os.path.join(localfilesdir,'.gitignore')
> +          with open(gitignorefile, 'w') as f:
> +                  f.write('# Ignore local files, by default. Remove this file if you want to commit the directory to Git\n')
> +                  f.write('*\n')

We should already be handling this in devtool-source.bbclass, but I'm guessing the hardlinking you're adding bypasses that, so we likely need to handle that separately during devtool modify rather than here (preferably without duplicating code).

> +    parser_menuconfig = subparsers.add_parser('menuconfig',help='allows altering the system component configuration', description='launches the make menuconfig command, allows user to make changes to configuration. creates a config fragment', group='advanced') 
> +    parser_menuconfig.add_argument('component', help='compenent to alter config')

To be consistent with other subcommands and a bit more readable this should be something like:

    parser_menuconfig = subparsers.add_parser('menuconfig', help='Alter build-time configuration for a recipe', description='Launches the "make menuconfig" command (for recipes where do_menuconfig is available), allowing you to make changes to the build-time configuration. Creates a config fragment corresponding to changes made.', group='advanced') 
    parser_menuconfig.add_argument('recipename', help='Recipe to reconfigure')

Additionally, I would really like to see some oe-selftest tests for this along with the changes.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 3/3] devtool: provide support for devtool menuconfig command.

[Chandana Kalluri]

-----Original Message-----
From: Paul Eggleton <paul.eggleton@linux.intel.com> 
Sent: Monday, March 4, 2019 3:46 PM
To: Chandana Kalluri <ckalluri@xilinx.com>
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [OE-Core][PATCH v2 3/3] devtool: provide support for devtool menuconfig command.

On Saturday, 26 January 2019 8:53:02 AM NZDT Sai Hari Chandana Kalluri wrote:
> +def menuconfig(args, config, basepath, workspace):
> +    """Entry point for the devtool 'menuconfig' subcommand"""
> +
> +    rd = "" 
> +    kconfigpath = ""
> +    pn_src = ""
> +    localfilesdir = ""
> +    workspace_dir = ""
> +    tinfoil = setup_tinfoil(basepath=basepath)
> +    try:
> +      rd = parse_recipe(config, tinfoil, args.component, appends=True, filter_workspace=False)
> +      if not rd:
> +         return 1
> +
> +      pn =  rd.getVar('PN', True)
> +      if pn not in workspace:
> +         raise DevtoolError("Run devtool modify before calling menuconfig for %s" %pn)

Strictly speaking we have check_workspace_recipe() for this. It could be that we should extend the message printed by that to be a bit more friendly and suggest devtool add/modify/upgrade first, but I'd like to be consistent.

>...
> +      #add check to see if oe_local_files exists or not
> +      localfilesdir = os.path.join(pn_src,'oe-local-files') 
> +      if not os.path.exists(localfilesdir):
> +          bb.utils.mkdirhier(localfilesdir)
> +          #Add gitignore to ensure source tree is clean
> +          gitignorefile = os.path.join(localfilesdir,'.gitignore')
> +          with open(gitignorefile, 'w') as f:
> +                  f.write('# Ignore local files, by default. Remove this file if you want to commit the directory to Git\n')
> +                  f.write('*\n')

We should already be handling this in devtool-source.bbclass, but I'm guessing the hardlinking you're adding bypasses that, so we likely need to handle that separately during devtool modify rather than here (preferably without duplicating code).

This check for oe_local_files is specific for all other packages except linux-yocto. This is needed if the package uses make menuconfig to generate cfg and for devtool finish to pick up the cfg fragment. 
For linux-yocto, after the hardlinking is done, the check for oe-local-files is taken care for within devtool modify( this is in PATCH 1/3 ). 

The above check for oe-local-files for all packages could be handled within devtool modify and can be done in two ways:
1. Either all packages(even the ones that do not run menuconfig command) have an empty oe-local-files dir as a part of workspace/source OR
2. have a check within modify to see if it runs menuconfig task and create the oe-local-files. 

What would you recommend ?

> +    parser_menuconfig = subparsers.add_parser('menuconfig',help='allows altering the system component configuration', description='launches the make menuconfig command, allows user to make changes to configuration. creates a config fragment', group='advanced') 
> +    parser_menuconfig.add_argument('component', help='compenent to alter config')

To be consistent with other subcommands and a bit more readable this should be something like:

    parser_menuconfig = subparsers.add_parser('menuconfig', help='Alter build-time configuration for a recipe', description='Launches the "make menuconfig" command (for recipes where do_menuconfig is available), allowing you to make changes to the build-time configuration. Creates a config fragment corresponding to changes made.', group='advanced') 
    parser_menuconfig.add_argument('recipename', help='Recipe to reconfigure')

Additionally, I would really like to see some oe-selftest tests for this along with the changes.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre