[PATCH v2 0/2] Make source code browsable locally - devtool srctree

[PATCH v2 0/2] Make source code browsable locally - devtool srctree

[Tobias Hagelborn]

UPDATE v2:
- Rebased on latest Poky
- Fixed issue with bad paths
- Dirname based on $PN

In our organization, there is a demand to be able download source code for searching.
With the use of sstate-cache this has been challenge. Also, the source is not gathered
but rather spread in a larger directory structure together with build output.
There are probably others that also want to be able to do the same.

This addition does:
* Force download of all source via the srctree_all task
* Symlink the source together in a separate directory structure for easier search without
  the risk of matching with build output.

This is achived with a new class srctree.bbclass that implements the needed tasks and
a devtool command srctree which offers a convenience command to set up source for 
a recipe and it's dependencies.

There is also a way of organising the package source code in your own way.
This is done by implementing a custom dir generation function and assign to SRCTREE_PN_PATH
in a custom class and set SRCTREE_EXTENSION in a configuration file.


Tobias Hagelborn (2):
  srctree.bbclass: Download sourcecode and make it searchable
  devtool: srctree: New command for setting up browsable source-code

 meta/classes/srctree.bbclass   | 90 ++++++++++++++++++++++++++++++++++++++++++
 scripts/lib/devtool/srctree.py | 87 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 177 insertions(+)
 create mode 100644 meta/classes/srctree.bbclass
 create mode 100644 scripts/lib/devtool/srctree.py

-- 
2.1.4

[PATCH v2 1/2] srctree.bbclass: Download sourcecode and make it searchable

[Tobias Hagelborn]

This class is used to create a symlink to the source in ${S} in a
separate directory structure. ${SECTION} is used, if available, for
naming the path.

Intended for 'devtool srctree <recipe>' to set up a tree with the
source for all specified recipes and their dependencies to be able
to do a more convenient local search in source code.

Example:

nfs-utils (console/network) maps to ->
   console/
     network/
       nfs-utils

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
---
 meta/classes/srctree.bbclass | 90 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)
 create mode 100644 meta/classes/srctree.bbclass

diff --git a/meta/classes/srctree.bbclass b/meta/classes/srctree.bbclass
new file mode 100644
index 0000000..40351d8
--- /dev/null
+++ b/meta/classes/srctree.bbclass
@@ -0,0 +1,90 @@
+# Create symlinks to source (${S}) in a separate directory structure.
+#
+# Copyright (C) 2016-2017 Axis Communications AB
+# Author: Tobias Hagelborn <tobias.hagelborn@axis.com>
+#
+# Released under the MIT license (see COPYING.MIT for the terms)
+#
+# This class is used to create a symlink to the source in ${S} in a
+# separate directory structure. ${SECTION} is used, if available, for
+# naming the path.
+#
+# Intended for 'devtool srctree <recipe>' to set up a tree with the
+# source for all specified recipes and their dependencies to be able
+# to do a more convenient local search in source code.
+#
+# Example:
+#
+# nfs-utils (console/network) maps to ->
+#   console/
+#     network/
+#       nfs-utils
+
+# Use the following to extend the srctree class with custom functions
+SRCTREE_EXTENSION ?= ""
+
+inherit ${SRCTREE_EXTENSION}
+
+# Default PN sub-path function
+def srctree_get_path(d):
+    section = d.getVar("SECTION", True)
+    pn = d.getVar("PN", True)
+    return os.path.join(section.split()[0], pn) if section else pn
+
+# Configurable browsable source tree root
+SRCTREE_DIR ?= "${TOPDIR}/src"
+
+# Path within ${SRCTREE_DIR} for the current PN. Can be overridden in
+# a custom class via ${SRCTREE_EXTENSION} or set to a fixed value.
+# Example: SRCTREE_PN_PATH = "${PN}"
+SRCTREE_PN_PATH ?= "${@srctree_get_path(d)}"
+SYM_S = "${SRCTREE_DIR}/${SRCTREE_PN_PATH}"
+
+create_srctree_readme() {
+	readme="${SRCTREE_DIR}/README"
+
+	[ -e "$readme" ] || cat >"$readme" <<EOF
+This directory contains a directory tree for source code browsing.
+
+IMPORTANT:
+
+ * The source tree is intended for CODE BROWSING ONLY.
+ * The tree contains symbolic links directly to the source as used by BitBake.
+ * Any CHANGES made to the source code MAY BE REMOVED by BitBake AT ANY TIME.
+ * BitBake will NOT automatically rebuild packages due to code changes made
+   directly in the source tree (which you should not do in the first place).
+ * The source tree will NOT reflect changes made to recipes or, e.g., by
+   running "devtool modify -x <recipe>" until "devtool srctree" is re-executed.
+EOF
+}
+
+do_srctree[nostamp] = "1"
+do_srctree[dirs] = "${SRCTREE_DIR}"
+addtask srctree after do_patch
+do_srctree() {
+	if [ "${S}" = "${WORKDIR}" ]; then
+		[ "${SRCTREE_VERBOSE}" != 1 ] ||
+			bbwarn "Not setting up a link for '${SRCTREE_PN_PATH}' (since $""{S} = $""{WORKDIR})"
+		return
+	fi
+
+	# Ignore all cases where S is not properly set up for any reason
+	if [ ! -d "${S}" ]; then
+		[ "${SRCTREE_VERBOSE}" != 1 ] ||
+			bbwarn "No source available for ${PN}";
+		return
+	fi
+
+	mkdir -p $(dirname "${SYM_S}")
+	ln -sfn ${S} ${SYM_S}
+
+	create_srctree_readme
+}
+
+addtask srctree_all after do_srctree
+do_srctree_all[recrdeptask] = "do_srctree_all do_srctree"
+do_srctree_all[recideptask] = "do_${BB_DEFAULT_TASK}"
+do_srctree_all[doc] = "Setup source tree recursivly"
+do_srctree_all() {
+	:
+}
-- 
2.1.4

[PATCH v2 2/2] devtool: srctree: New command for setting up browsable source-code

[Tobias Hagelborn]

Setup a directory tree for source code according to section name.
(if known).

This tree is intended for code browsing ONLY.
If source is moved, for instance with devtool modify,
this command should be rerun to be up-to-date with that move.

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
---
 scripts/lib/devtool/srctree.py | 87 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)
 create mode 100644 scripts/lib/devtool/srctree.py

diff --git a/scripts/lib/devtool/srctree.py b/scripts/lib/devtool/srctree.py
new file mode 100644
index 0000000..dbf1608
--- /dev/null
+++ b/scripts/lib/devtool/srctree.py
@@ -0,0 +1,87 @@
+# Development tool - srctree command plugin
+#
+# Copyright (C) 2016-2017 Axis Communications AB
+# Author: Tobias Hagelborn <tobias.hagelborn@axis.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+"""Devtool srctree plugin"""
+
+import os
+import logging
+import tempfile
+import bb
+
+from devtool import exec_build_env_command
+
+logger = logging.getLogger('devtool')
+
+
+def _create_conf_file(values, conf_file=None):
+    """ Create bitbake config for use with srctree devtool command """
+    if not conf_file:
+        _, conf_file = tempfile.mkstemp(suffix='.conf')
+    elif not os.path.exists(os.path.dirname(conf_file)):
+        logger.debug("Creating folder %s", os.path.dirname(conf_file))
+        bb.utils.mkdirhier(os.path.dirname(conf_file))
+    if values:
+        with open(conf_file, 'w') as f:
+            for key, value in values.items():
+                f.write('%s = "%s"\n' % (key, value))
+    return conf_file
+
+
+#pylint: disable=unused-argument
+def srctree(args, config, basepath, workspace):
+    """Entry point for the devtool 'srctree' subcommand"""
+
+    values = {}
+    if args.destination:
+        values['SRCTREE_DIR'] = os.path.abspath(args.destination)
+        logger.info('Setting up srctree in: %s', values['SRCTREE_DIR'])
+    if args.verbose:
+        values['SRCTREE_VERBOSE'] = "1"
+    conf_file = _create_conf_file(values)
+
+    with open(conf_file, 'a') as f:
+        f.write('INHERIT += "srctree"\n')
+
+    workdir = basepath
+    exec_build_env_command(
+        config.init_path, workdir,
+        'bitbake -k -R %s -c srctree_all %s' % (conf_file, ' '.join(args.recipename)),
+        watch=True)
+    os.unlink(conf_file)
+
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    parser_search = subparsers.add_parser(
+        'srctree',
+        help='Setup a directory tree for source code browsing',
+        description='Setup a directory tree for source code according to section name (if known). '
+                    'This tree is intended for code browsing ONLY. '
+                    'If source is moved, for instance with devtool modify, '
+                    'this command should be rerun to be up-to-date with that move. '
+                    'Please read the README file that is created together with '
+                    'the source tree for more information.')
+    parser_search.add_argument('recipename', nargs='+',
+                               help='Recipe to use. Multiple recipes are accepted. '
+                               'Example: core-image-minimal')
+    parser_search.add_argument('-D', '--destination',
+                               help='Destination root folder. Default: ${TOPDIR}/src.')
+    parser_search.add_argument('-v', '--verbose', action="store_true",
+                               help='Be more verbose.')
+    parser_search.set_defaults(func=srctree)
-- 
2.1.4

Re: [PATCH v2 0/2] Make source code browsable locally - devtool srctree

[Tobias Hagelborn]

On 07/07/2017 07:32 AM, Tobias Hagelborn wrote:
> UPDATE v2:
> - Rebased on latest Poky
> - Fixed issue with bad paths
> - Dirname based on $PN
> 
> In our organization, there is a demand to be able download source code for searching.
> With the use of sstate-cache this has been challenge. Also, the source is not gathered
> but rather spread in a larger directory structure together with build output.
> There are probably others that also want to be able to do the same.
> 
> This addition does:
> * Force download of all source via the srctree_all task
> * Symlink the source together in a separate directory structure for easier search without
>    the risk of matching with build output.
> 
> This is achived with a new class srctree.bbclass that implements the needed tasks and
> a devtool command srctree which offers a convenience command to set up source for
> a recipe and it's dependencies.
> 
> There is also a way of organising the package source code in your own way.
> This is done by implementing a custom dir generation function and assign to SRCTREE_PN_PATH
> in a custom class and set SRCTREE_EXTENSION in a configuration file.
> 
> 
> Tobias Hagelborn (2):
>    srctree.bbclass: Download sourcecode and make it searchable
>    devtool: srctree: New command for setting up browsable source-code
> 
>   meta/classes/srctree.bbclass   | 90 ++++++++++++++++++++++++++++++++++++++++++
>   scripts/lib/devtool/srctree.py | 87 ++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 177 insertions(+)
>   create mode 100644 meta/classes/srctree.bbclass
>   create mode 100644 scripts/lib/devtool/srctree.py
> 

Hi

I did not receive any feedback on this one.

We do make good use of this feature and we can't be the only ones who 
want to browse the source of the distro at times?

Cheers
Tobias

Re: [PATCH v2 0/2] Make source code browsable locally - devtool srctree

[Andre McCurdy]

On Wed, Aug 30, 2017 at 2:03 AM, Tobias Hagelborn
<tobias.hagelborn@axis.com> wrote:
> On 07/07/2017 07:32 AM, Tobias Hagelborn wrote:
>>
>> UPDATE v2:
>> - Rebased on latest Poky
>> - Fixed issue with bad paths
>> - Dirname based on $PN
>>
>> In our organization, there is a demand to be able download source code for
>> searching.
>> With the use of sstate-cache this has been challenge. Also, the source is
>> not gathered
>> but rather spread in a larger directory structure together with build
>> output.
>> There are probably others that also want to be able to do the same.
>>
>> This addition does:
>> * Force download of all source via the srctree_all task
>> * Symlink the source together in a separate directory structure for easier
>> search without
>>    the risk of matching with build output.
>>
>> This is achived with a new class srctree.bbclass that implements the
>> needed tasks and
>> a devtool command srctree which offers a convenience command to set up
>> source for
>> a recipe and it's dependencies.
>>
>> There is also a way of organising the package source code in your own way.
>> This is done by implementing a custom dir generation function and assign
>> to SRCTREE_PN_PATH
>> in a custom class and set SRCTREE_EXTENSION in a configuration file.
>>
>>
>> Tobias Hagelborn (2):
>>    srctree.bbclass: Download sourcecode and make it searchable
>>    devtool: srctree: New command for setting up browsable source-code
>>
>>   meta/classes/srctree.bbclass   | 90
>> ++++++++++++++++++++++++++++++++++++++++++
>>   scripts/lib/devtool/srctree.py | 87
>> ++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 177 insertions(+)
>>   create mode 100644 meta/classes/srctree.bbclass
>>   create mode 100644 scripts/lib/devtool/srctree.py
>>
>
> Hi
>
> I did not receive any feedback on this one.
>
> We do make good use of this feature and we can't be the only ones who want
> to browse the source of the distro at times?

Just for reference, I think you could achieve something quite similar with:

  bitbake <your-image> --runall clean  ( or rm -rf tmp )
  bitbake <your-image> --runall patch


> Cheers
> Tobias
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH v2 0/2] Make source code browsable locally - devtool srctree

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Andre McCurdy
> Sent: den 31 augusti 2017 01:17
> To: Tobias Hagelborn <Tobias.Hagelborn@axis.com>
> Cc: Tobias Hagelborn <Tobias.Hagelborn@axis.com>; OE Core mailing list
> <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH v2 0/2] Make source code browsable
> locally - devtool srctree
> 
> On Wed, Aug 30, 2017 at 2:03 AM, Tobias Hagelborn
> <tobias.hagelborn@axis.com> wrote:
> > On 07/07/2017 07:32 AM, Tobias Hagelborn wrote:
> >>
> >> UPDATE v2:
> >> - Rebased on latest Poky
> >> - Fixed issue with bad paths
> >> - Dirname based on $PN
> >>
> >> In our organization, there is a demand to be able download source
> code for
> >> searching.
> >> With the use of sstate-cache this has been challenge. Also, the
> source is
> >> not gathered
> >> but rather spread in a larger directory structure together with
> build
> >> output.
> >> There are probably others that also want to be able to do the same.
> >>
> >> This addition does:
> >> * Force download of all source via the srctree_all task
> >> * Symlink the source together in a separate directory structure for
> easier
> >> search without
> >>    the risk of matching with build output.
> >>
> >> This is achived with a new class srctree.bbclass that implements the
> >> needed tasks and
> >> a devtool command srctree which offers a convenience command to set
> up
> >> source for
> >> a recipe and it's dependencies.
> >>
> >> There is also a way of organising the package source code in your
> own way.
> >> This is done by implementing a custom dir generation function and
> assign
> >> to SRCTREE_PN_PATH
> >> in a custom class and set SRCTREE_EXTENSION in a configuration file.
> >>
> >>
> >> Tobias Hagelborn (2):
> >>    srctree.bbclass: Download sourcecode and make it searchable
> >>    devtool: srctree: New command for setting up browsable source-
> code
> >>
> >>   meta/classes/srctree.bbclass   | 90
> >> ++++++++++++++++++++++++++++++++++++++++++
> >>   scripts/lib/devtool/srctree.py | 87
> >> ++++++++++++++++++++++++++++++++++++++++
> >>   2 files changed, 177 insertions(+)
> >>   create mode 100644 meta/classes/srctree.bbclass
> >>   create mode 100644 scripts/lib/devtool/srctree.py
> >>
> >
> > Hi
> >
> > I did not receive any feedback on this one.
> >
> > We do make good use of this feature and we can't be the only ones who
> want
> > to browse the source of the distro at times?
> 
> Just for reference, I think you could achieve something quite similar
> with:
> 
>   bitbake <your-image> --runall clean  ( or rm -rf tmp )
>   bitbake <your-image> --runall patch

Yes, but the drawback with this is that to find the source code for the 
individual packages, you have to dive down into tmp/work/... which can 
be quite discouraging for someone who is not an expert in bitbake. What 
our developers requested was a way to get easy access to all the source 
code being used, typically to be able to search through it all looking 
for some specific feature.

> > Cheers
> > Tobias

//Peter

Re: [PATCH v2 0/2] Make source code browsable locally - devtool srctree

[Andre McCurdy]

On Wed, Aug 30, 2017 at 11:32 PM, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>>
>> Just for reference, I think you could achieve something quite similar
>> with:
>>
>>   bitbake <your-image> --runall clean  ( or rm -rf tmp )
>>   bitbake <your-image> --runall patch
>
> Yes, but the drawback with this is that to find the source code for the
> individual packages, you have to dive down into tmp/work/... which can
> be quite discouraging for someone who is not an expert in bitbake. What
> our developers requested was a way to get easy access to all the source
> code being used, typically to be able to search through it all looking
> for some specific feature.

Right. I wasn't trying to suggest that the two approaches were
completely equivalent, just drawing some attention to the bitbake
--runall option, which was added fairly recently (specifically to
solve this problem).

The directory structure split under tmp/work between arch and machine
specific recipes might be a little confusing but obviously serves a
purpose (target and native sources may need different sets of patches,
etc) so maybe worth the effort to explain to developers. Below that
level I don't think there are many surprises in the paths to the
sources directories?

Re: [PATCH v2 0/2] Make source code browsable locally - devtool srctree

[Peter Kjellerstedt]

> -----Original Message-----
> From: Andre McCurdy [mailto:armccurdy@gmail.com]
> Sent: den 31 augusti 2017 22:10
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: Tobias Hagelborn <Tobias.Hagelborn@axis.com>; OE Core mailing list
> <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH v2 0/2] Make source code browsable
> locally - devtool srctree
> 
> On Wed, Aug 30, 2017 at 11:32 PM, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> >>
> >> Just for reference, I think you could achieve something quite
> similar
> >> with:
> >>
> >>   bitbake <your-image> --runall clean  ( or rm -rf tmp )
> >>   bitbake <your-image> --runall patch
> >
> > Yes, but the drawback with this is that to find the source code for
> the
> > individual packages, you have to dive down into tmp/work/... which
> can
> > be quite discouraging for someone who is not an expert in bitbake.
> What
> > our developers requested was a way to get easy access to all the
> source
> > code being used, typically to be able to search through it all
> looking
> > for some specific feature.
> 
> Right. I wasn't trying to suggest that the two approaches were
> completely equivalent, just drawing some attention to the bitbake
> --runall option, which was added fairly recently (specifically to
> solve this problem).
> 
> The directory structure split under tmp/work between arch and machine
> specific recipes might be a little confusing but obviously serves a
> purpose (target and native sources may need different sets of patches,
> etc) so maybe worth the effort to explain to developers. Below that
> level I don't think there are many surprises in the paths to the
> sources directories?

Well, the most common request we've gotten from the developers is that 
they want to be able to search through all the sources, e.g., to find 
out where a particular function is used. The tmp directory does not 
lend itself very well to this, given that it also contains all build 
trees, applied patches and other meta information that bitbake needs.
Thus the idea for the parallel tree with only the source references.

//Peter

Re: [PATCH v2 0/2] Make source code browsable locally - devtool srctree

[Randy MacLeod]

On 2017-09-01 08:00 AM, Peter Kjellerstedt wrote:
> Well, the most common request we've gotten from the developers is that
> they want to be able to search through all the sources, e.g., to find
> out where a particular function is used. The tmp directory does not
> lend itself very well to this, given that it also contains all build
> trees, applied patches and other meta information that bitbake needs.
> Thus the idea for the parallel tree with only the source references.
> 
> //Peter

A shadow tree of links to the source is "mostly harmless" to experts
and useful for newbies and code browsers as you explained.

The src dir should be under tmp/src though since anything
that can be regenerated shouldn't be in the top level
project directory.

We don't pay that much (enough?) attention to the SECTION tag
and something like half of the oe-core recicpes doen't
have one. I suppose this feature would encourage us to
fix that.

Have you run your script with meta-openembedded and other layers?

-- 
# Randy MacLeod.  WR Linux
# Wind River an Intel Company

Re: [PATCH v2 0/2] Make source code browsable locally - devtool srctree

[Tobias Hagelborn]

On 09/05/2017 02:10 PM, Randy MacLeod wrote:
> On 2017-09-01 08:00 AM, Peter Kjellerstedt wrote:
>> Well, the most common request we've gotten from the developers is that
>> they want to be able to search through all the sources, e.g., to find
>> out where a particular function is used. The tmp directory does not
>> lend itself very well to this, given that it also contains all build
>> trees, applied patches and other meta information that bitbake needs.
>> Thus the idea for the parallel tree with only the source references.
>>
>> //Peter
> 
> A shadow tree of links to the source is "mostly harmless" to experts
> and useful for newbies and code browsers as you explained.
> 
> The src dir should be under tmp/src though since anything
> that can be regenerated shouldn't be in the top level
> project directory.

I'll fix that: tmp/srctree

> 
> We don't pay that much (enough?) attention to the SECTION tag
> and something like half of the oe-core recicpes doen't
> have one. I suppose this feature would encourage us to
> fix that.
> 
> Have you run your script with meta-openembedded and other layers?

I've run it with core-image-minimal and core-image-sato.
(I do not think they pull anything in from meta-openembedded)


A new patchset  will soon be available.


//Tobias