[PATCH v2 1/2] binman: Do not pollute source tree when build with `make O=...`

[PATCH v2 1/2] binman: Do not pollute source tree when build with `make O=...`

[Andy Shevchenko]

Importing libraries in Python caches the bytecode by default.
Since we run scripts in source tree it ignores the current directory
settings, which is $(srctree), and creates cache just in the middle
of the source tree. Move cache to the current directory.

Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
v2: reused our_path
 tools/binman/main.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/tools/binman/main.py b/tools/binman/main.py
index 8c1e478d54ce..4d8b124c7468 100755
--- a/tools/binman/main.py
+++ b/tools/binman/main.py
@@ -16,9 +16,18 @@ import sys
 import traceback
 import unittest
 
+# Get the absolute path to this file at run-time
+our_path = os.path.dirname(sys.argv[0])
+
+#
+# Do not pollute source tree with cache files:
+# https://stackoverflow.com/a/60024195/2511795
+# https://bugs.python.org/issue33499
+#
+sys.pycache_prefix = os.path.relpath(our_path, os.environ['srctree'])
+
 # Bring in the patman and dtoc libraries (but don't override the first path
 # in PYTHONPATH)
-our_path = os.path.dirname(os.path.realpath(__file__))
 sys.path.insert(2, os.path.join(our_path, '..'))
 
 from patman import test_util
-- 
2.33.0

[PATCH v2 2/2] binman: Use less hard coded magic when inserting new PATH

[Andy Shevchenko]

Instead of joining hard coded '..' to the run-time path of the executable,
take just a dirname out of it. Besides that, use $(srctree) where it makes
sense.

Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
v2: new patch
 tools/binman/main.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/tools/binman/main.py b/tools/binman/main.py
index 4d8b124c7468..2b3e58d1aabd 100755
--- a/tools/binman/main.py
+++ b/tools/binman/main.py
@@ -19,24 +19,26 @@ import unittest
 # Get the absolute path to this file at run-time
 our_path = os.path.dirname(sys.argv[0])
 
+# Extract $(srctree)  from Kbuild environment
+srctree = os.environ['srctree']
+
 #
 # Do not pollute source tree with cache files:
 # https://stackoverflow.com/a/60024195/2511795
 # https://bugs.python.org/issue33499
 #
-sys.pycache_prefix = os.path.relpath(our_path, os.environ['srctree'])
+sys.pycache_prefix = os.path.relpath(our_path, srctree)
 
 # Bring in the patman and dtoc libraries (but don't override the first path
 # in PYTHONPATH)
-sys.path.insert(2, os.path.join(our_path, '..'))
+sys.path.insert(2, os.path.dirname(our_path))
 
 from patman import test_util
 
 # Bring in the libfdt module
 sys.path.insert(2, 'scripts/dtc/pylibfdt')
-sys.path.insert(2, os.path.join(our_path, '../../scripts/dtc/pylibfdt'))
-sys.path.insert(2, os.path.join(our_path,
-                '../../build-sandbox_spl/scripts/dtc/pylibfdt'))
+sys.path.insert(2, os.path.join(srctree, 'scripts/dtc/pylibfdt'))
+sys.path.insert(2, os.path.join(srctree, 'build-sandbox_spl/scripts/dtc/pylibfdt'))
 
 # When running under python-coverage on Ubuntu 16.04, the dist-packages
 # directories are dropped from the python path. Add them in so that we can find
-- 
2.33.0

Re: [PATCH v2 1/2] binman: Do not pollute source tree when build with `make O=...`

[Simon Glass]

Hi Andy,

On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> Importing libraries in Python caches the bytecode by default.
> Since we run scripts in source tree it ignores the current directory
> settings, which is $(srctree), and creates cache just in the middle
> of the source tree. Move cache to the current directory.
>
> Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> ---
> v2: reused our_path
>  tools/binman/main.py | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)

This look useful, but we cannot rely on 'srcdir' being in the
environment. For example, most binman development is done just by
running 'binman test' in the source tre. So perhaps default to the
current directory is 'srcdir' is not set?

Regards,
Simon


>
> diff --git a/tools/binman/main.py b/tools/binman/main.py
> index 8c1e478d54ce..4d8b124c7468 100755
> --- a/tools/binman/main.py
> +++ b/tools/binman/main.py
> @@ -16,9 +16,18 @@ import sys
>  import traceback
>  import unittest
>
> +# Get the absolute path to this file at run-time
> +our_path = os.path.dirname(sys.argv[0])
> +
> +#
> +# Do not pollute source tree with cache files:
> +# https://stackoverflow.com/a/60024195/2511795
> +# https://bugs.python.org/issue33499
> +#
> +sys.pycache_prefix = os.path.relpath(our_path, os.environ['srctree'])
> +
>  # Bring in the patman and dtoc libraries (but don't override the first path
>  # in PYTHONPATH)
> -our_path = os.path.dirname(os.path.realpath(__file__))
>  sys.path.insert(2, os.path.join(our_path, '..'))
>
>  from patman import test_util
> --
> 2.33.0
>

Re: [PATCH v2 2/2] binman: Use less hard coded magic when inserting new PATH

[Simon Glass]

Hi Andy,

On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> Instead of joining hard coded '..' to the run-time path of the executable,
> take just a dirname out of it. Besides that, use $(srctree) where it makes
> sense.
>
> Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> ---
> v2: new patch
>  tools/binman/main.py | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)

Looks good, apart from my comment on the previous patch, which applies here too.

Regards,
Simon

Re: [PATCH v2 1/2] binman: Do not pollute source tree when build with `make O=...`

[Andy Shevchenko]

On Friday, December 3, 2021, Simon Glass <sjg@chromium.org> wrote:

> Hi Andy,
>
> On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko
> <andriy.shevchenko@linux.intel.com> wrote:
> >
> > Importing libraries in Python caches the bytecode by default.
> > Since we run scripts in source tree it ignores the current directory
> > settings, which is $(srctree), and creates cache just in the middle
> > of the source tree. Move cache to the current directory.
> >
> > Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> > ---
> > v2: reused our_path
> >  tools/binman/main.py | 11 ++++++++++-
> >  1 file changed, 10 insertions(+), 1 deletion(-)
>
> This look useful, but we cannot rely on 'srcdir' being in the
> environment.


True and code is aware of that. Nothing needs to be fixed.

>
> For example, most binman development is done just by
> running 'binman test' in the source tre. So perhaps default to the
> current directory is 'srcdir' is not set?
>
> Regards,
> Simon
>
>
> >
> > diff --git a/tools/binman/main.py b/tools/binman/main.py
> > index 8c1e478d54ce..4d8b124c7468 100755
> > --- a/tools/binman/main.py
> > +++ b/tools/binman/main.py
> > @@ -16,9 +16,18 @@ import sys
> >  import traceback
> >  import unittest
> >
> > +# Get the absolute path to this file at run-time
> > +our_path = os.path.dirname(sys.argv[0])
> > +
> > +#
> > +# Do not pollute source tree with cache files:
> > +# https://stackoverflow.com/a/60024195/2511795
> > +# https://bugs.python.org/issue33499
> > +#
> > +sys.pycache_prefix = os.path.relpath(our_path, os.environ['srctree'])
> > +
> >  # Bring in the patman and dtoc libraries (but don't override the first
> path
> >  # in PYTHONPATH)
> > -our_path = os.path.dirname(os.path.realpath(__file__))
> >  sys.path.insert(2, os.path.join(our_path, '..'))
> >
> >  from patman import test_util
> > --
> > 2.33.0
> >
>


-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v2 1/2] binman: Do not pollute source tree when build with `make O=...`

[Simon Glass]

Hi Andy,

On Fri, 3 Dec 2021 at 00:55, Andy Shevchenko <andy.shevchenko@gmail.com> wrote:
>
>
>
> On Friday, December 3, 2021, Simon Glass <sjg@chromium.org> wrote:
>>
>> Hi Andy,
>>
>> On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko
>> <andriy.shevchenko@linux.intel.com> wrote:
>> >
>> > Importing libraries in Python caches the bytecode by default.
>> > Since we run scripts in source tree it ignores the current directory
>> > settings, which is $(srctree), and creates cache just in the middle
>> > of the source tree. Move cache to the current directory.
>> >
>> > Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
>> > ---
>> > v2: reused our_path
>> >  tools/binman/main.py | 11 ++++++++++-
>> >  1 file changed, 10 insertions(+), 1 deletion(-)
>>
>> This look useful, but we cannot rely on 'srcdir' being in the
>> environment.
>
>
> True and code is aware of that. Nothing needs to be fixed.

What am I missing?

$ binman test
Traceback (most recent call last):
  File "/home/sglass/bin/binman", line 23, in <module>
    srctree = os.environ['srctree']
  File "/usr/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'srctree'

>>
>>
>> For example, most binman development is done just by
>> running 'binman test' in the source tre. So perhaps default to the
>> current directory is 'srcdir' is not set?
>>
[..]

Regards,
Simon

Re: [PATCH v2 1/2] binman: Do not pollute source tree when build with `make O=...`

[Andy Shevchenko]

On Fri, Dec 03, 2021 at 01:13:12PM -0700, Simon Glass wrote:
> On Fri, 3 Dec 2021 at 00:55, Andy Shevchenko <andy.shevchenko@gmail.com> wrote:
> > On Friday, December 3, 2021, Simon Glass <sjg@chromium.org> wrote:
> >> On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko
> >> <andriy.shevchenko@linux.intel.com> wrote:

...

> >> This look useful, but we cannot rely on 'srcdir' being in the
> >> environment.
> >
> > True and code is aware of that. Nothing needs to be fixed.
> 
> What am I missing?
> 
> $ binman test
> Traceback (most recent call last):
>   File "/home/sglass/bin/binman", line 23, in <module>
>     srctree = os.environ['srctree']
>   File "/usr/lib/python3.8/os.py", line 675, in __getitem__
>     raise KeyError(key) from None
> KeyError: 'srctree'

I see, you mean that you run it when it's not in build tree?

> >> For example, most binman development is done just by
> >> running 'binman test' in the source tre. So perhaps default to the
> >> current directory is 'srcdir' is not set?

Ah, you mean you run it manually and not via `make`.

	os.environ.get('srctree', '')

should help I suppose.

P.S. What is the 'srcdir' you are referring to all the time?

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v2 1/2] binman: Do not pollute source tree when build with `make O=...`

[Andy Shevchenko]

On Fri, Dec 03, 2021 at 10:29:10PM +0200, Andy Shevchenko wrote:
> On Fri, Dec 03, 2021 at 01:13:12PM -0700, Simon Glass wrote:

...

> Ah, you mean you run it manually and not via `make`.
> 
> 	os.environ.get('srctree', '')
> 
> should help I suppose.

I haven't heard back from you, I suppose that the above solves the issue,
I'm about to send v3.

> P.S. What is the 'srcdir' you are referring to all the time?

And please clarify, maybe I missed something, what is the 'srcdir'?
Should I use it instead of 'srctree' or what?

-- 
With Best Regards,
Andy Shevchenko