* [Xen-devel] [PATCH v2 1/3] xen/flask: Drop the gen-policy.py script
2019-12-07 18:18 [Xen-devel] [PATCH for-4.13 v2 0/3] xen: Build fixes related to Python3 Andrew Cooper
@ 2019-12-07 18:18 ` Andrew Cooper
2019-12-07 18:18 ` [Xen-devel] [PATCH v2 2/3] xen/banner: Drop the fig-to-oct.py script Andrew Cooper
2019-12-07 18:18 ` [Xen-devel] [PATCH v2 3/3] xen/build: Automatically locate a suitable python interpreter Andrew Cooper
2 siblings, 0 replies; 4+ messages in thread
From: Andrew Cooper @ 2019-12-07 18:18 UTC (permalink / raw)
To: Xen-devel; +Cc: Juergen Gross, Andrew Cooper, Daniel De Graaf
The script is Python 2 specific, and fails with string/binary issues with
Python 3:
Traceback (most recent call last):
File "gen-policy.py", line 14, in <module>
for char in sys.stdin.read():
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 0: invalid start byte
Fixing the script to be compatible isn't hard, but using python here is
wasteful. Drop the script entirely, and write a short flask-policy.S instead.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Daniel De Graaf <dgdegra@tycho.nsa.gov>
CC: Juergen Gross <jgross@suse.com>
v2:
* Fix tabs vs spaces issues
For 4.13. This is a blocker to our intent to by Py3-clean in this release.
Discovered entirely accidently when testing the final patch.
---
xen/xsm/flask/Makefile | 6 ++----
xen/xsm/flask/flask-policy.S | 20 ++++++++++++++++++++
xen/xsm/flask/gen-policy.py | 23 -----------------------
3 files changed, 22 insertions(+), 27 deletions(-)
create mode 100644 xen/xsm/flask/flask-policy.S
delete mode 100644 xen/xsm/flask/gen-policy.py
diff --git a/xen/xsm/flask/Makefile b/xen/xsm/flask/Makefile
index f5ffab1226..7c3f381287 100644
--- a/xen/xsm/flask/Makefile
+++ b/xen/xsm/flask/Makefile
@@ -27,7 +27,8 @@ $(FLASK_H_FILES): $(FLASK_H_DEPEND)
$(AV_H_FILES): $(AV_H_DEPEND)
$(CONFIG_SHELL) policy/mkaccess_vector.sh $(AWK) $(AV_H_DEPEND)
-obj-$(CONFIG_XSM_FLASK_POLICY) += policy.o
+obj-bin-$(CONFIG_XSM_FLASK_POLICY) += flask-policy.o
+flask-policy.o: policy.bin
FLASK_BUILD_DIR := $(CURDIR)
POLICY_SRC := $(FLASK_BUILD_DIR)/xenpolicy-$(XEN_FULLVERSION)
@@ -36,9 +37,6 @@ policy.bin: FORCE
$(MAKE) -f $(XEN_ROOT)/tools/flask/policy/Makefile.common -C $(XEN_ROOT)/tools/flask/policy FLASK_BUILD_DIR=$(FLASK_BUILD_DIR)
cmp -s $(POLICY_SRC) $@ || cp $(POLICY_SRC) $@
-policy.c: policy.bin gen-policy.py
- $(PYTHON) gen-policy.py < $< > $@
-
.PHONY: clean
clean::
rm -f $(ALL_H_FILES) *.o $(DEPS_RM) policy.* $(POLICY_SRC)
diff --git a/xen/xsm/flask/flask-policy.S b/xen/xsm/flask/flask-policy.S
new file mode 100644
index 0000000000..b63a14851d
--- /dev/null
+++ b/xen/xsm/flask/flask-policy.S
@@ -0,0 +1,20 @@
+ .section .init.rodata, "a", @progbits
+
+/* const unsigned char xsm_flask_init_policy[] __initconst */
+ .align 4
+ .global xsm_flask_init_policy
+xsm_flask_init_policy:
+ .incbin "policy.bin"
+.Lend:
+
+ .type xsm_flask_init_policy, @object
+ .size xsm_flask_init_policy, . - xsm_flask_init_policy
+
+/* const unsigned int __initconst xsm_flask_init_policy_size */
+ .align 4
+ .global xsm_flask_init_policy_size
+xsm_flask_init_policy_size:
+ .long .Lend - xsm_flask_init_policy
+
+ .type xsm_flask_init_policy_size, @object
+ .size xsm_flask_init_policy_size, . - xsm_flask_init_policy_size
diff --git a/xen/xsm/flask/gen-policy.py b/xen/xsm/flask/gen-policy.py
deleted file mode 100644
index c7501e4614..0000000000
--- a/xen/xsm/flask/gen-policy.py
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/usr/bin/env python
-import sys
-
-policy_size = 0
-
-sys.stdout.write("""
-/* This file is autogenerated by gen_policy.py */
-#include <xen/init.h>
-#include <xsm/xsm.h>
-
-const unsigned char xsm_flask_init_policy[] __initconst = {
-""")
-
-for char in sys.stdin.read():
- sys.stdout.write(" 0x%02x," % ord(char))
- policy_size = policy_size + 1
- if policy_size % 13 == 0:
- sys.stdout.write("\n")
-
-sys.stdout.write("""
-};
-const unsigned int __initconst xsm_flask_init_policy_size = %d;
-""" % policy_size)
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [Xen-devel] [PATCH v2 2/3] xen/banner: Drop the fig-to-oct.py script
2019-12-07 18:18 [Xen-devel] [PATCH for-4.13 v2 0/3] xen: Build fixes related to Python3 Andrew Cooper
2019-12-07 18:18 ` [Xen-devel] [PATCH v2 1/3] xen/flask: Drop the gen-policy.py script Andrew Cooper
@ 2019-12-07 18:18 ` Andrew Cooper
2019-12-07 18:18 ` [Xen-devel] [PATCH v2 3/3] xen/build: Automatically locate a suitable python interpreter Andrew Cooper
2 siblings, 0 replies; 4+ messages in thread
From: Andrew Cooper @ 2019-12-07 18:18 UTC (permalink / raw)
To: Xen-devel
Cc: Juergen Gross, Stefano Stabellini, Julien Grall, Wei Liu,
Konrad Rzeszutek Wilk, George Dunlap, Andrew Cooper, Jan Beulich,
Ian Jackson
The script is 664 rather than 775, so the banner conversion doesn't actually
work if $(PYTHON) is empty:
/bin/sh: tools/fig-to-oct.py: Permission denied
make[3]: *** [include/xen/compile.h] Error 126
make[3]: Leaving directory `/builds/xen-project/people/andyhhp/xen/xen'
Fixing this is easy, but using python here is wasteful. compile.h doesn't
need XEN_BANNER rendering in octal, and text is much more simple to handle.
Replace fig-to-oct.py with a sed oneliner.
While changing this logic, take the opportunity to optimise the banner
space (and time on the serial port) by dropping trailing whitespace, which is
84 characters for current staging.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: George Dunlap <George.Dunlap@eu.citrix.com>
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Wei Liu <wl@xen.org>
CC: Julien Grall <julien@xen.org>
CC: Juergen Gross <jgross@suse.com>
v2:
* New
Spotted by Gitlab CI, caused by `which` not being present in some of the
CentOS containers. While this is more of a container bug than anything else,
it does highlight that the build ought to cope.
---
xen/Makefile | 5 ++++-
xen/tools/fig-to-oct.py | 18 ------------------
2 files changed, 4 insertions(+), 19 deletions(-)
delete mode 100644 xen/tools/fig-to-oct.py
diff --git a/xen/Makefile b/xen/Makefile
index 99701e3165..13ae1b6011 100644
--- a/xen/Makefile
+++ b/xen/Makefile
@@ -163,6 +163,9 @@ delete-unfresh-files:
@mv -f $@.tmp $@
# compile.h contains dynamic build info. Rebuilt on every 'make' invocation.
+#
+# For .banner sed-ary, strip trailing whitespace, escape backslashes, and wrap
+# each line in '"...\n" \' to become a valid C string
include/xen/compile.h: include/xen/compile.h.in .banner
@sed -e 's/@@date@@/$(XEN_BUILD_DATE)/g' \
-e 's/@@time@@/$(XEN_BUILD_TIME)/g' \
@@ -176,7 +179,7 @@ include/xen/compile.h: include/xen/compile.h.in .banner
-e 's!@@changeset@@!$(shell tools/scmversion $(XEN_ROOT) || echo "unavailable")!g' \
< include/xen/compile.h.in > $@.new
@cat .banner
- @$(PYTHON) tools/fig-to-oct.py < .banner >> $@.new
+ @sed -e 's_[ ]*$$__' -e 's_\\_\\\\_g' -e 's_\(.*\)_"\1\\n" \\_' < .banner >> $@.new
@mv -f $@.new $@
include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
diff --git a/xen/tools/fig-to-oct.py b/xen/tools/fig-to-oct.py
deleted file mode 100644
index db4fd32159..0000000000
--- a/xen/tools/fig-to-oct.py
+++ /dev/null
@@ -1,18 +0,0 @@
-#!/usr/bin/env python
-import sys
-
-chars_per_line = 18
-chars_so_far = 0
-
-sys.stdout.write('"')
-
-for char in sys.stdin.read():
-
- sys.stdout.write("\\%03o" % ord(char))
- chars_so_far = chars_so_far + 1
-
- if chars_so_far == chars_per_line:
- chars_so_far = 0
- sys.stdout.write('" \\\n"')
-
-sys.stdout.write('"\n')
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [Xen-devel] [PATCH v2 3/3] xen/build: Automatically locate a suitable python interpreter
2019-12-07 18:18 [Xen-devel] [PATCH for-4.13 v2 0/3] xen: Build fixes related to Python3 Andrew Cooper
2019-12-07 18:18 ` [Xen-devel] [PATCH v2 1/3] xen/flask: Drop the gen-policy.py script Andrew Cooper
2019-12-07 18:18 ` [Xen-devel] [PATCH v2 2/3] xen/banner: Drop the fig-to-oct.py script Andrew Cooper
@ 2019-12-07 18:18 ` Andrew Cooper
2 siblings, 0 replies; 4+ messages in thread
From: Andrew Cooper @ 2019-12-07 18:18 UTC (permalink / raw)
To: Xen-devel
Cc: Juergen Gross, Stefano Stabellini, Julien Grall, Wei Liu,
Konrad Rzeszutek Wilk, George Dunlap, Andrew Cooper,
Steven Haigh, Jan Beulich, Ian Jackson
Needing to pass PYTHON=python3 into hypervisor builds is irritating and
unnecessary. Locate a suitable interpreter automatically, defaulting to Py3
if it is available.
Reported-by: Steven Haigh <netwiz@crc.id.au>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: George Dunlap <George.Dunlap@eu.citrix.com>
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Wei Liu <wl@xen.org>
CC: Julien Grall <julien@xen.org>
CC: Steven Haigh <netwiz@crc.id.au>
CC: Juergen Gross <jgross@suse.com>
v2:
* Cope with `which` not being present in the system.
* Only evaulate the shell command one, rather than once per $(PTHON) usage
For 4.13. This is a very-nice-to-have WRT our Py3-clean intention.
---
xen/Makefile | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/xen/Makefile b/xen/Makefile
index 13ae1b6011..4f02aca3ac 100644
--- a/xen/Makefile
+++ b/xen/Makefile
@@ -13,6 +13,11 @@ export XEN_BUILD_TIME ?= $(shell LC_ALL=C date +%T)
export XEN_BUILD_HOST ?= $(shell hostname)
export XEN_CONFIG_EXPERT ?= n
+# Best effort attempt to find a python interpreter, defaulting to Python 3 if
+# available. Fall back to just `python` if `which` is nowhere to be found.
+PYTHON_INTERPRETER := $(word 1,$(shell which python3 python python2 2>/dev/null) python)
+export PYTHON ?= $(PYTHON_INTERPRETER)
+
export BASEDIR := $(CURDIR)
export XEN_ROOT := $(BASEDIR)/..
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 4+ messages in thread