[PATCH 0/4] scripts: add basic python version library and use it

[PATCH 0/4] scripts: add basic python version library and use it

[Luis R. Rodriguez]

This series depends on the last set of coccicheck enhancements which
add parmap support and indexing heuristics.

Coccinelle SmPL files may often require advanced grammar techniques not
available in older versions of Coccinelle. We can use --parse-cocci and
bail if a file does not parse, however this doesn't tell us anything about
requirements. An alternative approach is to specifically do a check for
a version of coccinelle.

Since checking for versions can be generic we can just write a python
library to do these sorts of checks for us, this allows us to provide
a generic kernel library for version checks for any binary. This adds
that and then makes use of it as an example within a coccinelle SmPL
file which require a later version of coccinelle.

Luis R. Rodriguez (4):
  coccicheck: propagate error and stop processing after first error
  scripts: add reqs python library
  coccicheck: enable use of the kernel's python library
  scripts/coccinelle: require coccinelle >= 1.0.4 on
    device_node_continue.cocci

 MAINTAINERS                                        |   1 +
 scripts/coccicheck                                 |   7 +
 .../iterators/device_node_continue.cocci           |  13 ++
 scripts/lib/__init__.py                            |   1 +
 scripts/lib/reqs.py                                | 211 +++++++++++++++++++++
 5 files changed, 233 insertions(+)
 create mode 100644 scripts/lib/__init__.py
 create mode 100644 scripts/lib/reqs.py

-- 
2.8.2

[PATCH 1/4] coccicheck: propagate error and stop processing after first error

[Luis R. Rodriguez]

Propagate back in the shell Coccinelle's error. Also stop
processing if an error has occured. This lets us handle some
errors in coccinelle cocci files and if they bail out we should
inspect the errors. This will be more useful later to help annotate
coccinelle version dependency requirements.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 37db7ac1dfae..ba7301ab0a3d 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -215,6 +215,11 @@ run_cmd_parmap() {
 		echo "Running ($NPROC in parallel): $@"
 	fi
 	$@ 2> ${DIR}/coccicheck.$$.err
+	if [[ $? -ne 0 ]]; then
+		echo "coccicheck failed"
+		cat $${DIR}/coccicheck.$$.err
+		exit $?
+	fi
 }
 
 run_cmd_old() {
-- 
2.8.2

[PATCH 2/4] scripts: add reqs python library

[Luis R. Rodriguez]

This library can be used in other python scripts to require
specific binary version requirements. It will be used first
with coccinelle's python bindings to enable coccinelle SmPL
files to specify version requirements per cocci file if it
has any.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 MAINTAINERS             |   1 +
 scripts/lib/__init__.py |   1 +
 scripts/lib/reqs.py     | 211 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 213 insertions(+)
 create mode 100644 scripts/lib/__init__.py
 create mode 100644 scripts/lib/reqs.py

diff --git a/MAINTAINERS b/MAINTAINERS
index f83e19a2dd97..fdebbb513c1b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6521,6 +6521,7 @@ F:	scripts/Makefile.*
 F:	scripts/basic/
 F:	scripts/mk*
 F:	scripts/package/
+F:	scripts/lib/
 
 KERNEL JANITORS
 L:	kernel-janitors@vger.kernel.org
diff --git a/scripts/lib/__init__.py b/scripts/lib/__init__.py
new file mode 100644
index 000000000000..1bb8bf6d7fd4
--- /dev/null
+++ b/scripts/lib/__init__.py
@@ -0,0 +1 @@
+# empty
diff --git a/scripts/lib/reqs.py b/scripts/lib/reqs.py
new file mode 100644
index 000000000000..1325fd21a87a
--- /dev/null
+++ b/scripts/lib/reqs.py
@@ -0,0 +1,211 @@
+import subprocess, os, sys, re
+"""
+Often enough Python code can grow to depend on binaries
+on a system, you may also require only specific versions
+of these. This small library helps with this. It also has
+helpers for packages which we know to handle already.
+"""
+
+class ReqError(Exception):
+    pass
+class ExecutionError(ReqError):
+    def __init__(self, errcode):
+        self.error_code = errcode
+
+class Req:
+    "To be used for verifying binay package dependencies on Python code"
+    def __init__(self):
+        self.all_reqs_ok = True
+        self.debug = False
+    def enable_debug(self):
+        self.debug = True
+    def reqs_match(self):
+        if self.all_reqs_ok:
+            return True
+        sys.stdout.write("You have unfulfilled binary requirements\n")
+        return False
+    def req_missing(self, program):
+        self.all_reqs_ok = False
+        sys.stdout.write("You need to have installed: %s\n" % program)
+    def req_old_program(self, program, version_req):
+        self.all_reqs_ok = False
+        sys.stdout.write("You need to have installed: %s >= %s\n" % (program, version_req))
+    def which(self, program):
+        cmd = ['which', program]
+        process = subprocess.Popen(cmd,
+                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                                   close_fds=True, universal_newlines=True)
+        stdout = process.communicate()[0]
+        process.wait()
+        if process.returncode != 0:
+            raise ExecutionError(process.returncode)
+        return stdout
+    def req_exists(self, program):
+        cmd = ['which', program]
+        process = subprocess.Popen(cmd,
+                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                                   close_fds=True, universal_newlines=True)
+        stdout = process.communicate()[0]
+        process.wait()
+        if process.returncode == 0:
+            return True
+        return False
+    def req_get_prog_version(self, program, version_query, version_pos):
+        '''
+        Suppose you have a binary that outputs:
+        $ spatch --version
+        spatch version 1.0.0-rc21 with Python support and with PCRE support
+
+        Every program veries what it wants you to query it for a version string,
+        prog_version() is designed so that you pass what the program expects for
+        its version query, and the position you expect the version string to be
+        on using python list.
+        '''
+        cmd = [program, version_query]
+        process = subprocess.Popen(cmd,
+                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                                   close_fds=True, universal_newlines=True)
+        stdout = process.communicate()[0]
+        process.wait()
+        if process.returncode != 0:
+            raise ExecutionError(process.returncode)
+        if self.debug:
+            sys.stdout.write("Running '%s' got us this break down:\n%s\n" %
+                             (
+                             ' '.join(cmd),
+                             "\n".join(map(str, [[i, x] for i, x in enumerate(stdout.split())])),
+                             ))
+            sys.stdout.write("You are using for version: %s\n" % stdout.split()[version_pos])
+            sys.stdout.write("Specifically your idx, element: %s\n" % ([[i, x] for i, x in enumerate(stdout.split())][version_pos]))
+        return stdout.split()[version_pos]
+
+    MAX_RC = 25
+    def __compute_rel_weight(self, rel_specs):
+        weight = 0
+        extra = 0
+        sublevel = 0
+        relmod = 0
+
+        if self.debug:
+            sys.stdout.write("VERSION       = %s\n" % rel_specs['VERSION'])
+            sys.stdout.write("PATCHLEVEL    = %s\n" % rel_specs['PATCHLEVEL'])
+            sys.stdout.write("SUBLEVEL      = %s\n" % rel_specs['SUBLEVEL'])
+            sys.stdout.write("EXTRAVERSION  = %s\n" % rel_specs['EXTRAVERSION'])
+            sys.stdout.write("RELMOD_UPDATE = %s\n" % rel_specs['RELMOD_UPDATE'])
+
+        if rel_specs['EXTRAVERSION'] != '':
+            if ("." in rel_specs['EXTRAVERSION'] or
+                    "rc" in rel_specs['EXTRAVERSION']):
+                rc = rel_specs['EXTRAVERSION'].lstrip("-rc")
+                if (rc == ""):
+                    rc = 0
+                else:
+                    rc = int(rc) - (Req.MAX_RC + 1)
+                extra = int(rc)
+            else:
+                extra = int(rel_specs['EXTRAVERSION']) + 10
+
+        if rel_specs['SUBLEVEL'] != '':
+            sublevel = int(rel_specs['SUBLEVEL'].lstrip(".")) * 20
+        else:
+            sublevel = 5
+
+        if rel_specs['RELMOD_UPDATE'] != '':
+            mod = rel_specs['RELMOD_UPDATE']
+            if (mod == ""):
+                mod = 0
+            else:
+                mod = int(mod)
+            relmod = int(mod)
+
+        weight = (int(rel_specs['VERSION'])    << 32) + \
+                 (int(rel_specs['PATCHLEVEL']) << 16) + \
+                 (sublevel   		       << 8 ) + \
+                 (extra * 60) + (relmod * 2)
+
+        return weight
+    def req_get_rel_spec(self, rel):
+        if "rc" in rel:
+            m = re.match(r"v*(?P<VERSION>\d+)\.+"
+                         "(?P<PATCHLEVEL>\d+)[.]*"
+                         "(?P<SUBLEVEL>\d*)"
+                         "(?P<EXTRAVERSION>[-rc]+\w*)\-*"
+                         "(?P<RELMOD_UPDATE>\d*)[-]*",
+                         rel)
+        else:
+            m = re.match(r"v*(?P<VERSION>\d+)\.+"
+                         "(?P<PATCHLEVEL>\d+)[.]*"
+                         "(?P<SUBLEVEL>\d*)[.]*"
+                         "(?P<EXTRAVERSION>\w*)\-*"
+                         "(?P<RELMOD_UPDATE>\d*)[-]*",
+                         rel)
+        if not m:
+            return m
+        rel_specs = m.groupdict()
+        return rel_specs
+    def compute_rel_weight(self, rel):
+        rel_specs = self.req_get_rel_spec(rel)
+        if not rel_specs:
+            return 0
+        return self.__compute_rel_weight(rel_specs)
+    def linux_version_cmp(self, version_req, version):
+        '''
+        If the program follows the linux version style scheme you can
+        use this to compare versions.
+        '''
+        weight_has = self.compute_rel_weight(version)
+        weight_req = self.compute_rel_weight(version_req)
+
+        if self.debug:
+            sys.stdout.write("You have program weight: %s\n" % weight_has)
+            sys.stdout.write("Required program weight: %s\n" % weight_req)
+
+        if weight_has < weight_req:
+            return -1
+        return 0
+    def require_version(self, program, version_query, version_req, version_pos, version_cmp):
+        '''
+        If you have a program version requirement you can specify it here,
+        as for the other flags refer to prog_version.
+        '''
+        if not self.require(program):
+            return False
+        version = self.req_get_prog_version(program, version_query, version_pos)
+        if self.debug:
+            sys.stdout.write("Checking release specs and weight: for: %s\n" % program)
+            sys.stdout.write("You have version: %s\n" % version)
+            sys.stdout.write("Required version: %s\n" % version_req)
+        if version_cmp(version_req, version) != 0:
+            self.req_old_program(program, version_req)
+            return False
+        return True
+    def require(self, program):
+        if self.req_exists(program):
+            return True
+        self.req_missing(program)
+        return False
+    def require_hint(self, program, package_hint):
+        if self.require(program):
+            return True
+        sys.stdout.write("Try installing the package: %s\n" % package_hint)
+        return False
+    def coccinelle(self, version):
+        if self.require_version('spatch', '--version', version, 2, self.linux_version_cmp):
+            return True
+        sys.stdout.write("Try installing the package: coccinelle\n")
+        sys.stdout.write("If that is too old go grab the code from source:\n\n")
+        sys.stdout.write("git clone https://github.com/coccinelle/coccinelle.git\n\n")
+        sys.stdout.write("To build you will need: ocaml ncurses-devel\n\n")
+        sys.stdout.write("If on SUSE / OpenSUSE you will also need: ocaml-ocamldoc\n\n")
+        return False
+    def kup(self):
+        if self.require('kup'):
+            return True
+        sys.stdout.write("Try installing the package: kup\n")
+        sys.stdout.write("If your distribution lacks that go get from source:\n\n")
+        sys.stdout.write("git clone git://git.kernel.org/pub/scm/utils/kup/kup.git\n\n")
+        return False
+    def make(self, version):
+        return self.require_version('make', '--version', version, 2, self.linux_version_cmp)
+    def gcc(self, version):
+        return self.require_version('gcc', '--version', version, 3, self.linux_version_cmp)
-- 
2.8.2

[PATCH 3/4] coccicheck: enable use of the kernel's python library

[Luis R. Rodriguez]

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index ba7301ab0a3d..a4d91d649ad9 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,6 +12,8 @@
 
 DIR=$(dirname $(readlink -f $0))
 DIR="${DIR}/../"
+export PYTHONPATH=${DIR}/scripts/
+
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
-- 
2.8.2

[PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Luis R. Rodriguez]

Make use of the new kernel python requirements library to be able to
specify coccinelle binary version requirements. The cocci file
device_node_continue.cocci requires at least coccinelle 1.0.4.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccinelle/iterators/device_node_continue.cocci | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/coccinelle/iterators/device_node_continue.cocci b/scripts/coccinelle/iterators/device_node_continue.cocci
index 38ab744a4037..b590de9418d1 100644
--- a/scripts/coccinelle/iterators/device_node_continue.cocci
+++ b/scripts/coccinelle/iterators/device_node_continue.cocci
@@ -12,6 +12,19 @@ virtual context
 virtual org
 virtual report
 
+// This uses a conjunction, which requires at least coccinelle >= 1.0.4
+@script:python@
+@@
+
+import sys
+from lib import reqs
+
+req = reqs.Req()
+req.coccinelle('1.0.4')
+if not req.reqs_match():
+    cocci.exit()
+    sys.exit(1)
+
 @r exists@
 expression e1,e2;
 local idexpression n;
-- 
2.8.2

Re: [PATCH 2/4] scripts: add reqs python library

[Julia Lawall]

On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:

> This library can be used in other python scripts to require
> specific binary version requirements. It will be used first
> with coccinelle's python bindings to enable coccinelle SmPL
> files to specify version requirements per cocci file if it
> has any.
> 
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  MAINTAINERS             |   1 +
>  scripts/lib/__init__.py |   1 +
>  scripts/lib/reqs.py     | 211 ++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 213 insertions(+)
>  create mode 100644 scripts/lib/__init__.py
>  create mode 100644 scripts/lib/reqs.py
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index f83e19a2dd97..fdebbb513c1b 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6521,6 +6521,7 @@ F:	scripts/Makefile.*
>  F:	scripts/basic/
>  F:	scripts/mk*
>  F:	scripts/package/
> +F:	scripts/lib/
>  
>  KERNEL JANITORS
>  L:	kernel-janitors@vger.kernel.org
> diff --git a/scripts/lib/__init__.py b/scripts/lib/__init__.py
> new file mode 100644
> index 000000000000..1bb8bf6d7fd4
> --- /dev/null
> +++ b/scripts/lib/__init__.py
> @@ -0,0 +1 @@
> +# empty
> diff --git a/scripts/lib/reqs.py b/scripts/lib/reqs.py
> new file mode 100644
> index 000000000000..1325fd21a87a
> --- /dev/null
> +++ b/scripts/lib/reqs.py
> @@ -0,0 +1,211 @@
> +import subprocess, os, sys, re
> +"""
> +Often enough Python code can grow to depend on binaries
> +on a system, you may also require only specific versions
> +of these. This small library helps with this. It also has
> +helpers for packages which we know to handle already.
> +"""
> +
> +class ReqError(Exception):
> +    pass
> +class ExecutionError(ReqError):
> +    def __init__(self, errcode):
> +        self.error_code = errcode
> +
> +class Req:
> +    "To be used for verifying binay package dependencies on Python code"

binay -> binary

> +    def __init__(self):
> +        self.all_reqs_ok = True
> +        self.debug = False
> +    def enable_debug(self):
> +        self.debug = True
> +    def reqs_match(self):
> +        if self.all_reqs_ok:
> +            return True
> +        sys.stdout.write("You have unfulfilled binary requirements\n")
> +        return False
> +    def req_missing(self, program):
> +        self.all_reqs_ok = False
> +        sys.stdout.write("You need to have installed: %s\n" % program)
> +    def req_old_program(self, program, version_req):
> +        self.all_reqs_ok = False
> +        sys.stdout.write("You need to have installed: %s >= %s\n" % (program, version_req))
> +    def which(self, program):
> +        cmd = ['which', program]
> +        process = subprocess.Popen(cmd,
> +                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
> +                                   close_fds=True, universal_newlines=True)
> +        stdout = process.communicate()[0]
> +        process.wait()
> +        if process.returncode != 0:
> +            raise ExecutionError(process.returncode)
> +        return stdout
> +    def req_exists(self, program):
> +        cmd = ['which', program]
> +        process = subprocess.Popen(cmd,
> +                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
> +                                   close_fds=True, universal_newlines=True)
> +        stdout = process.communicate()[0]
> +        process.wait()
> +        if process.returncode == 0:
> +            return True
> +        return False
> +    def req_get_prog_version(self, program, version_query, version_pos):
> +        '''
> +        Suppose you have a binary that outputs:
> +        $ spatch --version
> +        spatch version 1.0.0-rc21 with Python support and with PCRE support
> +
> +        Every program veries what it wants you to query it for a version string,
> +        prog_version() is designed so that you pass what the program expects for
> +        its version query, and the position you expect the version string to be
> +        on using python list.
> +        '''
> +        cmd = [program, version_query]
> +        process = subprocess.Popen(cmd,
> +                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
> +                                   close_fds=True, universal_newlines=True)
> +        stdout = process.communicate()[0]
> +        process.wait()
> +        if process.returncode != 0:
> +            raise ExecutionError(process.returncode)
> +        if self.debug:
> +            sys.stdout.write("Running '%s' got us this break down:\n%s\n" %
> +                             (
> +                             ' '.join(cmd),
> +                             "\n".join(map(str, [[i, x] for i, x in enumerate(stdout.split())])),
> +                             ))
> +            sys.stdout.write("You are using for version: %s\n" % stdout.split()[version_pos])
> +            sys.stdout.write("Specifically your idx, element: %s\n" % ([[i, x] for i, x in enumerate(stdout.split())][version_pos]))
> +        return stdout.split()[version_pos]
> +
> +    MAX_RC = 25
> +    def __compute_rel_weight(self, rel_specs):
> +        weight = 0
> +        extra = 0
> +        sublevel = 0
> +        relmod = 0
> +
> +        if self.debug:
> +            sys.stdout.write("VERSION       = %s\n" % rel_specs['VERSION'])
> +            sys.stdout.write("PATCHLEVEL    = %s\n" % rel_specs['PATCHLEVEL'])
> +            sys.stdout.write("SUBLEVEL      = %s\n" % rel_specs['SUBLEVEL'])
> +            sys.stdout.write("EXTRAVERSION  = %s\n" % rel_specs['EXTRAVERSION'])
> +            sys.stdout.write("RELMOD_UPDATE = %s\n" % rel_specs['RELMOD_UPDATE'])
> +
> +        if rel_specs['EXTRAVERSION'] != '':
> +            if ("." in rel_specs['EXTRAVERSION'] or
> +                    "rc" in rel_specs['EXTRAVERSION']):
> +                rc = rel_specs['EXTRAVERSION'].lstrip("-rc")
> +                if (rc == ""):
> +                    rc = 0
> +                else:
> +                    rc = int(rc) - (Req.MAX_RC + 1)
> +                extra = int(rc)
> +            else:
> +                extra = int(rel_specs['EXTRAVERSION']) + 10
> +
> +        if rel_specs['SUBLEVEL'] != '':
> +            sublevel = int(rel_specs['SUBLEVEL'].lstrip(".")) * 20
> +        else:
> +            sublevel = 5
> +
> +        if rel_specs['RELMOD_UPDATE'] != '':
> +            mod = rel_specs['RELMOD_UPDATE']
> +            if (mod == ""):
> +                mod = 0
> +            else:
> +                mod = int(mod)
> +            relmod = int(mod)
> +
> +        weight = (int(rel_specs['VERSION'])    << 32) + \
> +                 (int(rel_specs['PATCHLEVEL']) << 16) + \
> +                 (sublevel   		       << 8 ) + \
> +                 (extra * 60) + (relmod * 2)
> +
> +        return weight
> +    def req_get_rel_spec(self, rel):
> +        if "rc" in rel:
> +            m = re.match(r"v*(?P<VERSION>\d+)\.+"
> +                         "(?P<PATCHLEVEL>\d+)[.]*"
> +                         "(?P<SUBLEVEL>\d*)"
> +                         "(?P<EXTRAVERSION>[-rc]+\w*)\-*"
> +                         "(?P<RELMOD_UPDATE>\d*)[-]*",
> +                         rel)
> +        else:
> +            m = re.match(r"v*(?P<VERSION>\d+)\.+"
> +                         "(?P<PATCHLEVEL>\d+)[.]*"
> +                         "(?P<SUBLEVEL>\d*)[.]*"
> +                         "(?P<EXTRAVERSION>\w*)\-*"
> +                         "(?P<RELMOD_UPDATE>\d*)[-]*",
> +                         rel)
> +        if not m:
> +            return m
> +        rel_specs = m.groupdict()
> +        return rel_specs
> +    def compute_rel_weight(self, rel):
> +        rel_specs = self.req_get_rel_spec(rel)
> +        if not rel_specs:
> +            return 0
> +        return self.__compute_rel_weight(rel_specs)
> +    def linux_version_cmp(self, version_req, version):
> +        '''
> +        If the program follows the linux version style scheme you can
> +        use this to compare versions.
> +        '''
> +        weight_has = self.compute_rel_weight(version)
> +        weight_req = self.compute_rel_weight(version_req)
> +
> +        if self.debug:
> +            sys.stdout.write("You have program weight: %s\n" % weight_has)
> +            sys.stdout.write("Required program weight: %s\n" % weight_req)
> +
> +        if weight_has < weight_req:
> +            return -1
> +        return 0
> +    def require_version(self, program, version_query, version_req, version_pos, version_cmp):
> +        '''
> +        If you have a program version requirement you can specify it here,
> +        as for the other flags refer to prog_version.
> +        '''
> +        if not self.require(program):
> +            return False
> +        version = self.req_get_prog_version(program, version_query, version_pos)
> +        if self.debug:
> +            sys.stdout.write("Checking release specs and weight: for: %s\n" % program)
> +            sys.stdout.write("You have version: %s\n" % version)
> +            sys.stdout.write("Required version: %s\n" % version_req)
> +        if version_cmp(version_req, version) != 0:
> +            self.req_old_program(program, version_req)
> +            return False
> +        return True
> +    def require(self, program):
> +        if self.req_exists(program):
> +            return True
> +        self.req_missing(program)
> +        return False
> +    def require_hint(self, program, package_hint):
> +        if self.require(program):
> +            return True
> +        sys.stdout.write("Try installing the package: %s\n" % package_hint)
> +        return False
> +    def coccinelle(self, version):
> +        if self.require_version('spatch', '--version', version, 2, self.linux_version_cmp):
> +            return True
> +        sys.stdout.write("Try installing the package: coccinelle\n")
> +        sys.stdout.write("If that is too old go grab the code from source:\n\n")
> +        sys.stdout.write("git clone https://github.com/coccinelle/coccinelle.git\n\n")
> +        sys.stdout.write("To build you will need: ocaml ncurses-devel\n\n")
> +        sys.stdout.write("If on SUSE / OpenSUSE you will also need: ocaml-ocamldoc\n\n")
> +        return False
> +    def kup(self):
> +        if self.require('kup'):
> +            return True
> +        sys.stdout.write("Try installing the package: kup\n")
> +        sys.stdout.write("If your distribution lacks that go get from source:\n\n")
> +        sys.stdout.write("git clone git://git.kernel.org/pub/scm/utils/kup/kup.git\n\n")
> +        return False
> +    def make(self, version):
> +        return self.require_version('make', '--version', version, 2, self.linux_version_cmp)
> +    def gcc(self, version):
> +        return self.require_version('gcc', '--version', version, 3, self.linux_version_cmp)
> -- 
> 2.8.2
> 
> 

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Julia Lawall]

On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:

> Make use of the new kernel python requirements library to be able to
> specify coccinelle binary version requirements. The cocci file
> device_node_continue.cocci requires at least coccinelle 1.0.4.
> 
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  scripts/coccinelle/iterators/device_node_continue.cocci | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/scripts/coccinelle/iterators/device_node_continue.cocci b/scripts/coccinelle/iterators/device_node_continue.cocci
> index 38ab744a4037..b590de9418d1 100644
> --- a/scripts/coccinelle/iterators/device_node_continue.cocci
> +++ b/scripts/coccinelle/iterators/device_node_continue.cocci
> @@ -12,6 +12,19 @@ virtual context
>  virtual org
>  virtual report
>  
> +// This uses a conjunction, which requires at least coccinelle >= 1.0.4
> +@script:python@
> +@@
> +
> +import sys
> +from lib import reqs
> +
> +req = reqs.Req()
> +req.coccinelle('1.0.4')
> +if not req.reqs_match():
> +    cocci.exit()
> +    sys.exit(1)

This doesn't look very appealing to me.  Shouldn't Coccinelle handle this 
itself?

julia

> +
>  @r exists@
>  expression e1,e2;
>  local idexpression n;
> -- 
> 2.8.2
> 
> 

Re: [PATCH 2/4] scripts: add reqs python library

[Michal Marek]

On 2016-06-15 00:10, Luis R. Rodriguez wrote:
> +        weight = (int(rel_specs['VERSION'])    << 32) + \
> +                 (int(rel_specs['PATCHLEVEL']) << 16) + \
> +                 (sublevel   		       << 8 ) + \
> +                 (extra * 60) + (relmod * 2)

This is going to silently break as soon as we have a version number with
e.g. a time stamp embedded. And there is actually no need to convert the
version string to an integer. You can convert them to arrays of
components and compare the components one by one.

Michal

Re: [PATCH 3/4] coccicheck: enable use of the kernel's python library

[Michal Marek]

On 2016-06-15 00:10, Luis R. Rodriguez wrote:
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  scripts/coccicheck | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index ba7301ab0a3d..a4d91d649ad9 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -12,6 +12,8 @@
>  
>  DIR=$(dirname $(readlink -f $0))
>  DIR="${DIR}/../"
> +export PYTHONPATH=${DIR}/scripts/

The first assignment to DIR already sets the desired path.

Michal

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Julia Lawall]

How about the following, since Coccinelle knows what its version is?
This could of course be implemented in python as well.

julia

diff --git a/docs/Coccilib.3cocci b/docs/Coccilib.3cocci
index 0e4fbb8..ca5b061 100644
--- a/docs/Coccilib.3cocci
+++ b/docs/Coccilib.3cocci
@@ -232,6 +232,15 @@ is the empty list if spatch is not currently working on any file (eg,
 in an initialize or finalize rule).
 .sp

+.I val cocci_version
+:
+.B unit -> string
+.sp
+Returns the a string indicating the current version.  Note that if
+Coccinelle has been modified since a release, the version number will be
+postfixed with "-dirty".
+.sp
+
 .I val print_main
 :
 .B ?color:string -> string -> pos list -> unit
diff --git a/ocaml/coccilib.ml b/ocaml/coccilib.ml
index f60c6b2..2f352d8 100644
--- a/ocaml/coccilib.ml
+++ b/ocaml/coccilib.ml
@@ -168,6 +168,8 @@ let dir () = !Flag.dir

 let files () = !Flag.currentfiles

+let cocci_version () = Config.version
+
 (* ---------------------------------------------------------------------- *)
 (* org mode *)

Aw: [Cocci] [PATCH 2/4] scripts: add reqs python library

[SF Markus Elfring]

> +class Req:

Will a longer identifier like "requirement" be more useful than the suggested abbreviation?


> +    "To be used for verifying binay package dependencies on Python code"

Would you like to fix a typo here?

... binary ...


> +    def req_old_program(self, program, version_req):

Does such a name selection need also further considerations?

Regards,
Markus

Re: [PATCH 3/4] coccicheck: enable use of the kernel's python library

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 09:51:31AM +0200, Michal Marek wrote:
> On 2016-06-15 00:10, Luis R. Rodriguez wrote:
> > Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> > ---
> >  scripts/coccicheck | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/scripts/coccicheck b/scripts/coccicheck
> > index ba7301ab0a3d..a4d91d649ad9 100755
> > --- a/scripts/coccicheck
> > +++ b/scripts/coccicheck
> > @@ -12,6 +12,8 @@
> >  
> >  DIR=$(dirname $(readlink -f $0))
> >  DIR="${DIR}/../"
> > +export PYTHONPATH=${DIR}/scripts/
> 
> The first assignment to DIR already sets the desired path.

Ah yes, thanks. Fixed.

  Luis

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 08:08:41AM +0200, Julia Lawall wrote:
> 
> 
> On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> 
> > Make use of the new kernel python requirements library to be able to
> > specify coccinelle binary version requirements. The cocci file
> > device_node_continue.cocci requires at least coccinelle 1.0.4.
> > 
> > Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> > ---
> >  scripts/coccinelle/iterators/device_node_continue.cocci | 13 +++++++++++++
> >  1 file changed, 13 insertions(+)
> > 
> > diff --git a/scripts/coccinelle/iterators/device_node_continue.cocci b/scripts/coccinelle/iterators/device_node_continue.cocci
> > index 38ab744a4037..b590de9418d1 100644
> > --- a/scripts/coccinelle/iterators/device_node_continue.cocci
> > +++ b/scripts/coccinelle/iterators/device_node_continue.cocci
> > @@ -12,6 +12,19 @@ virtual context
> >  virtual org
> >  virtual report
> >  
> > +// This uses a conjunction, which requires at least coccinelle >= 1.0.4
> > +@script:python@
> > +@@
> > +
> > +import sys
> > +from lib import reqs
> > +
> > +req = reqs.Req()
> > +req.coccinelle('1.0.4')
> > +if not req.reqs_match():
> > +    cocci.exit()
> > +    sys.exit(1)
> 
> This doesn't look very appealing to me.  Shouldn't Coccinelle handle this 
> itself?

Oh I agree, however what options do we have at the moment instead of an odd
parse error complaint ? This lets one annotate version requirements and is
backward compatible.

  Luis

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 10:43:30AM +0200, Julia Lawall wrote:
> How about the following, since Coccinelle knows what its version is?
> This could of course be implemented in python as well.
> 
> julia
> 
> diff --git a/docs/Coccilib.3cocci b/docs/Coccilib.3cocci
> index 0e4fbb8..ca5b061 100644
> --- a/docs/Coccilib.3cocci
> +++ b/docs/Coccilib.3cocci
> @@ -232,6 +232,15 @@ is the empty list if spatch is not currently working on any file (eg,
>  in an initialize or finalize rule).
>  .sp
> 
> +.I val cocci_version
> +:
> +.B unit -> string
> +.sp
> +Returns the a string indicating the current version.  Note that if
> +Coccinelle has been modified since a release, the version number will be
> +postfixed with "-dirty".
> +.sp
> +
>  .I val print_main
>  :
>  .B ?color:string -> string -> pos list -> unit
> diff --git a/ocaml/coccilib.ml b/ocaml/coccilib.ml
> index f60c6b2..2f352d8 100644
> --- a/ocaml/coccilib.ml
> +++ b/ocaml/coccilib.ml
> @@ -168,6 +168,8 @@ let dir () = !Flag.dir
> 
>  let files () = !Flag.currentfiles
> 
> +let cocci_version () = Config.version
> +
>  (* ---------------------------------------------------------------------- *)
>  (* org mode *)
> 
> 

Anything to *only* get the version instead of a long list is nice, right now
spatch --version spits out:

spatch version 1.0.5 compiled with OCaml version 4.02.3
Flags passed to the configure script: [none]
Python scripting support: yes
Syntax of regular expresssions: PCRE

The Python library just parses the 3rd item at the top so it can extract
the version. But surely if spatch --version-only was available we'd use
that instead a well.

Other than this though how can we require coccinelle version checks per
SmPL file cleanly and also what should we do to make it backward compatible
with older versions of coccinelle?

  Luis

Re: [Cocci] [PATCH 2/4] scripts: add reqs python library

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 02:01:37PM +0200, SF Markus Elfring wrote:
> > +    "To be used for verifying binay package dependencies on Python code"
> 
> Would you like to fix a typo here?
> 
> ... binary ...

Fixed.

  Luis

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Julia Lawall]

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Wed, Jun 15, 2016 at 10:43:30AM +0200, Julia Lawall wrote:
> > How about the following, since Coccinelle knows what its version is?
> > This could of course be implemented in python as well.
> >
> > julia
> >
> > diff --git a/docs/Coccilib.3cocci b/docs/Coccilib.3cocci
> > index 0e4fbb8..ca5b061 100644
> > --- a/docs/Coccilib.3cocci
> > +++ b/docs/Coccilib.3cocci
> > @@ -232,6 +232,15 @@ is the empty list if spatch is not currently working on any file (eg,
> >  in an initialize or finalize rule).
> >  .sp
> >
> > +.I val cocci_version
> > +:
> > +.B unit -> string
> > +.sp
> > +Returns the a string indicating the current version.  Note that if
> > +Coccinelle has been modified since a release, the version number will be
> > +postfixed with "-dirty".
> > +.sp
> > +
> >  .I val print_main
> >  :
> >  .B ?color:string -> string -> pos list -> unit
> > diff --git a/ocaml/coccilib.ml b/ocaml/coccilib.ml
> > index f60c6b2..2f352d8 100644
> > --- a/ocaml/coccilib.ml
> > +++ b/ocaml/coccilib.ml
> > @@ -168,6 +168,8 @@ let dir () = !Flag.dir
> >
> >  let files () = !Flag.currentfiles
> >
> > +let cocci_version () = Config.version
> > +
> >  (* ---------------------------------------------------------------------- *)
> >  (* org mode *)
> >
> >
>
> Anything to *only* get the version instead of a long list is nice, right now
> spatch --version spits out:
>
> spatch version 1.0.5 compiled with OCaml version 4.02.3
> Flags passed to the configure script: [none]
> Python scripting support: yes
> Syntax of regular expresssions: PCRE
>
> The Python library just parses the 3rd item at the top so it can extract
> the version. But surely if spatch --version-only was available we'd use
> that instead a well.
>
> Other than this though how can we require coccinelle version checks per
> SmPL file cleanly and also what should we do to make it backward compatible
> with older versions of coccinelle?

I'm not sure that being backward compatible with older versions of
Coccinelle is worth adding new libraries to the Linux kernel, and adding
unpleasant python code to semantic patches.

The above ocaml code just produces eg 1.0.5 or 1.0.5-dirty.  I could drop
the -dirty at the coccilib level, if that seems desirable.

julia

Re: [PATCH 2/4] scripts: add reqs python library

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 09:50:11AM +0200, Michal Marek wrote:
> On 2016-06-15 00:10, Luis R. Rodriguez wrote:
> > +        weight = (int(rel_specs['VERSION'])    << 32) + \
> > +                 (int(rel_specs['PATCHLEVEL']) << 16) + \
> > +                 (sublevel   		       << 8 ) + \
> > +                 (extra * 60) + (relmod * 2)
> 
> This is going to silently break as soon as we have a version number with
> e.g. a time stamp embedded. 

Well this is adhering to a linux_version_cmp type, surely we can adjust
it with alternatives. It just happens that with the common stuff this
suffices.

Do you have a specific string in mind I can use to test against?

> And there is actually no need to convert the
> version string to an integer. You can convert them to arrays of
> components and compare the components one by one.

You can, however weight is used to as a generic formula to determine
in one shot if you have a release >= than what is required. Its inspired
by how KERNEL_VERSION() is used:

#define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))

  Luis

Re: [PATCH 2/4] scripts: add reqs python library

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 08:06:33AM +0200, Julia Lawall wrote:
> 
> 
> On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> 
> > This library can be used in other python scripts to require
> > specific binary version requirements. It will be used first
> > with coccinelle's python bindings to enable coccinelle SmPL
> > files to specify version requirements per cocci file if it
> > has any.
> > 
> > Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> > ---
> >  MAINTAINERS             |   1 +
> >  scripts/lib/__init__.py |   1 +
> >  scripts/lib/reqs.py     | 211 ++++++++++++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 213 insertions(+)
> >  create mode 100644 scripts/lib/__init__.py
> >  create mode 100644 scripts/lib/reqs.py
> > 
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index f83e19a2dd97..fdebbb513c1b 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -6521,6 +6521,7 @@ F:	scripts/Makefile.*
> >  F:	scripts/basic/
> >  F:	scripts/mk*
> >  F:	scripts/package/
> > +F:	scripts/lib/
> >  
> >  KERNEL JANITORS
> >  L:	kernel-janitors@vger.kernel.org
> > diff --git a/scripts/lib/__init__.py b/scripts/lib/__init__.py
> > new file mode 100644
> > index 000000000000..1bb8bf6d7fd4
> > --- /dev/null
> > +++ b/scripts/lib/__init__.py
> > @@ -0,0 +1 @@
> > +# empty
> > diff --git a/scripts/lib/reqs.py b/scripts/lib/reqs.py
> > new file mode 100644
> > index 000000000000..1325fd21a87a
> > --- /dev/null
> > +++ b/scripts/lib/reqs.py
> > @@ -0,0 +1,211 @@
> > +import subprocess, os, sys, re
> > +"""
> > +Often enough Python code can grow to depend on binaries
> > +on a system, you may also require only specific versions
> > +of these. This small library helps with this. It also has
> > +helpers for packages which we know to handle already.
> > +"""
> > +
> > +class ReqError(Exception):
> > +    pass
> > +class ExecutionError(ReqError):
> > +    def __init__(self, errcode):
> > +        self.error_code = errcode
> > +
> > +class Req:
> > +    "To be used for verifying binay package dependencies on Python code"
> 
> binay -> binary

Fixed, thanks.

  Luis

Re: scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[SF Markus Elfring]

> The above ocaml code just produces eg 1.0.5 or 1.0.5-dirty.  I could drop
> the -dirty at the coccilib level, if that seems desirable.

Are there risks that any more fine-grained checks will be needed for
special functionality?

Regards,
Markus

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 05:55:34PM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Wed, Jun 15, 2016 at 10:43:30AM +0200, Julia Lawall wrote:
> > > How about the following, since Coccinelle knows what its version is?
> > > This could of course be implemented in python as well.
> > >
> > > julia
> > >
> > > diff --git a/docs/Coccilib.3cocci b/docs/Coccilib.3cocci
> > > index 0e4fbb8..ca5b061 100644
> > > --- a/docs/Coccilib.3cocci
> > > +++ b/docs/Coccilib.3cocci
> > > @@ -232,6 +232,15 @@ is the empty list if spatch is not currently working on any file (eg,
> > >  in an initialize or finalize rule).
> > >  .sp
> > >
> > > +.I val cocci_version
> > > +:
> > > +.B unit -> string
> > > +.sp
> > > +Returns the a string indicating the current version.  Note that if
> > > +Coccinelle has been modified since a release, the version number will be
> > > +postfixed with "-dirty".
> > > +.sp
> > > +
> > >  .I val print_main
> > >  :
> > >  .B ?color:string -> string -> pos list -> unit
> > > diff --git a/ocaml/coccilib.ml b/ocaml/coccilib.ml
> > > index f60c6b2..2f352d8 100644
> > > --- a/ocaml/coccilib.ml
> > > +++ b/ocaml/coccilib.ml
> > > @@ -168,6 +168,8 @@ let dir () = !Flag.dir
> > >
> > >  let files () = !Flag.currentfiles
> > >
> > > +let cocci_version () = Config.version
> > > +
> > >  (* ---------------------------------------------------------------------- *)
> > >  (* org mode *)
> > >
> > >
> >
> > Anything to *only* get the version instead of a long list is nice, right now
> > spatch --version spits out:
> >
> > spatch version 1.0.5 compiled with OCaml version 4.02.3
> > Flags passed to the configure script: [none]
> > Python scripting support: yes
> > Syntax of regular expresssions: PCRE
> >
> > The Python library just parses the 3rd item at the top so it can extract
> > the version. But surely if spatch --version-only was available we'd use
> > that instead a well.
> >
> > Other than this though how can we require coccinelle version checks per
> > SmPL file cleanly and also what should we do to make it backward compatible
> > with older versions of coccinelle?
> 
> I'm not sure that being backward compatible with older versions of
> Coccinelle is worth adding new libraries to the Linux kernel, and adding
> unpleasant python code to semantic patches.

True. I'm more than happy to not have to add this crap.

> The above ocaml code just produces eg 1.0.5 or 1.0.5-dirty.  I could drop
> the -dirty at the coccilib level, if that seems desirable.

This is when spatch --cocci_version is passed ?

Its still unclear how we can require in a clean way coccinelle version
requirements in SmPL patches with this. Can you clarify?

If we embrace this or assume we'll get this in the next release we'll have
to just bump the kernel's coccinelle requirement recommendation, which I think
is far due anyway.

  Luis

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Julia Lawall]

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Wed, Jun 15, 2016 at 05:55:34PM +0200, Julia Lawall wrote:
> >
> >
> > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > On Wed, Jun 15, 2016 at 10:43:30AM +0200, Julia Lawall wrote:
> > > > How about the following, since Coccinelle knows what its version is?
> > > > This could of course be implemented in python as well.
> > > >
> > > > julia
> > > >
> > > > diff --git a/docs/Coccilib.3cocci b/docs/Coccilib.3cocci
> > > > index 0e4fbb8..ca5b061 100644
> > > > --- a/docs/Coccilib.3cocci
> > > > +++ b/docs/Coccilib.3cocci
> > > > @@ -232,6 +232,15 @@ is the empty list if spatch is not currently working on any file (eg,
> > > >  in an initialize or finalize rule).
> > > >  .sp
> > > >
> > > > +.I val cocci_version
> > > > +:
> > > > +.B unit -> string
> > > > +.sp
> > > > +Returns the a string indicating the current version.  Note that if
> > > > +Coccinelle has been modified since a release, the version number will be
> > > > +postfixed with "-dirty".
> > > > +.sp
> > > > +
> > > >  .I val print_main
> > > >  :
> > > >  .B ?color:string -> string -> pos list -> unit
> > > > diff --git a/ocaml/coccilib.ml b/ocaml/coccilib.ml
> > > > index f60c6b2..2f352d8 100644
> > > > --- a/ocaml/coccilib.ml
> > > > +++ b/ocaml/coccilib.ml
> > > > @@ -168,6 +168,8 @@ let dir () = !Flag.dir
> > > >
> > > >  let files () = !Flag.currentfiles
> > > >
> > > > +let cocci_version () = Config.version
> > > > +
> > > >  (* ---------------------------------------------------------------------- *)
> > > >  (* org mode *)
> > > >
> > > >
> > >
> > > Anything to *only* get the version instead of a long list is nice, right now
> > > spatch --version spits out:
> > >
> > > spatch version 1.0.5 compiled with OCaml version 4.02.3
> > > Flags passed to the configure script: [none]
> > > Python scripting support: yes
> > > Syntax of regular expresssions: PCRE
> > >
> > > The Python library just parses the 3rd item at the top so it can extract
> > > the version. But surely if spatch --version-only was available we'd use
> > > that instead a well.
> > >
> > > Other than this though how can we require coccinelle version checks per
> > > SmPL file cleanly and also what should we do to make it backward compatible
> > > with older versions of coccinelle?
> >
> > I'm not sure that being backward compatible with older versions of
> > Coccinelle is worth adding new libraries to the Linux kernel, and adding
> > unpleasant python code to semantic patches.
>
> True. I'm more than happy to not have to add this crap.
>
> > The above ocaml code just produces eg 1.0.5 or 1.0.5-dirty.  I could drop
> > the -dirty at the coccilib level, if that seems desirable.
>
> This is when spatch --cocci_version is passed ?

Perhaps it wasn't clear enough from the above nroff and ocaml code.  I
added a function Coccilib.version() that returns eg either 1.0.5 or
1.0.5-dirty.  Such a function could be implemented for python as well.

>
> Its still unclear how we can require in a clean way coccinelle version
> requirements in SmPL patches with this. Can you clarify?

Test the string that it returns and exit.  Like you are doing, but no need
for adding new libraries to the kernel.

> If we embrace this or assume we'll get this in the next release we'll have
> to just bump the kernel's coccinelle requirement recommendation, which I think
> is far due anyway.

Yes.

julia

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 06:11:57PM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Wed, Jun 15, 2016 at 05:55:34PM +0200, Julia Lawall wrote:
> > >
> > >
> > > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> > >
> > > > On Wed, Jun 15, 2016 at 10:43:30AM +0200, Julia Lawall wrote:
> > > > > How about the following, since Coccinelle knows what its version is?
> > > > > This could of course be implemented in python as well.
> > > > >
> > > > > julia
> > > > >
> > > > > diff --git a/docs/Coccilib.3cocci b/docs/Coccilib.3cocci
> > > > > index 0e4fbb8..ca5b061 100644
> > > > > --- a/docs/Coccilib.3cocci
> > > > > +++ b/docs/Coccilib.3cocci
> > > > > @@ -232,6 +232,15 @@ is the empty list if spatch is not currently working on any file (eg,
> > > > >  in an initialize or finalize rule).
> > > > >  .sp
> > > > >
> > > > > +.I val cocci_version
> > > > > +:
> > > > > +.B unit -> string
> > > > > +.sp
> > > > > +Returns the a string indicating the current version.  Note that if
> > > > > +Coccinelle has been modified since a release, the version number will be
> > > > > +postfixed with "-dirty".
> > > > > +.sp
> > > > > +
> > > > >  .I val print_main
> > > > >  :
> > > > >  .B ?color:string -> string -> pos list -> unit
> > > > > diff --git a/ocaml/coccilib.ml b/ocaml/coccilib.ml
> > > > > index f60c6b2..2f352d8 100644
> > > > > --- a/ocaml/coccilib.ml
> > > > > +++ b/ocaml/coccilib.ml
> > > > > @@ -168,6 +168,8 @@ let dir () = !Flag.dir
> > > > >
> > > > >  let files () = !Flag.currentfiles
> > > > >
> > > > > +let cocci_version () = Config.version
> > > > > +
> > > > >  (* ---------------------------------------------------------------------- *)
> > > > >  (* org mode *)
> > > > >
> > > > >
> > > >
> > > > Anything to *only* get the version instead of a long list is nice, right now
> > > > spatch --version spits out:
> > > >
> > > > spatch version 1.0.5 compiled with OCaml version 4.02.3
> > > > Flags passed to the configure script: [none]
> > > > Python scripting support: yes
> > > > Syntax of regular expresssions: PCRE
> > > >
> > > > The Python library just parses the 3rd item at the top so it can extract
> > > > the version. But surely if spatch --version-only was available we'd use
> > > > that instead a well.
> > > >
> > > > Other than this though how can we require coccinelle version checks per
> > > > SmPL file cleanly and also what should we do to make it backward compatible
> > > > with older versions of coccinelle?
> > >
> > > I'm not sure that being backward compatible with older versions of
> > > Coccinelle is worth adding new libraries to the Linux kernel, and adding
> > > unpleasant python code to semantic patches.
> >
> > True. I'm more than happy to not have to add this crap.
> >
> > > The above ocaml code just produces eg 1.0.5 or 1.0.5-dirty.  I could drop
> > > the -dirty at the coccilib level, if that seems desirable.
> >
> > This is when spatch --cocci_version is passed ?
> 
> Perhaps it wasn't clear enough from the above nroff and ocaml code.  I
> added a function Coccilib.version() that returns eg either 1.0.5 or
> 1.0.5-dirty.  Such a function could be implemented for python as well.
> 
> >
> > Its still unclear how we can require in a clean way coccinelle version
> > requirements in SmPL patches with this. Can you clarify?
> 
> Test the string that it returns and exit.  Like you are doing, but no need
> for adding new libraries to the kernel.

Ah then that's indeed welcome, however another function would be best too:

Coccilib.version_reqs() which lets us say what the requirement is and it
would return true or false, false when the req is not met.

> > If we embrace this or assume we'll get this in the next release we'll have
> > to just bump the kernel's coccinelle requirement recommendation, which I think
> > is far due anyway.
> 
> Yes.

Great. Then I'm all for dropping this python jüjü crap.

  Luis

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Julia Lawall]

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Wed, Jun 15, 2016 at 06:11:57PM +0200, Julia Lawall wrote:
> >
> >
> > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > On Wed, Jun 15, 2016 at 05:55:34PM +0200, Julia Lawall wrote:
> > > >
> > > >
> > > > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> > > >
> > > > > On Wed, Jun 15, 2016 at 10:43:30AM +0200, Julia Lawall wrote:
> > > > > > How about the following, since Coccinelle knows what its version is?
> > > > > > This could of course be implemented in python as well.
> > > > > >
> > > > > > julia
> > > > > >
> > > > > > diff --git a/docs/Coccilib.3cocci b/docs/Coccilib.3cocci
> > > > > > index 0e4fbb8..ca5b061 100644
> > > > > > --- a/docs/Coccilib.3cocci
> > > > > > +++ b/docs/Coccilib.3cocci
> > > > > > @@ -232,6 +232,15 @@ is the empty list if spatch is not currently working on any file (eg,
> > > > > >  in an initialize or finalize rule).
> > > > > >  .sp
> > > > > >
> > > > > > +.I val cocci_version
> > > > > > +:
> > > > > > +.B unit -> string
> > > > > > +.sp
> > > > > > +Returns the a string indicating the current version.  Note that if
> > > > > > +Coccinelle has been modified since a release, the version number will be
> > > > > > +postfixed with "-dirty".
> > > > > > +.sp
> > > > > > +
> > > > > >  .I val print_main
> > > > > >  :
> > > > > >  .B ?color:string -> string -> pos list -> unit
> > > > > > diff --git a/ocaml/coccilib.ml b/ocaml/coccilib.ml
> > > > > > index f60c6b2..2f352d8 100644
> > > > > > --- a/ocaml/coccilib.ml
> > > > > > +++ b/ocaml/coccilib.ml
> > > > > > @@ -168,6 +168,8 @@ let dir () = !Flag.dir
> > > > > >
> > > > > >  let files () = !Flag.currentfiles
> > > > > >
> > > > > > +let cocci_version () = Config.version
> > > > > > +
> > > > > >  (* ---------------------------------------------------------------------- *)
> > > > > >  (* org mode *)
> > > > > >
> > > > > >
> > > > >
> > > > > Anything to *only* get the version instead of a long list is nice, right now
> > > > > spatch --version spits out:
> > > > >
> > > > > spatch version 1.0.5 compiled with OCaml version 4.02.3
> > > > > Flags passed to the configure script: [none]
> > > > > Python scripting support: yes
> > > > > Syntax of regular expresssions: PCRE
> > > > >
> > > > > The Python library just parses the 3rd item at the top so it can extract
> > > > > the version. But surely if spatch --version-only was available we'd use
> > > > > that instead a well.
> > > > >
> > > > > Other than this though how can we require coccinelle version checks per
> > > > > SmPL file cleanly and also what should we do to make it backward compatible
> > > > > with older versions of coccinelle?
> > > >
> > > > I'm not sure that being backward compatible with older versions of
> > > > Coccinelle is worth adding new libraries to the Linux kernel, and adding
> > > > unpleasant python code to semantic patches.
> > >
> > > True. I'm more than happy to not have to add this crap.
> > >
> > > > The above ocaml code just produces eg 1.0.5 or 1.0.5-dirty.  I could drop
> > > > the -dirty at the coccilib level, if that seems desirable.
> > >
> > > This is when spatch --cocci_version is passed ?
> >
> > Perhaps it wasn't clear enough from the above nroff and ocaml code.  I
> > added a function Coccilib.version() that returns eg either 1.0.5 or
> > 1.0.5-dirty.  Such a function could be implemented for python as well.
> >
> > >
> > > Its still unclear how we can require in a clean way coccinelle version
> > > requirements in SmPL patches with this. Can you clarify?
> >
> > Test the string that it returns and exit.  Like you are doing, but no need
> > for adding new libraries to the kernel.
>
> Ah then that's indeed welcome, however another function would be best too:
>
> Coccilib.version_reqs() which lets us say what the requirement is and it
> would return true or false, false when the req is not met.

I'm not so fond of this.  It seems like a very specific use case.

I really think this should be managed by coccicheck, in the same way as
the options.

julia

Re: [PATCH 4/4] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 06:52:27PM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Wed, Jun 15, 2016 at 06:11:57PM +0200, Julia Lawall wrote:
> > >
> > >
> > > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> > >
> > > > On Wed, Jun 15, 2016 at 05:55:34PM +0200, Julia Lawall wrote:
> > > > >
> > > > >
> > > > > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> > > > >
> > > > > > On Wed, Jun 15, 2016 at 10:43:30AM +0200, Julia Lawall wrote:
> > > > > > > How about the following, since Coccinelle knows what its version is?
> > > > > > > This could of course be implemented in python as well.
> > > > > > >
> > > > > > > julia
> > > > > > >
> > > > > > > diff --git a/docs/Coccilib.3cocci b/docs/Coccilib.3cocci
> > > > > > > index 0e4fbb8..ca5b061 100644
> > > > > > > --- a/docs/Coccilib.3cocci
> > > > > > > +++ b/docs/Coccilib.3cocci
> > > > > > > @@ -232,6 +232,15 @@ is the empty list if spatch is not currently working on any file (eg,
> > > > > > >  in an initialize or finalize rule).
> > > > > > >  .sp
> > > > > > >
> > > > > > > +.I val cocci_version
> > > > > > > +:
> > > > > > > +.B unit -> string
> > > > > > > +.sp
> > > > > > > +Returns the a string indicating the current version.  Note that if
> > > > > > > +Coccinelle has been modified since a release, the version number will be
> > > > > > > +postfixed with "-dirty".
> > > > > > > +.sp
> > > > > > > +
> > > > > > >  .I val print_main
> > > > > > >  :
> > > > > > >  .B ?color:string -> string -> pos list -> unit
> > > > > > > diff --git a/ocaml/coccilib.ml b/ocaml/coccilib.ml
> > > > > > > index f60c6b2..2f352d8 100644
> > > > > > > --- a/ocaml/coccilib.ml
> > > > > > > +++ b/ocaml/coccilib.ml
> > > > > > > @@ -168,6 +168,8 @@ let dir () = !Flag.dir
> > > > > > >
> > > > > > >  let files () = !Flag.currentfiles
> > > > > > >
> > > > > > > +let cocci_version () = Config.version
> > > > > > > +
> > > > > > >  (* ---------------------------------------------------------------------- *)
> > > > > > >  (* org mode *)
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > Anything to *only* get the version instead of a long list is nice, right now
> > > > > > spatch --version spits out:
> > > > > >
> > > > > > spatch version 1.0.5 compiled with OCaml version 4.02.3
> > > > > > Flags passed to the configure script: [none]
> > > > > > Python scripting support: yes
> > > > > > Syntax of regular expresssions: PCRE
> > > > > >
> > > > > > The Python library just parses the 3rd item at the top so it can extract
> > > > > > the version. But surely if spatch --version-only was available we'd use
> > > > > > that instead a well.
> > > > > >
> > > > > > Other than this though how can we require coccinelle version checks per
> > > > > > SmPL file cleanly and also what should we do to make it backward compatible
> > > > > > with older versions of coccinelle?
> > > > >
> > > > > I'm not sure that being backward compatible with older versions of
> > > > > Coccinelle is worth adding new libraries to the Linux kernel, and adding
> > > > > unpleasant python code to semantic patches.
> > > >
> > > > True. I'm more than happy to not have to add this crap.
> > > >
> > > > > The above ocaml code just produces eg 1.0.5 or 1.0.5-dirty.  I could drop
> > > > > the -dirty at the coccilib level, if that seems desirable.
> > > >
> > > > This is when spatch --cocci_version is passed ?
> > >
> > > Perhaps it wasn't clear enough from the above nroff and ocaml code.  I
> > > added a function Coccilib.version() that returns eg either 1.0.5 or
> > > 1.0.5-dirty.  Such a function could be implemented for python as well.
> > >
> > > >
> > > > Its still unclear how we can require in a clean way coccinelle version
> > > > requirements in SmPL patches with this. Can you clarify?
> > >
> > > Test the string that it returns and exit.  Like you are doing, but no need
> > > for adding new libraries to the kernel.
> >
> > Ah then that's indeed welcome, however another function would be best too:
> >
> > Coccilib.version_reqs() which lets us say what the requirement is and it
> > would return true or false, false when the req is not met.
> 
> I'm not so fond of this.  It seems like a very specific use case.

Perhaps.

> I really think this should be managed by coccicheck, in the same way as
> the options.

OK if its up to coccicheck -- we'll need a solution there. I'd prefer to use a
generic library like reqs there and just have the check for the maximum
requirement there.

Michal do you have any preference ?

FWIW I had originally written the reqs library for for rel-html [0] which
enables arbitrary projects to make a shiny HTLM5 release project based on a
naked release page provided, the inferring of release needs some generic
heuristics on release matching (in the future the simpler approach is
to have a git tree have two PGP signatures, one for signed releases, and
another for deprecating release, then an alternative smarter heuristic
would be to only look for signed tags of currently supported releases),
the Linux kernel's strategy on versioning computation seemed the way to go.
I was surprised no standard library supported it properly. This library is
also used on backports not only for coccinell requirements but for other
generic binary requirements.

We can bash it out.. however this seems like it could be generally useful
for other tools we have in the kernel.

[0] https://git.kernel.org/cgit/linux/kernel/git/mcgrof/rel-html.git/

  Luis

Re: [PATCH 2/4] scripts: add reqs python library

[Michal Marek]

Dne 15.6.2016 v 18:02 Luis R. Rodriguez napsal(a):
> On Wed, Jun 15, 2016 at 09:50:11AM +0200, Michal Marek wrote:
>> On 2016-06-15 00:10, Luis R. Rodriguez wrote:
>>> +        weight = (int(rel_specs['VERSION'])    << 32) + \
>>> +                 (int(rel_specs['PATCHLEVEL']) << 16) + \
>>> +                 (sublevel   		       << 8 ) + \
>>> +                 (extra * 60) + (relmod * 2)
>>
>> This is going to silently break as soon as we have a version number with
>> e.g. a time stamp embedded. 
> 
> Well this is adhering to a linux_version_cmp type, surely we can adjust
> it with alternatives. It just happens that with the common stuff this
> suffices.
> 
> Do you have a specific string in mind I can use to test against?

You can have a look at the git history of scripts/ld-version.sh.


>> And there is actually no need to convert the
>> version string to an integer. You can convert them to arrays of
>> components and compare the components one by one.
> 
> You can, however weight is used to as a generic formula to determine
> in one shot if you have a release >= than what is required. Its inspired
> by how KERNEL_VERSION() is used:
> 
> #define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))

Yes, the kernel interface provides comparisons using the preprocessor
arithmetic, so it needs to convert the version strings to integers. But
you are providing methods to compare the versions and what happens in
these methods stays in these methods :).

Michal

Re: [PATCH 2/4] scripts: add reqs python library

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 09:11:45PM +0200, Michal Marek wrote:
> Dne 15.6.2016 v 18:02 Luis R. Rodriguez napsal(a):
> > On Wed, Jun 15, 2016 at 09:50:11AM +0200, Michal Marek wrote:
> >> On 2016-06-15 00:10, Luis R. Rodriguez wrote:
> >>> +        weight = (int(rel_specs['VERSION'])    << 32) + \
> >>> +                 (int(rel_specs['PATCHLEVEL']) << 16) + \
> >>> +                 (sublevel   		       << 8 ) + \
> >>> +                 (extra * 60) + (relmod * 2)
> >>
> >> This is going to silently break as soon as we have a version number with
> >> e.g. a time stamp embedded. 
> > 
> > Well this is adhering to a linux_version_cmp type, surely we can adjust
> > it with alternatives. It just happens that with the common stuff this
> > suffices.
> > 
> > Do you have a specific string in mind I can use to test against?
> 
> You can have a look at the git history of scripts/ld-version.sh.

Will use that instead. The SmPL patch can then have:

// Requires: 1.0.5

This ignores the -dirty stuff.

  Luis

Re: [PATCH 2/4] scripts: add reqs python library

[Julia Lawall]

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Wed, Jun 15, 2016 at 09:11:45PM +0200, Michal Marek wrote:
> > Dne 15.6.2016 v 18:02 Luis R. Rodriguez napsal(a):
> > > On Wed, Jun 15, 2016 at 09:50:11AM +0200, Michal Marek wrote:
> > >> On 2016-06-15 00:10, Luis R. Rodriguez wrote:
> > >>> +        weight = (int(rel_specs['VERSION'])    << 32) + \
> > >>> +                 (int(rel_specs['PATCHLEVEL']) << 16) + \
> > >>> +                 (sublevel   		       << 8 ) + \
> > >>> +                 (extra * 60) + (relmod * 2)
> > >>
> > >> This is going to silently break as soon as we have a version number with
> > >> e.g. a time stamp embedded. 
> > > 
> > > Well this is adhering to a linux_version_cmp type, surely we can adjust
> > > it with alternatives. It just happens that with the common stuff this
> > > suffices.
> > > 
> > > Do you have a specific string in mind I can use to test against?
> > 
> > You can have a look at the git history of scripts/ld-version.sh.
> 
> Will use that instead. The SmPL patch can then have:
> 
> // Requires: 1.0.5

This would be ideal, I think.

julia