* [PATCH 1/5] Added the PR service.
2011-05-19 10:29 [PATCH 0/5] network based PR service Lianhao Lu
@ 2011-05-19 10:29 ` Lianhao Lu
2011-05-19 10:29 ` [PATCH 2/5] conf/bitbake.conf: Added variables for " Lianhao Lu
` (6 subsequent siblings)
7 siblings, 0 replies; 22+ messages in thread
From: Lianhao Lu @ 2011-05-19 10:29 UTC (permalink / raw)
To: openembedded-core
From: Lianhao Lu <lianhao.lu@intel.com>
Added the initial implementation of the server side PR service.
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
bitbake/bin/bitbake-prserv | 53 +++++++++++
bitbake/lib/prserv/__init__.py | 11 ++
bitbake/lib/prserv/db.py | 100 ++++++++++++++++++++
bitbake/lib/prserv/serv.py | 198 ++++++++++++++++++++++++++++++++++++++++
4 files changed, 362 insertions(+), 0 deletions(-)
create mode 100755 bitbake/bin/bitbake-prserv
create mode 100644 bitbake/lib/prserv/__init__.py
create mode 100644 bitbake/lib/prserv/db.py
create mode 100644 bitbake/lib/prserv/serv.py
diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv
new file mode 100755
index 0000000..14073ca
--- /dev/null
+++ b/bitbake/bin/bitbake-prserv
@@ -0,0 +1,53 @@
+#!/usr/bin/env python
+import os
+import sys,logging
+import optparse
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),'lib'))
+
+import prserv
+import prserv.serv
+
+__version__="1.0.0"
+
+PRHOST_DEFAULT=''
+PRPORT_DEFAULT=8585
+
+def main():
+ parser = optparse.OptionParser(
+ version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__),
+ usage = "%prog [options]")
+
+ parser.add_option("-f", "--file", help="database filename(default prserv.db)", action="store",
+ dest="dbfile", type="string", default="prserv.db")
+ parser.add_option("-l", "--log", help="log filename(default prserv.log)", action="store",
+ dest="logfile", type="string", default="prserv.log")
+ parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG",
+ action = "store", type="string", dest="loglevel", default = "WARNING")
+ parser.add_option("--start", help="start daemon",
+ action="store_true", dest="start", default="True")
+ parser.add_option("--stop", help="stop daemon",
+ action="store_false", dest="start")
+ parser.add_option("--host", help="ip address to bind", action="store",
+ dest="host", type="string", default=PRHOST_DEFAULT)
+ parser.add_option("--port", help="port number(default 8585)", action="store",
+ dest="port", type="int", default=PRPORT_DEFAULT)
+
+ options, args = parser.parse_args(sys.argv)
+
+ prserv.init_logger(os.path.abspath(options.logfile),options.loglevel)
+
+ if options.start:
+ prserv.serv.start_daemon(options)
+ else:
+ prserv.serv.stop_daemon()
+
+if __name__ == "__main__":
+ try:
+ ret = main()
+ except Exception:
+ ret = 1
+ import traceback
+ traceback.print_exc(5)
+ sys.exit(ret)
+
diff --git a/bitbake/lib/prserv/__init__.py b/bitbake/lib/prserv/__init__.py
new file mode 100644
index 0000000..2837e13
--- /dev/null
+++ b/bitbake/lib/prserv/__init__.py
@@ -0,0 +1,11 @@
+__version__ = "1.0.0"
+
+import os, time
+import sys,logging
+
+def init_logger(logfile, loglevel):
+ numeric_level = getattr(logging, loglevel.upper(), None)
+ if not isinstance(numeric_level, int):
+ raise ValueError('Invalid log level: %s' % loglevel)
+ logging.basicConfig(level=numeric_level, filename=logfile)
+
diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
new file mode 100644
index 0000000..bbee931
--- /dev/null
+++ b/bitbake/lib/prserv/db.py
@@ -0,0 +1,100 @@
+import logging
+import os.path
+import errno
+import sys
+import warnings
+import sqlite3
+
+try:
+ import sqlite3
+except ImportError:
+ from pysqlite2 import dbapi2 as sqlite3
+
+sqlversion = sqlite3.sqlite_version_info
+if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
+ raise Exception("sqlite3 version 3.3.0 or later is required.")
+
+class NotFoundError(StandardError):
+ pass
+
+class PRTable():
+ def __init__(self,cursor,table):
+ self.cursor = cursor
+ self.table = table
+
+ #create the table
+ self._execute("CREATE TABLE IF NOT EXISTS %s \
+ (version TEXT NOT NULL, \
+ checksum TEXT NOT NULL, \
+ value INTEGER, \
+ PRIMARY KEY (version,checksum));"
+ % table)
+
+ def _execute(self, *query):
+ """Execute a query, waiting to acquire a lock if necessary"""
+ count = 0
+ while True:
+ try:
+ return self.cursor.execute(*query)
+ except sqlite3.OperationalError as exc:
+ if 'database is locked' in str(exc) and count < 500:
+ count = count + 1
+ continue
+ raise
+ except sqlite3.IntegrityError as exc:
+ print "Integrity error %s" % str(exc)
+ break
+
+ def getValue(self, version, checksum):
+ data=self._execute("SELECT value FROM %s WHERE version=? AND checksum=?;" % self.table,
+ (version,checksum))
+ row=data.fetchone()
+ if row != None:
+ return row[0]
+ else:
+ #no value found, try to insert
+ self._execute("INSERT INTO %s VALUES (?, ?, (select ifnull(max(value)+1,0) from %s where version=?));"
+ % (self.table,self.table),
+ (version,checksum,version))
+ data=self._execute("SELECT value FROM %s WHERE version=? AND checksum=?;" % self.table,
+ (version,checksum))
+ row=data.fetchone()
+ if row != None:
+ return row[0]
+ else:
+ raise NotFoundError
+
+class PRData(object):
+ """Object representing the PR database"""
+ def __init__(self, filename):
+ self.filename=os.path.abspath(filename)
+ #build directory hierarchy
+ try:
+ os.makedirs(os.path.dirname(self.filename))
+ except OSError as e:
+ if e.errno != errno.EEXIST:
+ raise e
+ self.connection=sqlite3.connect(self.filename, timeout=5,
+ isolation_level=None)
+ self.cursor=self.connection.cursor()
+ self._tables={}
+
+ def __del__(self):
+ print "PRData: closing DB %s" % self.filename
+ self.connection.close()
+
+ def __getitem__(self,tblname):
+ if not isinstance(tblname, basestring):
+ raise TypeError("tblname argument must be a string, not '%s'" %
+ type(tblname))
+ if tblname in self._tables:
+ return self._tables[tblname]
+ else:
+ tableobj = self._tables[tblname] = PRTable(self.cursor, tblname)
+ return tableobj
+
+ def __delitem__(self, tblname):
+ if tblname in self._tables:
+ del self._tables[tblname]
+ logging.info("drop table %s" % (tblname))
+ self.cursor.execute("DROP TABLE IF EXISTS %s;" % tblname)
diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
new file mode 100644
index 0000000..ecafe4f
--- /dev/null
+++ b/bitbake/lib/prserv/serv.py
@@ -0,0 +1,198 @@
+import os,sys,logging
+import signal,time, atexit
+from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
+import xmlrpclib,sqlite3
+
+import bb.server.xmlrpc
+import prserv
+import prserv.db
+
+if sys.hexversion < 0x020600F0:
+ print("Sorry, python 2.6 or later is required.")
+ sys.exit(1)
+
+class Handler(SimpleXMLRPCRequestHandler):
+ def _dispatch(self,method,params):
+ try:
+ value=self.server.funcs[method](*params)
+ except:
+ import traceback
+ traceback.print_exc()
+ raise
+ return value
+
+class PRServer(SimpleXMLRPCServer):
+ pidfile="/tmp/PRServer.pid"
+ def __init__(self, dbfile, logfile, interface, daemon=True):
+ ''' constructor '''
+ SimpleXMLRPCServer.__init__(self, interface,
+ requestHandler=SimpleXMLRPCRequestHandler,
+ logRequests=False, allow_none=True)
+ self.dbfile=dbfile
+ self.daemon=daemon
+ self.logfile=logfile
+ self.host, self.port = self.socket.getsockname()
+ self.db=prserv.db.PRData(dbfile)
+ self.table=self.db["PRMAIN"]
+
+ self.register_function(self.getPR, "getPR")
+ self.register_function(self.quit, "quit")
+ self.register_function(self.ping, "ping")
+ self.register_introspection_functions()
+
+ def ping(self):
+ return not self.quit
+
+ def getPR(self, version, checksum):
+ try:
+ return self.table.getValue(version,checksum)
+ except prserv.NotFoundError:
+ logging.error("can not find value for (%s, %s)",version,checksum)
+ return None
+ except sqlite3.Error as exc:
+ logging.error(str(exc))
+ return None
+
+ def quit(self):
+ self.quit=True
+ return
+
+ def _serve_forever(self):
+ self.quit = False
+ self.timeout = 0.5
+ while not self.quit:
+ self.handle_request()
+
+ logging.info("PRServer: stopping...")
+ self.server_close()
+ return
+
+ def start(self):
+ if self.daemon is True:
+ logging.info("PRServer: starting daemon...")
+ self.daemonize()
+ else:
+ logging.info("PRServer: starting...")
+ self._serve_forever()
+
+ def delpid(self):
+ os.remove(PRServer.pidfile)
+
+ def daemonize(self):
+ """
+ See Advanced Programming in the UNIX, Sec 13.3
+ """
+ os.umask(0)
+
+ try:
+ pid = os.fork()
+ if pid > 0:
+ sys.exit(0)
+ except OSError,e:
+ sys.stderr.write("1st fork failed: %d %s\n" % (e.errno, e.strerror))
+ sys.exit(1)
+
+ os.setsid()
+ """
+ fork again to make sure the daemon is not session leader,
+ which prevents it from acquiring controlling terminal
+ """
+ try:
+ pid = os.fork()
+ if pid > 0: #parent
+ sys.exit(0)
+ except OSError,e:
+ sys.stderr.write("2nd fork failed: %d %s\n" % (e.errno, e.strerror))
+ sys.exit(1)
+
+ os.chdir("/")
+
+ sys.stdout.flush()
+ sys.stderr.flush()
+ si = file('/dev/null', 'r')
+ so = file(self.logfile, 'a+')
+ se = so
+ os.dup2(si.fileno(),sys.stdin.fileno())
+ os.dup2(so.fileno(),sys.stdout.fileno())
+ os.dup2(se.fileno(),sys.stderr.fileno())
+
+ # write pidfile
+ atexit.register(self.delpid)
+ pid = str(os.getpid())
+ pf = file(PRServer.pidfile, 'w+')
+ pf.write("%s\n" % pid)
+ pf.write("%s\n" % self.host)
+ pf.write("%s\n" % self.port)
+ pf.close()
+
+ self._serve_forever()
+
+class PRServerConnection():
+ def __init__(self, host, port):
+ self.connection = bb.server.xmlrpc._create_server(host, port)
+ self.host = host
+ self.port = port
+
+ def terminate(self):
+ # Don't wait for server indefinitely
+ import socket
+ socket.setdefaulttimeout(2)
+ try:
+ self.connection.quit()
+ except:
+ pass
+
+ def getPR(self, version, checksum):
+ return self.connection.getPR(version, checksum)
+
+ def ping(self):
+ return self.connection.ping()
+
+def start_daemon(options):
+ try:
+ pf = file(PRServer.pidfile,'r')
+ pid = int(pf.readline().strip())
+ pf.close()
+ except IOError:
+ pid = None
+
+ if pid:
+ sys.stderr.write("pidfile %s already exist. Daemon already running?\n"
+ % PRServer.pidfile)
+ sys.exit(1)
+
+ server = PRServer(options.dbfile, interface=(options.host, options.port),
+ logfile=os.path.abspath(options.logfile))
+ server.start()
+
+def stop_daemon():
+ try:
+ pf = file(PRServer.pidfile,'r')
+ pid = int(pf.readline().strip())
+ host = pf.readline().strip()
+ port = int(pf.readline().strip())
+ pf.close()
+ except IOError:
+ pid = None
+
+ if not pid:
+ sys.stderr.write("pidfile %s does not exist. Daemon not running?\n"
+ % PRServer.pidfile)
+ sys.exit(1)
+
+ PRServerConnection(host,port).terminate()
+ time.sleep(0.5)
+
+ try:
+ while 1:
+ os.kill(pid,signal.SIGTERM)
+ time.sleep(0.1)
+ except OSError, err:
+ err = str(err)
+ if err.find("No such process") > 0:
+ if os.path.exists(PRServer.pidfile):
+ os.remove(PRServer.pidfile)
+ else:
+ print err
+ sys.exit(1)
+
--
1.7.0.4
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH 2/5] conf/bitbake.conf: Added variables for PR service.
2011-05-19 10:29 [PATCH 0/5] network based PR service Lianhao Lu
2011-05-19 10:29 ` [PATCH 1/5] Added the " Lianhao Lu
@ 2011-05-19 10:29 ` Lianhao Lu
2011-05-19 11:51 ` Richard Purdie
2011-05-19 10:29 ` [PATCH 3/5] classes/package(prserv).bbclass: Added PR service support Lianhao Lu
` (5 subsequent siblings)
7 siblings, 1 reply; 22+ messages in thread
From: Lianhao Lu @ 2011-05-19 10:29 UTC (permalink / raw)
To: openembedded-core
From: Lianhao Lu <lianhao.lu@intel.com>
1. change BB_SIGNATURE_HANDLER from basic to basichash.
2. Added following variables for PR service:
USE_PR_SERV: flag of whether to use the network PR service
PRFORMAT: format of revision to be used in tasks package_write_xxx.
PRAUTOINX: search index for the network PR service
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
meta/conf/bitbake.conf | 14 ++++++++++++--
1 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index a0af672..381f301 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -153,8 +153,13 @@ PR = "${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[2] or 'r0
PF = "${PN}-${EXTENDPE}${PV}-${PR}"
EXTENDPE = "${@['','${PE\x7d_'][bb.data.getVar('PE',d,1) > 0]}"
EXTENDPEVER = "${@['','${PE\x7d:'][bb.data.getVar('PE',d,1) > 0]}"
-EXTENDPV = "${EXTENDPEVER}${PV}-${PR}"
+EXTENDPV = "${EXTENDPEVER}${PV}-${PRFORMAT}"
P = "${PN}-${PV}"
+EXTENDPREXTRA = "${@['.${PREXTRA\x7d',''][bb.data.getVar('PREXTRA',d,1) is None]}"
+EXTENDPRAUTO = "${@['.${PRAUTO\x7d',''][bb.data.getVar('PRAUTO',d,1) is None]}"
+PRAUTOINX = "${PF}${EXTENDPREXTRA}"
+PRFORMAT = "${PR}${EXTENDPREXTRA}${EXTENDPRAUTO}"
+
# Base package name
# Automatically derives "foo" from "foo-native", "foo-cross" or "foo-initial"
@@ -163,6 +168,11 @@ SPECIAL_PKGSUFFIX = "-native -cross -initial -intermediate -nativesdk -crosssdk
BPN = "${@base_prune_suffix(bb.data.getVar('PN', d, True), bb.data.getVar('SPECIAL_PKGSUFFIX', d, True).split(), d)}"
BP = "${BPN}-${PV}"
+#
+# network based PR service
+#
+USE_PR_SERV = "${@[1,0][(bb.data.getVar('PRSERV_HOST',d,1) is None) or (bb.data.getVar('PRSERV_PORT',d,1) is None)]}"
+
# Package info.
SECTION = "base"
@@ -707,7 +717,7 @@ DISTRO[unexport] = "1"
TRANSLATED_TARGET_ARCH ??= ${TARGET_ARCH}
# Setup our default hash policy
-BB_SIGNATURE_HANDLER ?= "basic"
+BB_SIGNATURE_HANDLER ?= "basichash"
BB_HASHTASK_WHITELIST ?= "(.*-cross$|.*-native$|.*-cross-initial$|.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE"
--
1.7.0.4
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 2/5] conf/bitbake.conf: Added variables for PR service.
2011-05-19 10:29 ` [PATCH 2/5] conf/bitbake.conf: Added variables for " Lianhao Lu
@ 2011-05-19 11:51 ` Richard Purdie
0 siblings, 0 replies; 22+ messages in thread
From: Richard Purdie @ 2011-05-19 11:51 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2011-05-19 at 18:29 +0800, Lianhao Lu wrote:
> From: Lianhao Lu <lianhao.lu@intel.com>
>
> 1. change BB_SIGNATURE_HANDLER from basic to basichash.
>
> 2. Added following variables for PR service:
> USE_PR_SERV: flag of whether to use the network PR service
> PRFORMAT: format of revision to be used in tasks package_write_xxx.
> PRAUTOINX: search index for the network PR service
>
> Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
> ---
> meta/conf/bitbake.conf | 14 ++++++++++++--
> 1 files changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index a0af672..381f301 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -153,8 +153,13 @@ PR = "${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[2] or 'r0
> PF = "${PN}-${EXTENDPE}${PV}-${PR}"
> EXTENDPE = "${@['','${PE\x7d_'][bb.data.getVar('PE',d,1) > 0]}"
> EXTENDPEVER = "${@['','${PE\x7d:'][bb.data.getVar('PE',d,1) > 0]}"
> -EXTENDPV = "${EXTENDPEVER}${PV}-${PR}"
> +EXTENDPV = "${EXTENDPEVER}${PV}-${PRFORMAT}"
> P = "${PN}-${PV}"
> +EXTENDPREXTRA = "${@['.${PREXTRA\x7d',''][bb.data.getVar('PREXTRA',d,1) is None]}"
> +EXTENDPRAUTO = "${@['.${PRAUTO\x7d',''][bb.data.getVar('PRAUTO',d,1) is None]}"
> +PRAUTOINX = "${PF}${EXTENDPREXTRA}"
> +PRFORMAT = "${PR}${EXTENDPREXTRA}${EXTENDPRAUTO}"
> +
There is also one thing we need to sync up with regarding openembedded
and these patches which is the use of PKGV and PKGR in package.bbclass.
Those patches are not merged into OE-Core yet but are have a similar
intention of allowing customisation to the PV and PR fields. I would
like to see if we can use PKGR for the use case we have here as well.
See:
http://git.openembedded.net/cgit.cgi/openembedded/tree/classes/package.bbclass
(I was only reminded of this recently in an IRC conversation with
Otavio)
Minor nitpick but new code should replace:
bb.data.getVar('PRAUTO',d,1)
with
d.getVar('PRAUTO', True)
which is functionally equivalent, neater and more pythonic. Its not a
major issue but worth highlighting.
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 3/5] classes/package(prserv).bbclass: Added PR service support.
2011-05-19 10:29 [PATCH 0/5] network based PR service Lianhao Lu
2011-05-19 10:29 ` [PATCH 1/5] Added the " Lianhao Lu
2011-05-19 10:29 ` [PATCH 2/5] conf/bitbake.conf: Added variables for " Lianhao Lu
@ 2011-05-19 10:29 ` Lianhao Lu
2011-05-19 11:54 ` Richard Purdie
2011-05-19 10:29 ` [PATCH 4/5] classes/package_xxx.class: " Lianhao Lu
` (4 subsequent siblings)
7 siblings, 1 reply; 22+ messages in thread
From: Lianhao Lu @ 2011-05-19 10:29 UTC (permalink / raw)
To: openembedded-core
From: Lianhao Lu <lianhao.lu@intel.com>
1. Added package_get_auto_rev to PACKAGEFUNCS to get the auto
incremented value(PRAUTO) from remote PR service.
2. Save PRFORMAT to pkgdata to be used by package_write_xxx.
3. Added supporting functions in prserv.bbclass.
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
meta/classes/package.bbclass | 36 +++++++++++++++++++++++++-----------
meta/classes/prserv.bbclass | 29 +++++++++++++++++++++++++++++
2 files changed, 54 insertions(+), 11 deletions(-)
create mode 100644 meta/classes/prserv.bbclass
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 4eb349d..efadbbd 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -7,34 +7,37 @@
#
# There are the following default steps but PACKAGEFUNCS can be extended:
#
-# a) perform_packagecopy - Copy D into PKGD
+# a) package_get_auto_rev - get PRAUTO from remote PR service
#
-# b) package_do_split_locales - Split out the locale files, updates FILES and PACKAGES
+# b) perform_packagecopy - Copy D into PKGD
#
-# c) split_and_strip_files - split the files into runtime and debug and strip them.
+# c) package_do_split_locales - Split out the locale files, updates FILES and PACKAGES
+#
+# d) split_and_strip_files - split the files into runtime and debug and strip them.
# Debug files include debug info split, and associated sources that end up in -dbg packages
#
-# d) populate_packages - Split the files in PKGD into separate packages in PKGDEST/<pkgname>
+# e) populate_packages - Split the files in PKGD into separate packages in PKGDEST/<pkgname>
# Also triggers the binary stripping code to put files in -dbg packages.
#
-# e) package_do_filedeps - Collect perfile run-time dependency metadata
+# f) package_do_filedeps - Collect perfile run-time dependency metadata
# The data is stores in FILER{PROVIDES,DEPENDS}_file_pkg variables with
# a list of affected files in FILER{PROVIDES,DEPENDS}FLIST_pkg
#
-# f) package_do_shlibs - Look at the shared libraries generated and autotmatically add any
+# g) package_do_shlibs - Look at the shared libraries generated and autotmatically add any
# depenedencies found. Also stores the package name so anyone else using this library
# knows which package to depend on.
#
-# g) package_do_pkgconfig - Keep track of which packages need and provide which .pc files
+# h) package_do_pkgconfig - Keep track of which packages need and provide which .pc files
#
-# h) read_shlibdeps - Reads the stored shlibs information into the metadata
+# i) read_shlibdeps - Reads the stored shlibs information into the metadata
#
-# i) package_depchains - Adds automatic dependencies to -dbg and -dev packages
+# j) package_depchains - Adds automatic dependencies to -dbg and -dev packages
#
-# j) emit_pkgdata - saves the packaging data into PKGDATA_DIR for use in later
+# k) emit_pkgdata - saves the packaging data into PKGDATA_DIR for use in later
# packaging steps
inherit packagedata
+inherit prserv
PKGD = "${WORKDIR}/package"
PKGDEST = "${WORKDIR}/packages-split"
@@ -324,6 +327,15 @@ def runtime_mapping_rename (varname, d):
# Package functions suitable for inclusion in PACKAGEFUNCS
#
+python package_get_auto_rev() {
+ if bb.data.getVar('USE_PR_SERV', d, True):
+ auto_rev=get_auto_rev(d)
+ if auto_rev is None:
+ bb.fatal("Can NOT get auto revision from remote PR service")
+ return
+ bb.data.setVar('PRAUTO',str(auto_rev),d)
+}
+
python package_do_split_locales() {
if (bb.data.getVar('PACKAGE_NO_LOCALE', d, True) == '1'):
bb.debug(1, "package requested not splitting locales")
@@ -771,6 +783,7 @@ python emit_pkgdata() {
write_if_exists(sf, pkg, 'PN')
write_if_exists(sf, pkg, 'PV')
write_if_exists(sf, pkg, 'PR')
+ write_if_exists(sf, pkg, 'PRFORMAT')
write_if_exists(sf, pkg, 'DESCRIPTION')
write_if_exists(sf, pkg, 'SUMMARY')
write_if_exists(sf, pkg, 'RDEPENDS')
@@ -1346,7 +1359,8 @@ python package_depchains() {
}
PACKAGE_PREPROCESS_FUNCS ?= ""
-PACKAGEFUNCS ?= "perform_packagecopy \
+PACKAGEFUNCS ?= "package_get_auto_rev \
+ perform_packagecopy \
${PACKAGE_PREPROCESS_FUNCS} \
package_do_split_locales \
split_and_strip_files \
diff --git a/meta/classes/prserv.bbclass b/meta/classes/prserv.bbclass
new file mode 100644
index 0000000..67c88f9
--- /dev/null
+++ b/meta/classes/prserv.bbclass
@@ -0,0 +1,29 @@
+def make_conn(d):
+ import prserv.serv
+ host=bb.data.getVar("PRSERV_HOST",d,True)
+ port=bb.data.getVar("PRSERV_PORT",d,True)
+ try:
+ conn=None
+ conn=prserv.serv.PRServerConnection(host,int(port))
+ bb.data.setVar("__PRSERV_CONN",conn,d)
+ except Exception, exc:
+ bb.fatal("Connecting to PR service %s:%s failed: %s" % (host, port, str(exc)))
+
+ return conn
+
+def get_auto_rev(d):
+ if not bb.data.getVar('USE_PR_SERV', d, True):
+ bb.warn("Not using network based PR service")
+ return None
+
+ conn=bb.data.getVar("__PRSERV_CONN", d, True)
+ if conn is None:
+ conn=make_conn(d)
+ if conn is None:
+ return None
+
+ version=bb.data.getVar("PF", d, True)
+ checksum=bb.data.getVar("BB_TASKHASH", d, True)
+ auto_rev=conn.getPR(version,checksum)
+ bb.debug(1,"get_auto_rev: version: %s checksum: %s result %d" % (version, checksum, auto_rev))
+ return auto_rev
--
1.7.0.4
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 3/5] classes/package(prserv).bbclass: Added PR service support.
2011-05-19 10:29 ` [PATCH 3/5] classes/package(prserv).bbclass: Added PR service support Lianhao Lu
@ 2011-05-19 11:54 ` Richard Purdie
0 siblings, 0 replies; 22+ messages in thread
From: Richard Purdie @ 2011-05-19 11:54 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2011-05-19 at 18:29 +0800, Lianhao Lu wrote:
> From: Lianhao Lu <lianhao.lu@intel.com>
>
> 1. Added package_get_auto_rev to PACKAGEFUNCS to get the auto
> incremented value(PRAUTO) from remote PR service.
>
> 2. Save PRFORMAT to pkgdata to be used by package_write_xxx.
>
> 3. Added supporting functions in prserv.bbclass.
>
> Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
> ---
> meta/classes/package.bbclass | 36 +++++++++++++++++++++++++-----------
> meta/classes/prserv.bbclass | 29 +++++++++++++++++++++++++++++
> 2 files changed, 54 insertions(+), 11 deletions(-)
> create mode 100644 meta/classes/prserv.bbclass
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 4eb349d..efadbbd 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -324,6 +327,15 @@ def runtime_mapping_rename (varname, d):
> # Package functions suitable for inclusion in PACKAGEFUNCS
> #
>
> +python package_get_auto_rev() {
> + if bb.data.getVar('USE_PR_SERV', d, True):
> + auto_rev=get_auto_rev(d)
> + if auto_rev is None:
> + bb.fatal("Can NOT get auto revision from remote PR service")
> + return
> + bb.data.setVar('PRAUTO',str(auto_rev),d)
> +}
> +
> python package_do_split_locales() {
> if (bb.data.getVar('PACKAGE_NO_LOCALE', d, True) == '1'):
> bb.debug(1, "package requested not splitting locales")
This looks a little confusing to me as at first glance I'd have said it
was related to SRCREV auto incrementing. Can we add something to to with
pr to the function name? Function names like get_auto_rev() and
make_conf() also need clearer namespacing to include prserv as a prefix
so its clearer where the functions are from and what they do.
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 4/5] classes/package_xxx.class: Added PR service support.
2011-05-19 10:29 [PATCH 0/5] network based PR service Lianhao Lu
` (2 preceding siblings ...)
2011-05-19 10:29 ` [PATCH 3/5] classes/package(prserv).bbclass: Added PR service support Lianhao Lu
@ 2011-05-19 10:29 ` Lianhao Lu
2011-05-19 10:29 ` [PATCH 5/5] meta-yocto/local.conf.sample: Added PRSERV_HOST and PRSERV_PORT Lianhao Lu
` (3 subsequent siblings)
7 siblings, 0 replies; 22+ messages in thread
From: Lianhao Lu @ 2011-05-19 10:29 UTC (permalink / raw)
To: openembedded-core
From: Lianhao Lu <lianhao.lu@intel.com>
Use PRFORMAT instead of PR to build various package feed in tasks
of pacakge_write_xxx.
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
meta/classes/package_deb.bbclass | 6 +++---
meta/classes/package_ipk.bbclass | 6 +++---
meta/classes/package_rpm.bbclass | 4 ++--
meta/classes/package_tar.bbclass | 4 ++--
4 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 4faeb4a..1f26d34 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -272,7 +272,7 @@ python do_package_deb () {
except ValueError:
pass
if not g and bb.data.getVar('ALLOW_EMPTY', localdata) != "1":
- bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, True), bb.data.getVar('PR', localdata, True)))
+ bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, True), bb.data.getVar('PRFORMAT', localdata, True)))
bb.utils.unlockfile(lf)
continue
@@ -290,9 +290,9 @@ python do_package_deb () {
fields = []
pe = bb.data.getVar('PE', d, True)
if pe and int(pe) > 0:
- fields.append(["Version: %s:%s-%s\n", ['PE', 'PV', 'PR']])
+ fields.append(["Version: %s:%s-%s\n", ['PE', 'PV', 'PRFORMAT']])
else:
- fields.append(["Version: %s-%s\n", ['PV', 'PR']])
+ fields.append(["Version: %s-%s\n", ['PV', 'PRFORMAT']])
fields.append(["Description: %s\n", ['DESCRIPTION']])
fields.append(["Section: %s\n", ['SECTION']])
fields.append(["Priority: %s\n", ['PRIORITY']])
diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 3c2472b..7bec136 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -241,7 +241,7 @@ python do_package_ipk () {
except ValueError:
pass
if not g and bb.data.getVar('ALLOW_EMPTY', localdata) != "1":
- bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, 1), bb.data.getVar('PR', localdata, 1)))
+ bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, 1), bb.data.getVar('PRFORMAT', localdata, 1)))
bb.utils.unlockfile(lf)
continue
@@ -256,9 +256,9 @@ python do_package_ipk () {
fields = []
pe = bb.data.getVar('PE', d, 1)
if pe and int(pe) > 0:
- fields.append(["Version: %s:%s-%s\n", ['PE', 'PV', 'PR']])
+ fields.append(["Version: %s:%s-%s\n", ['PE', 'PV', 'PRFORMAT']])
else:
- fields.append(["Version: %s-%s\n", ['PV', 'PR']])
+ fields.append(["Version: %s-%s\n", ['PV', 'PRFORMAT']])
fields.append(["Description: %s\n", ['DESCRIPTION']])
fields.append(["Section: %s\n", ['SECTION']])
fields.append(["Priority: %s\n", ['PRIORITY']])
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 1cf9f79..ecb6f45 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -383,7 +383,7 @@ python write_specfile () {
srcname = bb.data.getVar('PN', d, True)
srcsummary = (bb.data.getVar('SUMMARY', d, True) or bb.data.getVar('DESCRIPTION', d, True) or ".")
srcversion = bb.data.getVar('PV', d, True).replace('-', '+')
- srcrelease = bb.data.getVar('PR', d, True)
+ srcrelease = bb.data.getVar('PRFORMAT', d, True)
srcepoch = (bb.data.getVar('PE', d, True) or "")
srclicense = bb.data.getVar('LICENSE', d, True)
srcsection = bb.data.getVar('SECTION', d, True)
@@ -438,7 +438,7 @@ python write_specfile () {
splitsummary = (bb.data.getVar('SUMMARY', localdata, True) or bb.data.getVar('DESCRIPTION', localdata, True) or ".")
splitversion = (bb.data.getVar('PV', localdata, True) or "").replace('-', '+')
- splitrelease = (bb.data.getVar('PR', localdata, True) or "")
+ splitrelease = (bb.data.getVar('PRFORMAT', localdata, True) or "")
splitepoch = (bb.data.getVar('PE', localdata, True) or "")
splitlicense = (bb.data.getVar('LICENSE', localdata, True) or "")
splitsection = (bb.data.getVar('SECTION', localdata, True) or "")
diff --git a/meta/classes/package_tar.bbclass b/meta/classes/package_tar.bbclass
index e546eb7..d4d7980 100644
--- a/meta/classes/package_tar.bbclass
+++ b/meta/classes/package_tar.bbclass
@@ -3,7 +3,7 @@ inherit package
IMAGE_PKGTYPE ?= "tar"
python package_tar_fn () {
- fn = os.path.join(bb.data.getVar('DEPLOY_DIR_TAR', d), "%s-%s-%s.tar.gz" % (bb.data.getVar('PKG', d), bb.data.getVar('PV', d), bb.data.getVar('PR', d)))
+ fn = os.path.join(bb.data.getVar('DEPLOY_DIR_TAR', d), "%s-%s-%s.tar.gz" % (bb.data.getVar('PKG', d), bb.data.getVar('PV', d), bb.data.getVar('PRFORMAT', d)))
fn = bb.data.expand(fn, d)
bb.data.setVar('PKGFN', fn, d)
}
@@ -83,7 +83,7 @@ python do_package_tar () {
os.chdir(root)
from glob import glob
if not glob('*'):
- bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, 1), bb.data.getVar('PR', localdata, 1)))
+ bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, 1), bb.data.getVar('PRFORMAT', localdata, 1)))
continue
ret = os.system("tar -czf %s %s" % (tarfn, '.'))
if ret != 0:
--
1.7.0.4
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH 5/5] meta-yocto/local.conf.sample: Added PRSERV_HOST and PRSERV_PORT.
2011-05-19 10:29 [PATCH 0/5] network based PR service Lianhao Lu
` (3 preceding siblings ...)
2011-05-19 10:29 ` [PATCH 4/5] classes/package_xxx.class: " Lianhao Lu
@ 2011-05-19 10:29 ` Lianhao Lu
2011-05-19 10:54 ` [PATCH 0/5] network based PR service Koen Kooi
` (2 subsequent siblings)
7 siblings, 0 replies; 22+ messages in thread
From: Lianhao Lu @ 2011-05-19 10:29 UTC (permalink / raw)
To: openembedded-core
From: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
meta-yocto/conf/local.conf.sample | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/meta-yocto/conf/local.conf.sample b/meta-yocto/conf/local.conf.sample
index a9b4998..85c26ee 100644
--- a/meta-yocto/conf/local.conf.sample
+++ b/meta-yocto/conf/local.conf.sample
@@ -216,3 +216,8 @@ NO32LIBS = "1"
# GNOME, SCREEN, XTERM and KONSOLE
#TERMCMD = "${KONSOLE_TERMCMD}"
#TERMCMDRUN = "${KONSOLE_TERMCMDRUN}"
+
+
+# The network based PR service host and port
+PRSERV_HOST = "localhost"
+PRSERV_PORT = "8585"
--
1.7.0.4
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 0/5] network based PR service
2011-05-19 10:29 [PATCH 0/5] network based PR service Lianhao Lu
` (4 preceding siblings ...)
2011-05-19 10:29 ` [PATCH 5/5] meta-yocto/local.conf.sample: Added PRSERV_HOST and PRSERV_PORT Lianhao Lu
@ 2011-05-19 10:54 ` Koen Kooi
2011-05-19 11:38 ` Richard Purdie
2011-05-19 11:01 ` Frans Meulenbroeks
2011-05-19 12:02 ` Richard Purdie
7 siblings, 1 reply; 22+ messages in thread
From: Koen Kooi @ 2011-05-19 10:54 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
Op 19 mei 2011, om 12:29 heeft Lianhao Lu het volgende geschreven:
> From: Lianhao Lu <lianhao.lu@intel.com>
>
> This series of 5 patches implemented the network based PR service and enabled
> the poky to use it during the task do_package and do_package_write_xxx. By
> using the network based PR service and the basichash for BB_SIGNATURE_HANDLER,
> the poky user may not need to bump the PR manually everytime he/she changes
> the recipe. The package will get automatically rebuilt and new revision number
> reflecting the change will be included in the package feed.
Does it have a public/private mode? In the angstrom case, we only want developers to 'submit' PR bumps and users only retrieve them.
regards,
Koen
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH 0/5] network based PR service
2011-05-19 10:54 ` [PATCH 0/5] network based PR service Koen Kooi
@ 2011-05-19 11:38 ` Richard Purdie
2011-05-19 11:51 ` Koen Kooi
0 siblings, 1 reply; 22+ messages in thread
From: Richard Purdie @ 2011-05-19 11:38 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2011-05-19 at 12:54 +0200, Koen Kooi wrote:
> Op 19 mei 2011, om 12:29 heeft Lianhao Lu het volgende geschreven:
>
> > From: Lianhao Lu <lianhao.lu@intel.com>
> >
> > This series of 5 patches implemented the network based PR service and enabled
> > the poky to use it during the task do_package and do_package_write_xxx. By
> > using the network based PR service and the basichash for BB_SIGNATURE_HANDLER,
> > the poky user may not need to bump the PR manually everytime he/she changes
> > the recipe. The package will get automatically rebuilt and new revision number
> > reflecting the change will be included in the package feed.
>
> Does it have a public/private mode? In the angstrom case, we only want
> developers to 'submit' PR bumps and users only retrieve them.
It doesn't look like it but it wouldn't be hard to only submit PR
changes if some kind of token was present.
The big question would be, if a user tried building something for which
PR no record was on the server and didn't have submit privileges, what
should the server return and how should the build behave? Hard error?
If its not a hard error, you are likely to run into local
reproducibility issues and also conflicts with the server?
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 11:38 ` Richard Purdie
@ 2011-05-19 11:51 ` Koen Kooi
2011-05-19 12:10 ` Richard Purdie
0 siblings, 1 reply; 22+ messages in thread
From: Koen Kooi @ 2011-05-19 11:51 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
Op 19 mei 2011, om 13:38 heeft Richard Purdie het volgende geschreven:
> On Thu, 2011-05-19 at 12:54 +0200, Koen Kooi wrote:
>> Op 19 mei 2011, om 12:29 heeft Lianhao Lu het volgende geschreven:
>>
>>> From: Lianhao Lu <lianhao.lu@intel.com>
>>>
>>> This series of 5 patches implemented the network based PR service and enabled
>>> the poky to use it during the task do_package and do_package_write_xxx. By
>>> using the network based PR service and the basichash for BB_SIGNATURE_HANDLER,
>>> the poky user may not need to bump the PR manually everytime he/she changes
>>> the recipe. The package will get automatically rebuilt and new revision number
>>> reflecting the change will be included in the package feed.
>>
>> Does it have a public/private mode? In the angstrom case, we only want
>> developers to 'submit' PR bumps and users only retrieve them.
>
> It doesn't look like it but it wouldn't be hard to only submit PR
> changes if some kind of token was present.
>
> The big question would be, if a user tried building something for which
> PR no record was on the server and didn't have submit privileges, what
> should the server return and how should the build behave? Hard error?
>
> If its not a hard error, you are likely to run into local
> reproducibility issues and also conflicts with the server?
I would go for a warning on the client side and an error in the log in the server side. If the server people care enough they'll check the log for errors and add the missing info.
I think we need to give this a try and see what improvements we need.
regards,
Koen
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 11:51 ` Koen Kooi
@ 2011-05-19 12:10 ` Richard Purdie
0 siblings, 0 replies; 22+ messages in thread
From: Richard Purdie @ 2011-05-19 12:10 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2011-05-19 at 13:51 +0200, Koen Kooi wrote:
> Op 19 mei 2011, om 13:38 heeft Richard Purdie het volgende geschreven:
>
> > On Thu, 2011-05-19 at 12:54 +0200, Koen Kooi wrote:
> >> Op 19 mei 2011, om 12:29 heeft Lianhao Lu het volgende geschreven:
> >>
> >>> From: Lianhao Lu <lianhao.lu@intel.com>
> >>>
> >>> This series of 5 patches implemented the network based PR service and enabled
> >>> the poky to use it during the task do_package and do_package_write_xxx. By
> >>> using the network based PR service and the basichash for BB_SIGNATURE_HANDLER,
> >>> the poky user may not need to bump the PR manually everytime he/she changes
> >>> the recipe. The package will get automatically rebuilt and new revision number
> >>> reflecting the change will be included in the package feed.
> >>
> >> Does it have a public/private mode? In the angstrom case, we only want
> >> developers to 'submit' PR bumps and users only retrieve them.
> >
> > It doesn't look like it but it wouldn't be hard to only submit PR
> > changes if some kind of token was present.
> >
> > The big question would be, if a user tried building something for which
> > PR no record was on the server and didn't have submit privileges, what
> > should the server return and how should the build behave? Hard error?
> >
> > If its not a hard error, you are likely to run into local
> > reproducibility issues and also conflicts with the server?
>
> I would go for a warning on the client side and an error in the log in
> the server side. If the server people care enough they'll check the
> log for errors and add the missing info.
How is the client meant to continue on when the value it gets from the
server is effectively a "don't know" reply? Default to 0?
Thinking about it, its probably fine as long as it defaulted back to
"lr" or something to show there was likely a local change that caused it
to deviate from what was on the server. It just needs to be clear it
doesn't correspond to the upstream server.
> I think we need to give this a try and see what improvements we need.
Agreed.
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 10:29 [PATCH 0/5] network based PR service Lianhao Lu
` (5 preceding siblings ...)
2011-05-19 10:54 ` [PATCH 0/5] network based PR service Koen Kooi
@ 2011-05-19 11:01 ` Frans Meulenbroeks
2011-05-19 11:27 ` Frans Meulenbroeks
2011-05-19 11:35 ` Richard Purdie
2011-05-19 12:02 ` Richard Purdie
7 siblings, 2 replies; 22+ messages in thread
From: Frans Meulenbroeks @ 2011-05-19 11:01 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
2011/5/19 Lianhao Lu <lianhao.lu@intel.com>
> From: Lianhao Lu <lianhao.lu@intel.com>
>
> This series of 5 patches implemented the network based PR service and
> enabled
> the poky to use it during the task do_package and do_package_write_xxx. By
> using the network based PR service and the basichash for
> BB_SIGNATURE_HANDLER,
> the poky user may not need to bump the PR manually everytime he/she changes
> the recipe. The package will get automatically rebuilt and new revision
> number
> reflecting the change will be included in the package feed.
>
> The first patch "[PATCH 1/5] Added the PR service." implemented the network
> based PR service on the server side. It is a XMLRPC server with the sqlite3
> as the backend database. The users query an automated value from this
> service
> by giving a tuple of (version, checksum). The returned value will
> be used as a part of the package revision value during the package feed
> creation process. The following algorihtm is used so this PR service could
> give
> multiple build servers new values for the new index tuple and same values
> for
> the same index tuple:
>
> IF the index tuple(version, checksum) is found, THEN
> return the value.
> ELSE
> find the max value in the database with the same version.
> increment it and save it into the database along with the index tuple.
> return the incremented value.
> (Note: If no matching version is found, save and return the value of 0).
> ENDIF
>
> To start the network based PR service daemon, run the following command
> after
> "sourcing" the environment file oe-init-build-env:
>
> bitbake-prserv --start
>
> See bitbake-prserv --help to see a detailed description of the options.
>
> The remaining 4 patches enable the poky to use the PR service. In order to
> use
> it, the user needs to set 2 varialbes of PRSERV_HOST and PRSERV_PORT (which
> specify the PR service ip address and port, default is "localhost" and
> "8585"
> respectively) in the file conf/local.conf under the building directory.
> Unsetting these 2 variables would disable the poky to use the PR service
> for
> backward compatibility.
>
> The following important new variables are introduced in bitbake.conf:
>
> PRFORMAT: The final value goes into the package revision. Default is
> something
> like PR.PRAUTO, where PRAUTO is the value returned by the PR service.
>
> PRAUTOINX: The version part of the index tuple for the PR service. Default
> is PF.
> The checksum of that tuple is the task checksum of do_package.
>
> Pull URL: git://git.pokylinux.org/poky-contrib.git
> Branch: llu/PR-service
> Browse:
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=llu/PR-service
>
> Thanks,
> Lianhao Lu <lianhao.lu@intel.com>
> ---
>
>
> Lianhao Lu (5):
> Added the PR service.
> conf/bitbake.conf: Added variables for PR service.
> classes/package(prserv).bbclass: Added PR service support.
> classes/package_xxx.class: Added PR service support.
> meta-yocto/local.conf.sample: Added PRSERV_HOST and PRSERV_PORT.
>
> bitbake/bin/bitbake-prserv | 53 ++++++++++
> bitbake/lib/prserv/__init__.py | 11 ++
> bitbake/lib/prserv/db.py | 100 +++++++++++++++++++
> bitbake/lib/prserv/serv.py | 198
> +++++++++++++++++++++++++++++++++++++
> meta-yocto/conf/local.conf.sample | 5 +
> meta/classes/package.bbclass | 36 +++++--
> meta/classes/package_deb.bbclass | 6 +-
> meta/classes/package_ipk.bbclass | 6 +-
> meta/classes/package_rpm.bbclass | 4 +-
> meta/classes/package_tar.bbclass | 4 +-
> meta/classes/prserv.bbclass | 29 ++++++
> meta/conf/bitbake.conf | 14 +++-
> 12 files changed, 443 insertions(+), 23 deletions(-)
> create mode 100755 bitbake/bin/bitbake-prserv
> create mode 100644 bitbake/lib/prserv/__init__.py
> create mode 100644 bitbake/lib/prserv/db.py
> create mode 100644 bitbake/lib/prserv/serv.py
> create mode 100644 meta/classes/prserv.bbclass
>
>
> What should I do to either disable this for some recipes, or use a
different (private) server?
We do not wish to rely on an external server for proprietary recipes (or for
recipes for which we made a local change in an overlay).
How is this case handled?
Frans
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH 0/5] network based PR service
2011-05-19 11:01 ` Frans Meulenbroeks
@ 2011-05-19 11:27 ` Frans Meulenbroeks
2011-05-19 11:35 ` Richard Purdie
1 sibling, 0 replies; 22+ messages in thread
From: Frans Meulenbroeks @ 2011-05-19 11:27 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
>
>> What should I do to either disable this for some recipes, or use a
> different (private) server?
> We do not wish to rely on an external server for proprietary recipes (or
> for recipes for which we made a local change in an overlay).
> How is this case handled?
>
> Ok found some of the answers in the patches, but that has raised some more
questions for which I did not see the answer right away.
Would it be possible to have a PR_SERVER per layer?
And are the values cached locally? (e.g. what happens if the PR_SERVER is
not availalbe).
I guess it would also be helpful to have some usage documentation.
(actually I seem to recall that at some point in time within OE we had the
policy that if you introduced a new var you should also update the docs for
it; not sure if that was formalized somewhere, but definitely seems a good
plan).
Frans
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 11:01 ` Frans Meulenbroeks
2011-05-19 11:27 ` Frans Meulenbroeks
@ 2011-05-19 11:35 ` Richard Purdie
2011-05-19 12:02 ` Frans Meulenbroeks
1 sibling, 1 reply; 22+ messages in thread
From: Richard Purdie @ 2011-05-19 11:35 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2011-05-19 at 13:01 +0200, Frans Meulenbroeks wrote:
> What should I do to either disable this for some recipes, or use a
> different (private) server?
> We do not wish to rely on an external server for proprietary recipes (or for
> recipes for which we made a local change in an overlay).
> How is this case handled?
This could be handled by doing things like setting:
PRSERV_HOST_pn-myprivaterecipe = "somelocalhost"
or
PRSERV_HOST_pn-myprivaterecipe = ""
to disable it.
One thing we lack is a good way to apply changes like this, only if code
is within a given layer. Even in that case, it should be possible with
anonymous python to look at the location of the current .bb file and
then conditionally set these variables as appropriate to the right
server.
Certainly this needs to be thought about and documented but I think
there are ways to do it.
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 11:35 ` Richard Purdie
@ 2011-05-19 12:02 ` Frans Meulenbroeks
2011-05-19 12:22 ` Richard Purdie
0 siblings, 1 reply; 22+ messages in thread
From: Frans Meulenbroeks @ 2011-05-19 12:02 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
2011/5/19 Richard Purdie <richard.purdie@linuxfoundation.org>
> On Thu, 2011-05-19 at 13:01 +0200, Frans Meulenbroeks wrote:
> > What should I do to either disable this for some recipes, or use a
> > different (private) server?
> > We do not wish to rely on an external server for proprietary recipes (or
> for
> > recipes for which we made a local change in an overlay).
> > How is this case handled?
>
> This could be handled by doing things like setting:
>
> PRSERV_HOST_pn-myprivaterecipe = "somelocalhost"
>
> or
>
> PRSERV_HOST_pn-myprivaterecipe = ""
>
> to disable it.
>
> One thing we lack is a good way to apply changes like this, only if code
> is within a given layer. Even in that case, it should be possible with
> anonymous python to look at the location of the current .bb file and
> then conditionally set these variables as appropriate to the right
> server.
>
> Certainly this needs to be thought about and documented but I think
> there are ways to do it.
>
> Pardon my ignorance, but I do not really understand the complete flow and
way of working.
Anyway if I do something like:
PRSERV_HOST_pn-myprivaterecipe = ""
would I still be able to use PR in my recipe (like I do today)?
Also we do have the issue that it is desired to be able to rebuild without
network connectivity (e.g. while temporary offline while travelling). Would
that still be possible?
For layers, one solution could be to allow variable overriding on the
overlay level. I can imagine there are more uses for that (and I understand
this requires changes to the bitbake machinery).
Frans.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 12:02 ` Frans Meulenbroeks
@ 2011-05-19 12:22 ` Richard Purdie
2011-05-19 12:43 ` Frans Meulenbroeks
0 siblings, 1 reply; 22+ messages in thread
From: Richard Purdie @ 2011-05-19 12:22 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2011-05-19 at 14:02 +0200, Frans Meulenbroeks wrote:
> 2011/5/19 Richard Purdie <richard.purdie@linuxfoundation.org>
>
> > On Thu, 2011-05-19 at 13:01 +0200, Frans Meulenbroeks wrote:
> > > What should I do to either disable this for some recipes, or use a
> > > different (private) server?
> > > We do not wish to rely on an external server for proprietary recipes (or
> > for
> > > recipes for which we made a local change in an overlay).
> > > How is this case handled?
> >
> > This could be handled by doing things like setting:
> >
> > PRSERV_HOST_pn-myprivaterecipe = "somelocalhost"
> >
> > or
> >
> > PRSERV_HOST_pn-myprivaterecipe = ""
> >
> > to disable it.
> >
> > One thing we lack is a good way to apply changes like this, only if code
> > is within a given layer. Even in that case, it should be possible with
> > anonymous python to look at the location of the current .bb file and
> > then conditionally set these variables as appropriate to the right
> > server.
> >
> > Certainly this needs to be thought about and documented but I think
> > there are ways to do it.
> >
> > Pardon my ignorance, but I do not really understand the complete flow and
> way of working.
>
> Anyway if I do something like:
> PRSERV_HOST_pn-myprivaterecipe = ""
> would I still be able to use PR in my recipe (like I do today)?
Yes, since the PR server appends to this.
> Also we do have the issue that it is desired to be able to rebuild without
> network connectivity (e.g. while temporary offline while travelling). Would
> that still be possible?
You could use a local PR server. Obviously connecting to one central
server without any network connectivity isn't going to happen so we have
to be realistic about expectations.
To make a perfect rebuild the local PR server would need a dump of the
database on the central server. There isn't code for that at the moment
and I don't think its the highest priority task out there or the most
important use case but its certainly possible for someone to add.
> For layers, one solution could be to allow variable overriding on the
> overlay level. I can imagine there are more uses for that (and I understand
> this requires changes to the bitbake machinery).
There is certainly a use case for something like this. The exact
implementation and workings needs a lot more thought and discussion
though. I believe its at least already possible in anonymous python (and
if not, any extensions needed shouldn't be invasive by comparison).
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 12:22 ` Richard Purdie
@ 2011-05-19 12:43 ` Frans Meulenbroeks
2011-05-19 13:13 ` Richard Purdie
0 siblings, 1 reply; 22+ messages in thread
From: Frans Meulenbroeks @ 2011-05-19 12:43 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
2011/5/19 Richard Purdie <richard.purdie@linuxfoundation.org>
> On Thu, 2011-05-19 at 14:02 +0200, Frans Meulenbroeks wrote:
> > 2011/5/19 Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> > > On Thu, 2011-05-19 at 13:01 +0200, Frans Meulenbroeks wrote:
> > > > What should I do to either disable this for some recipes, or use a
> > > > different (private) server?
> > > > We do not wish to rely on an external server for proprietary recipes
> (or
> > > for
> > > > recipes for which we made a local change in an overlay).
> > > > How is this case handled?
> > >
> > > This could be handled by doing things like setting:
> > >
> > > PRSERV_HOST_pn-myprivaterecipe = "somelocalhost"
> > >
> > > or
> > >
> > > PRSERV_HOST_pn-myprivaterecipe = ""
> > >
> > > to disable it.
> > >
> > > One thing we lack is a good way to apply changes like this, only if
> code
> > > is within a given layer. Even in that case, it should be possible with
> > > anonymous python to look at the location of the current .bb file and
> > > then conditionally set these variables as appropriate to the right
> > > server.
> > >
> > > Certainly this needs to be thought about and documented but I think
> > > there are ways to do it.
> > >
> > > Pardon my ignorance, but I do not really understand the complete flow
> and
> > way of working.
> >
> > Anyway if I do something like:
> > PRSERV_HOST_pn-myprivaterecipe = ""
> > would I still be able to use PR in my recipe (like I do today)?
>
> Yes, since the PR server appends to this.
>
> > Also we do have the issue that it is desired to be able to rebuild
> without
> > network connectivity (e.g. while temporary offline while travelling).
> Would
> > that still be possible?
>
> You could use a local PR server. Obviously connecting to one central
> server without any network connectivity isn't going to happen so we have
> to be realistic about expectations.
>
> To make a perfect rebuild the local PR server would need a dump of the
> database on the central server. There isn't code for that at the moment
> and I don't think its the highest priority task out there or the most
> important use case but its certainly possible for someone to add.
>
I'd say it would already be nice if some caching is being done locally (just
like is done with e.g. downloads).
>
> > For layers, one solution could be to allow variable overriding on the
> > overlay level. I can imagine there are more uses for that (and I
> understand
> > this requires changes to the bitbake machinery).
>
> There is certainly a use case for something like this. The exact
> implementation and workings needs a lot more thought and discussion
> though. I believe its at least already possible in anonymous python (and
> if not, any extensions needed shouldn't be invasive by comparison).
>
> Hm. you consider this PR change to be non-invasive?
BTW I am not saying it is not good, and I understand the problem that you
want to solve, but I feel this could require some more thought wrt the
issues I raised before in this thread (and some more documentation and usage
info).
Frans.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 12:43 ` Frans Meulenbroeks
@ 2011-05-19 13:13 ` Richard Purdie
2011-05-19 14:58 ` Mark Hatle
0 siblings, 1 reply; 22+ messages in thread
From: Richard Purdie @ 2011-05-19 13:13 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2011-05-19 at 14:43 +0200, Frans Meulenbroeks wrote:
> 2011/5/19 Richard Purdie <richard.purdie@linuxfoundation.org>
>
> > On Thu, 2011-05-19 at 14:02 +0200, Frans Meulenbroeks wrote:
> > > 2011/5/19 Richard Purdie <richard.purdie@linuxfoundation.org>
> > >
> > > > On Thu, 2011-05-19 at 13:01 +0200, Frans Meulenbroeks wrote:
> > You could use a local PR server. Obviously connecting to one central
> > server without any network connectivity isn't going to happen so we have
> > to be realistic about expectations.
> >
> > To make a perfect rebuild the local PR server would need a dump of the
> > database on the central server. There isn't code for that at the moment
> > and I don't think its the highest priority task out there or the most
> > important use case but its certainly possible for someone to add.
> >
>
> I'd say it would already be nice if some caching is being done locally (just
> like is done with e.g. downloads).
The difference is you need all the data, not just a given value. The
equivalent in downloads is downloading every possible download version
in advance just in case so the analogy isn't a good one. In this case
it would be possible to do that though as the data is smaller.
> > > For layers, one solution could be to allow variable overriding on the
> > > overlay level. I can imagine there are more uses for that (and I
> > understand
> > > this requires changes to the bitbake machinery).
> >
> > There is certainly a use case for something like this. The exact
> > implementation and workings needs a lot more thought and discussion
> > though. I believe its at least already possible in anonymous python (and
> > if not, any extensions needed shouldn't be invasive by comparison).
>
> Hm. you consider this PR change to be non-invasive?
No, this isn't what I said. "allow variable overriding on the
overlay level" is invasive. Adding functionality to allow this kind of
thing from anonymous python is by comparison less invasive.
Its also a tangential issue to the PR part although obviously useful in
connection with it.
> BTW I am not saying it is not good, and I understand the problem that you
> want to solve, but I feel this could require some more thought wrt the
> issues I raised before in this thread (and some more documentation and usage
> info).
At this point, an implementation which we can look at, experiment with
and document is better than none at all :).
Ideally some of these issues would have come up at the design stage but
we'll work through this and I think things area heading the right way.
Nobody is saying this is 100% complete or solves every problem.
I don't believe this code needs to solve every problem straight off but
it does at least need to be extensible to cover them in future by design
as far as possible.
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 13:13 ` Richard Purdie
@ 2011-05-19 14:58 ` Mark Hatle
0 siblings, 0 replies; 22+ messages in thread
From: Mark Hatle @ 2011-05-19 14:58 UTC (permalink / raw)
To: openembedded-core
On 5/19/11 8:13 AM, Richard Purdie wrote:
> On Thu, 2011-05-19 at 14:43 +0200, Frans Meulenbroeks wrote:
>> 2011/5/19 Richard Purdie <richard.purdie@linuxfoundation.org>
>>
>>> On Thu, 2011-05-19 at 14:02 +0200, Frans Meulenbroeks wrote:
>>>> 2011/5/19 Richard Purdie <richard.purdie@linuxfoundation.org>
>>>>
>>>>> On Thu, 2011-05-19 at 13:01 +0200, Frans Meulenbroeks wrote:
>>> You could use a local PR server. Obviously connecting to one central
>>> server without any network connectivity isn't going to happen so we have
>>> to be realistic about expectations.
>>>
>>> To make a perfect rebuild the local PR server would need a dump of the
>>> database on the central server. There isn't code for that at the moment
>>> and I don't think its the highest priority task out there or the most
>>> important use case but its certainly possible for someone to add.
>>>
>>
>> I'd say it would already be nice if some caching is being done locally (just
>> like is done with e.g. downloads).
>
> The difference is you need all the data, not just a given value. The
> equivalent in downloads is downloading every possible download version
> in advance just in case so the analogy isn't a good one. In this case
> it would be possible to do that though as the data is smaller.
I've been thinking about this. There are really three use-cases that I see.
The first (difficult one) is what is implemented:
Q: I have a group of developers all building software, I want everyone's
packages to get the same version string if the package is the same.
This answers that pretty well, a global server can be enabled for a group of
developers to work with. Of course the read/write question is "interesting".
One possible solution for the read-only user is to allow for a default version
to be specified. At a former company, when the users ran their own builds we
simply returned "custom" as the build version string.
Second case, raised by Koen:
Q: I'm building a distribution that I want to share with others, how can they
work with it -- yet make their own changes?
In this case I think it makes sense for the distribution vendor to export their
database of checksums and version values. Then the end user can use this to
seed their PR server. (Would be nice if you could look at multiple servers,
starting with an authoritative and working down from there until its found..
assuming R/O means "not found". If it's not found then writing to a specific
location would make sense to me.)
Third case...
Q: I'm building the software myself and simply want to track versions.
This one to me is likely a common use case. Pretty similar to the first group
case, but I don't need or want a "network" service.. just a way to manage things
on my local build machines. So either run it as a local service, or figure out
a way to directly access a database as part of build environment...
[Fourth case]
Q: I don't care about versioning beyond what the recipe contains.
This is the existing case, and certainly needs to continue working....
>>>> For layers, one solution could be to allow variable overriding on the
>>>> overlay level. I can imagine there are more uses for that (and I
>>> understand
>>>> this requires changes to the bitbake machinery).
>>>
>>> There is certainly a use case for something like this. The exact
>>> implementation and workings needs a lot more thought and discussion
>>> though. I believe its at least already possible in anonymous python (and
>>> if not, any extensions needed shouldn't be invasive by comparison).
>>
>> Hm. you consider this PR change to be non-invasive?
>
> No, this isn't what I said. "allow variable overriding on the
> overlay level" is invasive. Adding functionality to allow this kind of
> thing from anonymous python is by comparison less invasive.
>
> Its also a tangential issue to the PR part although obviously useful in
> connection with it.
>
>> BTW I am not saying it is not good, and I understand the problem that you
>> want to solve, but I feel this could require some more thought wrt the
>> issues I raised before in this thread (and some more documentation and usage
>> info).
>
> At this point, an implementation which we can look at, experiment with
> and document is better than none at all :).
>
> Ideally some of these issues would have come up at the design stage but
> we'll work through this and I think things area heading the right way.
> Nobody is saying this is 100% complete or solves every problem.
>
> I don't believe this code needs to solve every problem straight off but
> it does at least need to be extensible to cover them in future by design
> as far as possible.
Ya, I agree. The problem is fairly well know.. We need a way to track build
changes and share this tracking between builds. I think this is a reasonable
solution, but we certainly need to find and resolve the various issues that
people will have with it.
I'm certainly in favor of adding this into oe-core, but having it disabled by
default -- for now. Then those interested can experiment and work on solutions
to the problems they are facing. (The distribution problem I think will be the
trickiest to resolve.)
--Mark
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/5] network based PR service
2011-05-19 10:29 [PATCH 0/5] network based PR service Lianhao Lu
` (6 preceding siblings ...)
2011-05-19 11:01 ` Frans Meulenbroeks
@ 2011-05-19 12:02 ` Richard Purdie
7 siblings, 0 replies; 22+ messages in thread
From: Richard Purdie @ 2011-05-19 12:02 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
This patch series looks really good to me in general. I know I've given
feedback about some pieces that need tweaking but overall its good and
I'm pleased with this. There are a few corner cases people have raised
and some tweaks we need to make but nothing that can't be built from
this foundation! :)
On Thu, 2011-05-19 at 18:29 +0800, Lianhao Lu wrote:
> To start the network based PR service daemon, run the following command after
> "sourcing" the environment file oe-init-build-env:
>
> bitbake-prserv --start
The one other area I'm going to worry about is the "build out the box"
use case. Nowhere in the quickstart guide do we mention to the user that
they need to run this. I'd also argue that we shouldn't require the user
to run this at all either and we should be able to autolaunch and use it
in the localhost usecase. Most users shouldn't care, it should just
work. Those who need to use it can either preconfigure it for their
users, or will know how to setup and start the server themselves.
A case in point is an autobuilder running 4 different builds. Ideally,
in the default case, each automated build would setup and just use its
own server on a non-conflicting free port on localhost.
I'm not 100% sure how we make this happen and I can give it some thought
but it is the final piece of this puzzle we need to solve.
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread