* [PATCH v4 0/5] prserv: add support for an "upstream" server
@ 2024-04-23 9:37 michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 1/5] prserv: declare "max_package_pr" client hook michael.opdenacker
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: michael.opdenacker @ 2024-04-23 9:37 UTC (permalink / raw)
To: bitbake-devel
Cc: Michael Opdenacker, Thomas Petazzoni, Joshua Watt, Tim Orling
From: Michael Opdenacker <michael.opdenacker@bootlin.com>
This makes it possible to customize an "upstream" distribution
by modifying local packages. If the "upstream" package bears
revision "x", the local one will have revision "x.y", this
having priority over the upstream one.
Multiple levels of upstream servers are supported, so "x.y.z" revisions
are possible too.
This also includes BitBake selftests to check the expected
functionality and detect future regressions.
Note that database import and export functions aren't tested yet.
---
Changes in V4:
- Add BitBake selftests for the legacy and new PR server features
(database, client, server, support for upstream server, read-only mode, "history" and "no history" modes)
To run only these tests:
bitbake-selftest prserv.tests
- Pass the "history" mode through the client requests, instead
of storing it (globally) in the database name.
The PR database is now called "PRMAIN" instead of "PRMAIN_nohist".
This should cause a regression for builds which already
have a PR database.
- Fixes for "history" modes:
- Allow to store multiple PR values for the same checksum,
needed for the "no history" mode.
- Make the "history" mode return the minimum stored
PR value.
- Fixes and code reorganization for issues uncovered by the tests.
- Update the server version to "2.0.0"
Changes in V3:
- Revert the commit removing the so far unused "hist" mode, which
we wish to keep for binary reproducibility sake.
- Simplification of get_value() function to take
both "hist" and "nohist" modes with the same shared code.
- Add "history" parameter to the "getPR" request,
so that the client can ask for the mode of its choice.
This will also make it possible to implement tests
for both modes.
Note that more requests ("export", "import"...)
will also need a "history" parameter, in a future version,
after the first tests are implemented.
- Several bug fixes.
- Put all the new features at the tip of the branch,
to make the cleanup commits easier to merged.
Changes in V2:
- Add this new commit:
prserv: remove unused "hist" mode in the database backend
- Squash commit "prserv: fix read_only test" into
commit "prserv: simplify the PRServerClient() interface"
(Reported by Richard Purdie)
- Fix the code to support increasing "x.y.z" values, thus
supporting several levels of upstream servers.
- db.py: remove duplicate definition of find_max_value() function in db.py
- prserv.py: remove tabs before comments (Python didn't complain)
- db.py: now stores the revision ("value") as TEXT.
This way we can store "1.0" without having it transformed to "1"
when the default type was INTEGER.
- This allows to fix a regression when the first packages were created
with 'r0.1' instead of 'r0.0' initially.
- find_max_value: now returns None instead of '0' when no value is found
Before we couldn't tell the difference between a '0'
max value and the absence of such a value.
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Michael Opdenacker (5):
prserv: declare "max_package_pr" client hook
prserv: move code from __init__ to bitbake-prserv
prserv: add "upstream" server support
prserv: sync the database after each change
prserv: add bitbake selftests
bin/bitbake-prserv | 26 ++-
bin/bitbake-selftest | 2 +
lib/prserv/__init__.py | 87 ++++++++--
lib/prserv/client.py | 19 +-
lib/prserv/db.py | 194 ++++++++++-----------
lib/prserv/serv.py | 137 ++++++++++++---
lib/prserv/tests.py | 381 +++++++++++++++++++++++++++++++++++++++++
7 files changed, 703 insertions(+), 143 deletions(-)
create mode 100644 bitbake/lib/prserv/tests.py
--
2.34.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v4 1/5] prserv: declare "max_package_pr" client hook
2024-04-23 9:37 [PATCH v4 0/5] prserv: add support for an "upstream" server michael.opdenacker
@ 2024-04-23 9:37 ` michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 2/5] prserv: move code from __init__ to bitbake-prserv michael.opdenacker
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: michael.opdenacker @ 2024-04-23 9:37 UTC (permalink / raw)
To: bitbake-devel
Cc: Michael Opdenacker, Joshua Watt, Tim Orling, Thomas Petazzoni
From: Michael Opdenacker <michael.opdenacker@bootlin.com>
Add missing declaration for the max_package_pr client hook
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
lib/prserv/client.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/prserv/client.py b/lib/prserv/client.py
index 8471ee3046..99fc4e0f7f 100644
--- a/lib/prserv/client.py
+++ b/lib/prserv/client.py
@@ -65,7 +65,7 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
class PRClient(bb.asyncrpc.Client):
def __init__(self):
super().__init__()
- self._add_methods("getPR", "test_pr", "test_package", "importone", "export", "is_readonly")
+ self._add_methods("getPR", "test_pr", "test_package", "max_package_pr", "importone", "export", "is_readonly")
def _get_async_client(self):
return PRAsyncClient()
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v4 2/5] prserv: move code from __init__ to bitbake-prserv
2024-04-23 9:37 [PATCH v4 0/5] prserv: add support for an "upstream" server michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 1/5] prserv: declare "max_package_pr" client hook michael.opdenacker
@ 2024-04-23 9:37 ` michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 3/5] prserv: add "upstream" server support michael.opdenacker
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: michael.opdenacker @ 2024-04-23 9:37 UTC (permalink / raw)
To: bitbake-devel
Cc: Michael Opdenacker, Joshua Watt, Tim Orling, Thomas Petazzoni
From: Michael Opdenacker <michael.opdenacker@bootlin.com>
This script was the only user of this code.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
bin/bitbake-prserv | 9 ++++++++-
lib/prserv/__init__.py | 13 -------------
2 files changed, 8 insertions(+), 14 deletions(-)
diff --git a/bin/bitbake-prserv b/bin/bitbake-prserv
index ad0a069401..920663a1d8 100755
--- a/bin/bitbake-prserv
+++ b/bin/bitbake-prserv
@@ -21,6 +21,13 @@ VERSION = "1.1.0"
PRHOST_DEFAULT="0.0.0.0"
PRPORT_DEFAULT=8585
+def init_logger(logfile, loglevel):
+ numeric_level = getattr(logging, loglevel.upper(), None)
+ if not isinstance(numeric_level, int):
+ raise ValueError("Invalid log level: %s" % loglevel)
+ FORMAT = "%(asctime)-15s %(message)s"
+ logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT)
+
def main():
parser = argparse.ArgumentParser(
description="BitBake PR Server. Version=%s" % VERSION,
@@ -72,7 +79,7 @@ def main():
)
args = parser.parse_args()
- prserv.init_logger(os.path.abspath(args.log), args.loglevel)
+ init_logger(os.path.abspath(args.log), args.loglevel)
if args.start:
ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only)
diff --git a/lib/prserv/__init__.py b/lib/prserv/__init__.py
index 0e0aa34d0e..94658b815d 100644
--- a/lib/prserv/__init__.py
+++ b/lib/prserv/__init__.py
@@ -5,16 +5,3 @@
#
__version__ = "1.0.0"
-
-import os, time
-import sys, logging
-
-def init_logger(logfile, loglevel):
- numeric_level = getattr(logging, loglevel.upper(), None)
- if not isinstance(numeric_level, int):
- raise ValueError("Invalid log level: %s" % loglevel)
- FORMAT = "%(asctime)-15s %(message)s"
- logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT)
-
-class NotFoundError(Exception):
- pass
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v4 3/5] prserv: add "upstream" server support
2024-04-23 9:37 [PATCH v4 0/5] prserv: add support for an "upstream" server michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 1/5] prserv: declare "max_package_pr" client hook michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 2/5] prserv: move code from __init__ to bitbake-prserv michael.opdenacker
@ 2024-04-23 9:37 ` michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 4/5] prserv: sync the database after each change michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 5/5] prserv: add bitbake selftests michael.opdenacker
4 siblings, 0 replies; 6+ messages in thread
From: michael.opdenacker @ 2024-04-23 9:37 UTC (permalink / raw)
To: bitbake-devel
Cc: Michael Opdenacker, Joshua Watt, Tim Orling, Thomas Petazzoni
From: Michael Opdenacker <michael.opdenacker@bootlin.com>
Introduce a PRSERVER_UPSTREAM variable that makes the
local PR server connect to an "upstream" one.
This makes it possible to implement local fixes to an
upstream package (revision "x", in a way that gives the local
update priority (revision "x.y").
Update the calculation of the new revisions to support the
case when prior revisions are not integers, but have
an "x.y..." format."
Set the comments in the handle_get_pr() function in serv.py
for details about the calculation of the local revision.
This is done by going on supporting the "history" mode that
wasn't used so far (revisions can return to a previous historical value),
in addition to the default "no history" mode (revisions can never decrease).
Rather than storing the history mode in the database table
itself (i.e. "PRMAIN_hist" and "PRMAIN_nohist"), the history mode
is now passed through the client requests. As a consequence, the
table name is now "PRMAIN", which is incompatible with what
was generated before, but avoids confusion if we kept the "PRMAIN_nohist"
name for both "history" and "no history" modes.
Update the server version to "2.0.0".
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
bin/bitbake-prserv | 17 +++-
lib/prserv/__init__.py | 80 +++++++++++++++++-
lib/prserv/client.py | 17 ++--
lib/prserv/db.py | 183 ++++++++++++++++++++---------------------
lib/prserv/serv.py | 134 ++++++++++++++++++++++++++----
5 files changed, 311 insertions(+), 120 deletions(-)
diff --git a/bin/bitbake-prserv b/bin/bitbake-prserv
index 920663a1d8..580e021fda 100755
--- a/bin/bitbake-prserv
+++ b/bin/bitbake-prserv
@@ -16,7 +16,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib
import prserv
import prserv.serv
-VERSION = "1.1.0"
+VERSION = "2.0.0"
PRHOST_DEFAULT="0.0.0.0"
PRPORT_DEFAULT=8585
@@ -77,12 +77,25 @@ def main():
action="store_true",
help="open database in read-only mode",
)
+ parser.add_argument(
+ "-u",
+ "--upstream",
+ default=os.environ.get("PRSERVER_UPSTREAM", None),
+ help="Upstream PR service (host:port)",
+ )
args = parser.parse_args()
init_logger(os.path.abspath(args.log), args.loglevel)
if args.start:
- ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only)
+ ret=prserv.serv.start_daemon(
+ args.file,
+ args.host,
+ args.port,
+ os.path.abspath(args.log),
+ args.read_only,
+ args.upstream
+ )
elif args.stop:
ret=prserv.serv.stop_daemon(args.host, args.port)
else:
diff --git a/lib/prserv/__init__.py b/lib/prserv/__init__.py
index 94658b815d..5b206cd8d6 100644
--- a/lib/prserv/__init__.py
+++ b/lib/prserv/__init__.py
@@ -4,4 +4,82 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-__version__ = "1.0.0"
+
+__version__ = "2.0.0"
+
+import logging
+logger = logging.getLogger("BitBake.PRserv")
+
+from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS
+
+def create_server(addr, dbpath, upstream=None, read_only=False):
+ from . import serv
+
+ s = serv.PRServer(dbpath, upstream=upstream, read_only=read_only)
+ host, port = addr.split(":")
+ s.start_tcp_server(host, int(port))
+
+ return s
+
+def increase_revision(ver):
+ """Take a revision string such as "1" or "1.2.3" or even a number and increase its last number
+ This fails if the last number is not an integer"""
+
+ fields=str(ver).split('.')
+ last = fields[-1]
+
+ try:
+ val = int(last)
+ except Exception as e:
+ logger.critical("Unable to increase revision value %s: %s" % (ver, e))
+ raise e
+
+ return ".".join(fields[0:-1] + list(str(val + 1)))
+
+def revision_greater_or_equal(rev1, rev2):
+ """Compares x.y.z revision numbers, using integer comparison
+ Returns True if rev1 is greater or equal rev2"""
+
+ fields1 = rev1.split(".")
+ fields2 = rev2.split(".")
+ l1 = len(fields1)
+ l2 = len(fields2)
+
+ for i in range(l1):
+ val1 = int(fields1[i])
+ if i < l2:
+ val2 = int(fields2[i])
+ if val2 < val1:
+ return True
+ elif val2 > val1:
+ return False
+ else:
+ return True
+ return True
+
+def create_client(addr):
+ from . import client
+
+ c = client.PRClient()
+
+ try:
+ (typ, a) = parse_address(addr)
+ c.connect_tcp(*a)
+ return c
+ except Exception as e:
+ c.close()
+ raise e
+
+async def create_async_client(addr):
+ from . import client
+
+ c = client.PRAsyncClient()
+
+ try:
+ (typ, a) = parse_address(addr)
+ await c.connect_tcp(*a)
+ return c
+
+ except Exception as e:
+ await c.close()
+ raise e
diff --git a/lib/prserv/client.py b/lib/prserv/client.py
index 99fc4e0f7f..565c6f3872 100644
--- a/lib/prserv/client.py
+++ b/lib/prserv/client.py
@@ -6,6 +6,7 @@
import logging
import bb.asyncrpc
+from . import create_async_client
logger = logging.getLogger("BitBake.PRserv")
@@ -13,16 +14,16 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
def __init__(self):
super().__init__("PRSERVICE", "1.0", logger)
- async def getPR(self, version, pkgarch, checksum):
+ async def getPR(self, version, pkgarch, checksum, history=False):
response = await self.invoke(
- {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}}
+ {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}}
)
if response:
return response["value"]
- async def test_pr(self, version, pkgarch, checksum):
+ async def test_pr(self, version, pkgarch, checksum, history=False):
response = await self.invoke(
- {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}}
+ {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}}
)
if response:
return response["value"]
@@ -41,16 +42,16 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
if response:
return response["value"]
- async def importone(self, version, pkgarch, checksum, value):
+ async def importone(self, version, pkgarch, checksum, value, history=False):
response = await self.invoke(
- {"import-one": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "value": value}}
+ {"import-one": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "value": value, "history": history}}
)
if response:
return response["value"]
- async def export(self, version, pkgarch, checksum, colinfo):
+ async def export(self, version, pkgarch, checksum, colinfo, history=False):
response = await self.invoke(
- {"export": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "colinfo": colinfo}}
+ {"export": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "colinfo": colinfo, "history": history}}
)
if response:
return (response["metainfo"], response["datainfo"])
diff --git a/lib/prserv/db.py b/lib/prserv/db.py
index eb41508198..8a613d106a 100644
--- a/lib/prserv/db.py
+++ b/lib/prserv/db.py
@@ -10,6 +10,8 @@ import errno
import prserv
import time
+from . import increase_revision
+
try:
import sqlite3
except ImportError:
@@ -32,15 +34,11 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
#
class PRTable(object):
- def __init__(self, conn, table, nohist, read_only):
+ def __init__(self, conn, table, read_only):
self.conn = conn
- self.nohist = nohist
self.read_only = read_only
self.dirty = False
- if nohist:
- self.table = "%s_nohist" % table
- else:
- self.table = "%s_hist" % table
+ self.table = table
if self.read_only:
table_exists = self._execute(
@@ -53,8 +51,8 @@ class PRTable(object):
(version TEXT NOT NULL, \
pkgarch TEXT NOT NULL, \
checksum TEXT NOT NULL, \
- value INTEGER, \
- PRIMARY KEY (version, pkgarch, checksum));" % self.table)
+ value TEXT, \
+ PRIMARY KEY (version, pkgarch, checksum, value));" % self.table)
def _execute(self, *query):
"""Execute a query, waiting to acquire a lock if necessary"""
@@ -102,101 +100,103 @@ class PRTable(object):
else:
return False
- def find_value(self, version, pkgarch, checksum):
- """Returns the value for the specified checksum if found or None otherwise."""
+ def find_package_max_value(self, version, pkgarch):
+ """Returns the greatest value for (version, pkgarch), or None if not found. Doesn't create a new value"""
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row=data.fetchone()
- if row is not None:
+ data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=?;" % (self.table),
+ (version, pkgarch))
+ row = data.fetchone()
+ # With SELECT max() requests, you have an empty row when there are no values, therefore the test on row[0]
+ if row is not None and row[0] is not None:
return row[0]
else:
return None
- def find_max_value(self, version, pkgarch):
- """Returns the greatest value for (version, pkgarch), or None if not found. Doesn't create a new value"""
+ def find_value(self, version, pkgarch, checksum, history=False):
+ """Returns the value for the specified checksum if found or None otherwise."""
- data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=?;" % (self.table),
- (version, pkgarch))
+ if history:
+ return self.find_min_value(version, pkgarch, checksum)
+ else:
+ return self.find_max_value(version, pkgarch, checksum)
+
+
+ def find_min_value(self, version, pkgarch, checksum):
+ """Returns the minimum value for (version, pkgarch, checksum), or None if not found. Doesn't create a new value"""
+
+ data = self._execute("SELECT min(value) FROM %s where version=? AND pkgarch=? AND checksum=?;" % (self.table),
+ (version, pkgarch, checksum))
row = data.fetchone()
- if row is not None:
+ # With SELECT min() requests, you may have an empty row when there are no values, therefore the test on row[0]
+ if row is not None and row[0] is not None:
return row[0]
else:
return None
- def _get_value_hist(self, version, pkgarch, checksum):
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row=data.fetchone()
- if row is not None:
+ def find_max_value(self, version, pkgarch, checksum):
+ """Returns the max value for (version, pkgarch, checksum), or None if not found. Doesn't create a new value"""
+
+ data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=? AND checksum=?;" % (self.table),
+ (version, pkgarch, checksum))
+ row = data.fetchone()
+ # With SELECT max() requests, you may have an empty row when there are no values, therefore the test on row[0]
+ if row is not None and row[0] is not None:
return row[0]
else:
- #no value found, try to insert
- if self.read_only:
- data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table),
- (version, pkgarch))
- row = data.fetchone()
- if row is not None:
- return row[0]
- else:
- return 0
+ return None
- try:
- self._execute("INSERT INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));"
- % (self.table, self.table),
- (version, pkgarch, checksum, version, pkgarch))
- except sqlite3.IntegrityError as exc:
- logger.error(str(exc))
+ def find_new_subvalue(self, version, pkgarch, base):
+ """Take and increase the greatest "<base>.y" value for (version, pkgarch), or return "<base>.0" if not found.
+ This doesn't store a new value."""
- self.dirty = True
+ data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=? AND value LIKE '%s.%%';" % (self.table, base),
+ (version, pkgarch))
+ row = data.fetchone()
+ # With SELECT max() requests, you have an empty row when there are no values, therefore the test on row[0]
+ if row is not None and row[0] is not None:
+ return increase_revision(row[0])
+ else:
+ return base + ".0"
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row=data.fetchone()
- if row is not None:
- return row[0]
- else:
- raise prserv.NotFoundError
+ def store_value(self, version, pkgarch, checksum, value):
+ """Store new value in the database"""
- def _get_value_no_hist(self, version, pkgarch, checksum):
- data=self._execute("SELECT value FROM %s \
- WHERE version=? AND pkgarch=? AND checksum=? AND \
- value >= (select max(value) from %s where version=? AND pkgarch=?);"
- % (self.table, self.table),
- (version, pkgarch, checksum, version, pkgarch))
- row=data.fetchone()
- if row is not None:
- return row[0]
- else:
- #no value found, try to insert
- if self.read_only:
- data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table),
- (version, pkgarch))
- return data.fetchone()[0]
+ try:
+ self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table),
+ (version, pkgarch, checksum, value))
+ except sqlite3.IntegrityError as exc:
+ logger.error(str(exc))
- try:
- self._execute("INSERT OR REPLACE INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));"
- % (self.table, self.table),
- (version, pkgarch, checksum, version, pkgarch))
- except sqlite3.IntegrityError as exc:
- logger.error(str(exc))
- self.conn.rollback()
+ self.dirty = True
- self.dirty = True
+ def _get_value(self, version, pkgarch, checksum, history):
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row=data.fetchone()
- if row is not None:
- return row[0]
- else:
- raise prserv.NotFoundError
+ max_value = self.find_package_max_value(version, pkgarch)
+
+ if max_value is None:
+ # version, pkgarch completely unknown. Return initial value.
+ return "0"
+
+ value = self.find_value(version, pkgarch, checksum, history)
- def get_value(self, version, pkgarch, checksum):
- if self.nohist:
- return self._get_value_no_hist(version, pkgarch, checksum)
+ if value is None:
+ # version, pkgarch found but not checksum. Create a new value from the maximum one
+ return increase_revision(max_value)
+
+ if history:
+ return value
+
+ # "no history" mode - If the value is not the maximum value for the package, need to increase it.
+ if max_value > value:
+ return increase_revision(max_value)
else:
- return self._get_value_hist(version, pkgarch, checksum)
+ return value
+
+ def get_value(self, version, pkgarch, checksum, history):
+ value = self._get_value(version, pkgarch, checksum, history)
+ if not self.read_only:
+ self.store_value(version, pkgarch, checksum, value)
+ return value
def _import_hist(self, version, pkgarch, checksum, value):
if self.read_only:
@@ -252,13 +252,13 @@ class PRTable(object):
else:
return None
- def importone(self, version, pkgarch, checksum, value):
- if self.nohist:
- return self._import_no_hist(version, pkgarch, checksum, value)
- else:
+ def importone(self, version, pkgarch, checksum, value, history=False):
+ if history:
return self._import_hist(version, pkgarch, checksum, value)
+ else:
+ return self._import_no_hist(version, pkgarch, checksum, value)
- def export(self, version, pkgarch, checksum, colinfo):
+ def export(self, version, pkgarch, checksum, colinfo, history=False):
metainfo = {}
#column info
if colinfo:
@@ -278,12 +278,12 @@ class PRTable(object):
#data info
datainfo = []
- if self.nohist:
+ if history:
+ sqlstmt = "SELECT * FROM %s as T1 WHERE 1=1 " % self.table
+ else:
sqlstmt = "SELECT T1.version, T1.pkgarch, T1.checksum, T1.value FROM %s as T1, \
(SELECT version, pkgarch, max(value) as maxvalue FROM %s GROUP BY version, pkgarch) as T2 \
WHERE T1.version=T2.version AND T1.pkgarch=T2.pkgarch AND T1.value=T2.maxvalue " % (self.table, self.table)
- else:
- sqlstmt = "SELECT * FROM %s as T1 WHERE 1=1 " % self.table
sqlarg = []
where = ""
if version:
@@ -322,9 +322,8 @@ class PRTable(object):
class PRData(object):
"""Object representing the PR database"""
- def __init__(self, filename, nohist=True, read_only=False):
+ def __init__(self, filename, read_only=False):
self.filename=os.path.abspath(filename)
- self.nohist=nohist
self.read_only = read_only
#build directory hierarchy
try:
@@ -351,7 +350,7 @@ class PRData(object):
if tblname in self._tables:
return self._tables[tblname]
else:
- tableobj = self._tables[tblname] = PRTable(self.connection, tblname, self.nohist, self.read_only)
+ tableobj = self._tables[tblname] = PRTable(self.connection, tblname, self.read_only)
return tableobj
def __delitem__(self, tblname):
diff --git a/lib/prserv/serv.py b/lib/prserv/serv.py
index dc4be5b620..b4447912a8 100644
--- a/lib/prserv/serv.py
+++ b/lib/prserv/serv.py
@@ -12,6 +12,7 @@ import sqlite3
import prserv
import prserv.db
import errno
+from . import create_async_client, revision_greater_or_equal, increase_revision
import bb.asyncrpc
logger = logging.getLogger("BitBake.PRserv")
@@ -51,8 +52,9 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
version = request["version"]
pkgarch = request["pkgarch"]
checksum = request["checksum"]
+ history = request["history"]
- value = self.server.table.find_value(version, pkgarch, checksum)
+ value = self.server.table.find_value(version, pkgarch, checksum, history)
return {"value": value}
async def handle_test_package(self, request):
@@ -68,22 +70,110 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
version = request["version"]
pkgarch = request["pkgarch"]
- value = self.server.table.find_max_value(version, pkgarch)
+ value = self.server.table.find_package_max_value(version, pkgarch)
return {"value": value}
async def handle_get_pr(self, request):
version = request["version"]
pkgarch = request["pkgarch"]
checksum = request["checksum"]
+ history = request["history"]
- response = None
- try:
- value = self.server.table.get_value(version, pkgarch, checksum)
- response = {"value": value}
- except prserv.NotFoundError:
- self.logger.error("failure storing value in database for (%s, %s)",version, checksum)
+ if self.upstream_client is None:
+ value = self.server.table.get_value(version, pkgarch, checksum, history)
+ return {"value": value}
- return response
+ # We have an upstream server.
+ # Check whether the local server already knows the requested configuration.
+ # If the configuration is a new one, the generated value we will add will
+ # depend on what's on the upstream server. That's why we're calling find_value()
+ # instead of get_value() directly.
+
+ value = self.server.table.find_value(version, pkgarch, checksum, history)
+ upstream_max = await self.upstream_client.max_package_pr(version, pkgarch)
+
+ if value is not None:
+
+ # The configuration is already known locally.
+
+ if history:
+ value = self.server.table.get_value(version, pkgarch, checksum, history)
+ else:
+ existing_value = value
+ # In "no history", we need to make sure the value doesn't decrease
+ # and is at least greater than the maximum upstream value
+ # and the maximum local value
+
+ local_max = self.server.table.find_package_max_value(version, pkgarch)
+ if not revision_greater_or_equal(value, local_max):
+ value = increase_revision(local_max)
+
+ if not revision_greater_or_equal(value, upstream_max):
+ # Ask upstream whether it knows the checksum
+ upstream_value = await self.upstream_client.test_pr(version, pkgarch, checksum)
+ if upstream_value is None:
+ # Upstream doesn't have our checksum, let create a new one
+ value = upstream_max + ".0"
+ else:
+ # Fine to take the same value as upstream
+ value = upstream_max
+
+ if not value == existing_value and not self.server.read_only:
+ self.server.table.store_value(version, pkgarch, checksum, value)
+
+ return {"value": value}
+
+ # The configuration is a new one for the local server
+ # Let's ask the upstream server whether it knows it
+
+ known_upstream = await self.upstream_client.test_package(version, pkgarch)
+
+ if not known_upstream:
+
+ # The package is not known upstream, must be a local-only package
+ # Let's compute the PR number using the local-only method
+
+ value = self.server.table.get_value(version, pkgarch, checksum, history)
+ return {"value": value}
+
+ # The package is known upstream, let's ask the upstream server
+ # whether it knows our new output hash
+
+ value = await self.upstream_client.test_pr(version, pkgarch, checksum)
+
+ if value is not None:
+
+ # Upstream knows this output hash, let's store it and use it too.
+
+ if not self.server.read_only:
+ self.server.table.store_value(version, pkgarch, checksum, value)
+ # If the local server is read only, won't be able to store the new
+ # value in the database and will have to keep asking the upstream server
+ return {"value": value}
+
+ # The output hash doesn't exist upstream, get the most recent number from upstream (x)
+ # Then, we want to have a new PR value for the local server: x.y
+
+ upstream_max = await self.upstream_client.max_package_pr(version, pkgarch)
+ # Here we know that the package is known upstream, so upstream_max can't be None
+ subvalue = self.server.table.find_new_subvalue(version, pkgarch, upstream_max)
+
+ if not self.server.read_only:
+ self.server.table.store_value(version, pkgarch, checksum, subvalue)
+
+ return {"value": subvalue}
+
+ async def process_requests(self):
+ if self.server.upstream is not None:
+ self.upstream_client = await create_async_client(self.server.upstream)
+ else:
+ self.upstream_client = None
+
+ try:
+ await super().process_requests()
+ finally:
+ if self.upstream_client is not None:
+ await self.upstream_client.close()
async def handle_import_one(self, request):
response = None
@@ -92,8 +182,9 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
pkgarch = request["pkgarch"]
checksum = request["checksum"]
value = request["value"]
+ history = request["history"]
- value = self.server.table.importone(version, pkgarch, checksum, value)
+ value = self.server.table.importone(version, pkgarch, checksum, value, history)
if value is not None:
response = {"value": value}
@@ -104,9 +195,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
pkgarch = request["pkgarch"]
checksum = request["checksum"]
colinfo = request["colinfo"]
+ history = request["history"]
try:
- (metainfo, datainfo) = self.server.table.export(version, pkgarch, checksum, colinfo)
+ (metainfo, datainfo) = self.server.table.export(version, pkgarch, checksum, colinfo, history)
except sqlite3.Error as exc:
self.logger.error(str(exc))
metainfo = datainfo = None
@@ -117,11 +209,12 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
return {"readonly": self.server.read_only}
class PRServer(bb.asyncrpc.AsyncServer):
- def __init__(self, dbfile, read_only=False):
+ def __init__(self, dbfile, read_only=False, upstream=None):
super().__init__(logger)
self.dbfile = dbfile
self.table = None
self.read_only = read_only
+ self.upstream = upstream
def accept_client(self, socket):
return PRServerClient(socket, self)
@@ -134,6 +227,9 @@ class PRServer(bb.asyncrpc.AsyncServer):
self.logger.info("Started PRServer with DBfile: %s, Address: %s, PID: %s" %
(self.dbfile, self.address, str(os.getpid())))
+ if self.upstream is not None:
+ self.logger.info("And upstream PRServer: %s " % (self.upstream))
+
return tasks
async def stop(self):
@@ -147,14 +243,15 @@ class PRServer(bb.asyncrpc.AsyncServer):
self.table.sync()
class PRServSingleton(object):
- def __init__(self, dbfile, logfile, host, port):
+ def __init__(self, dbfile, logfile, host, port, upstream):
self.dbfile = dbfile
self.logfile = logfile
self.host = host
self.port = port
+ self.upstream = upstream
def start(self):
- self.prserv = PRServer(self.dbfile)
+ self.prserv = PRServer(self.dbfile, upstream=self.upstream)
self.prserv.start_tcp_server(socket.gethostbyname(self.host), self.port)
self.process = self.prserv.serve_as_process(log_level=logging.WARNING)
@@ -233,7 +330,7 @@ def run_as_daemon(func, pidfile, logfile):
os.remove(pidfile)
os._exit(0)
-def start_daemon(dbfile, host, port, logfile, read_only=False):
+def start_daemon(dbfile, host, port, logfile, read_only=False, upstream=None):
ip = socket.gethostbyname(host)
pidfile = PIDPREFIX % (ip, port)
try:
@@ -249,7 +346,7 @@ def start_daemon(dbfile, host, port, logfile, read_only=False):
dbfile = os.path.abspath(dbfile)
def daemon_main():
- server = PRServer(dbfile, read_only=read_only)
+ server = PRServer(dbfile, read_only=read_only, upstream=upstream)
server.start_tcp_server(ip, port)
server.serve_forever()
@@ -336,6 +433,9 @@ def auto_start(d):
host = host_params[0].strip().lower()
port = int(host_params[1])
+
+ upstream = d.getVar("PRSERV_UPSTREAM") or None
+
if is_local_special(host, port):
import bb.utils
cachedir = (d.getVar("PERSISTENT_DIR") or d.getVar("CACHE"))
@@ -350,7 +450,7 @@ def auto_start(d):
auto_shutdown()
if not singleton:
bb.utils.mkdirhier(cachedir)
- singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port)
+ singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port, upstream)
singleton.start()
if singleton:
host = singleton.host
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v4 4/5] prserv: sync the database after each change
2024-04-23 9:37 [PATCH v4 0/5] prserv: add support for an "upstream" server michael.opdenacker
` (2 preceding siblings ...)
2024-04-23 9:37 ` [PATCH v4 3/5] prserv: add "upstream" server support michael.opdenacker
@ 2024-04-23 9:37 ` michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 5/5] prserv: add bitbake selftests michael.opdenacker
4 siblings, 0 replies; 6+ messages in thread
From: michael.opdenacker @ 2024-04-23 9:37 UTC (permalink / raw)
To: bitbake-devel
Cc: Michael Opdenacker, Joshua Watt, Tim Orling, Thomas Petazzoni
From: Michael Opdenacker <michael.opdenacker@bootlin.com>
This removes the need for a "dirty" flag and simplifies the code.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
lib/prserv/db.py | 13 ++++---------
lib/prserv/serv.py | 3 ---
2 files changed, 4 insertions(+), 12 deletions(-)
diff --git a/lib/prserv/db.py b/lib/prserv/db.py
index 8a613d106a..598f015138 100644
--- a/lib/prserv/db.py
+++ b/lib/prserv/db.py
@@ -37,7 +37,6 @@ class PRTable(object):
def __init__(self, conn, table, read_only):
self.conn = conn
self.read_only = read_only
- self.dirty = False
self.table = table
if self.read_only:
@@ -53,6 +52,7 @@ class PRTable(object):
checksum TEXT NOT NULL, \
value TEXT, \
PRIMARY KEY (version, pkgarch, checksum, value));" % self.table)
+ self.sync()
def _execute(self, *query):
"""Execute a query, waiting to acquire a lock if necessary"""
@@ -71,11 +71,6 @@ class PRTable(object):
self.conn.commit()
self._execute("BEGIN EXCLUSIVE TRANSACTION")
- def sync_if_dirty(self):
- if self.dirty:
- self.sync()
- self.dirty = False
-
def test_package(self, version, pkgarch):
"""Returns whether the specified package version is found in the database for the specified architecture"""
@@ -167,7 +162,7 @@ class PRTable(object):
except sqlite3.IntegrityError as exc:
logger.error(str(exc))
- self.dirty = True
+ self.sync()
def _get_value(self, version, pkgarch, checksum, history):
@@ -216,7 +211,7 @@ class PRTable(object):
except sqlite3.IntegrityError as exc:
logger.error(str(exc))
- self.dirty = True
+ self.sync()
data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
(version, pkgarch, checksum))
@@ -242,7 +237,7 @@ class PRTable(object):
except sqlite3.IntegrityError as exc:
logger.error(str(exc))
- self.dirty = True
+ self.sync()
data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=? AND value>=?;" % self.table,
(version, pkgarch, checksum, value))
diff --git a/lib/prserv/serv.py b/lib/prserv/serv.py
index b4447912a8..b39957f936 100644
--- a/lib/prserv/serv.py
+++ b/lib/prserv/serv.py
@@ -44,8 +44,6 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
except:
self.server.table.sync()
raise
- else:
- self.server.table.sync_if_dirty()
async def handle_test_pr(self, request):
'''Finds the PR value corresponding to the request. If not found, returns None and doesn't insert a new value'''
@@ -233,7 +231,6 @@ class PRServer(bb.asyncrpc.AsyncServer):
return tasks
async def stop(self):
- self.table.sync_if_dirty()
self.db.disconnect()
await super().stop()
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v4 5/5] prserv: add bitbake selftests
2024-04-23 9:37 [PATCH v4 0/5] prserv: add support for an "upstream" server michael.opdenacker
` (3 preceding siblings ...)
2024-04-23 9:37 ` [PATCH v4 4/5] prserv: sync the database after each change michael.opdenacker
@ 2024-04-23 9:37 ` michael.opdenacker
4 siblings, 0 replies; 6+ messages in thread
From: michael.opdenacker @ 2024-04-23 9:37 UTC (permalink / raw)
To: bitbake-devel
Cc: Michael Opdenacker, Joshua Watt, Tim Orling, Thomas Petazzoni
From: Michael Opdenacker <michael.opdenacker@bootlin.com>
Run them with "bitbake-selftest prserv.tests"
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
bin/bitbake-selftest | 2 +
lib/prserv/tests.py | 381 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 383 insertions(+)
create mode 100644 bitbake/lib/prserv/tests.py
diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
index f25f23b1ae..ce901232fe 100755
--- a/bin/bitbake-selftest
+++ b/bin/bitbake-selftest
@@ -15,6 +15,7 @@ import unittest
try:
import bb
import hashserv
+ import prserv
import layerindexlib
except RuntimeError as exc:
sys.exit(str(exc))
@@ -33,6 +34,7 @@ tests = ["bb.tests.codeparser",
"bb.tests.utils",
"bb.tests.compression",
"hashserv.tests",
+ "prserv.tests",
"layerindexlib.tests.layerindexobj",
"layerindexlib.tests.restapi",
"layerindexlib.tests.cooker"]
diff --git a/lib/prserv/tests.py b/lib/prserv/tests.py
new file mode 100644
index 0000000000..d133c239f8
--- /dev/null
+++ b/lib/prserv/tests.py
@@ -0,0 +1,381 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2024 BitBake Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from . import create_server, create_client, increase_revision, revision_greater_or_equal
+import prserv.db as db
+from bb.asyncrpc import InvokeError
+import logging
+import os
+import sys
+import tempfile
+import unittest
+import socket
+import subprocess
+from pathlib import Path
+
+THIS_DIR = Path(__file__).parent
+BIN_DIR = THIS_DIR.parent.parent / "bin"
+
+version = "dummy-1.0-r0"
+pkgarch = "core2-64"
+other_arch = "aarch64"
+
+checksumX = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4f0"
+checksum0 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a0"
+checksum1 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a1"
+checksum2 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a2"
+checksum3 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a3"
+checksum4 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a4"
+checksum5 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a5"
+checksum6 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a6"
+checksum7 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a7"
+checksum8 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a8"
+checksum9 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a9"
+
+def server_prefunc(server, name):
+ logging.basicConfig(level=logging.DEBUG, filename='prserv-%s.log' % name, filemode='w',
+ format='%(levelname)s %(filename)s:%(lineno)d %(message)s')
+ server.logger.debug("Running server %s" % name)
+ sys.stdout = open('prserv-stdout-%s.log' % name, 'w')
+ sys.stderr = sys.stdout
+
+class PRTestSetup(object):
+
+ def start_server(self, name, dbfile, upstream=None, read_only=False, prefunc=server_prefunc):
+
+ def cleanup_server(server):
+ if server.process.exitcode is not None:
+ return
+ server.process.terminate()
+ server.process.join()
+
+ server = create_server(socket.gethostbyname("localhost") + ":0",
+ dbfile,
+ upstream=upstream,
+ read_only=read_only)
+
+ server.serve_as_process(prefunc=prefunc, args=(name,))
+ self.addCleanup(cleanup_server, server)
+
+ return server
+
+ def start_client(self, server_address):
+ def cleanup_client(client):
+ client.close()
+
+ client = create_client(server_address)
+ self.addCleanup(cleanup_client, client)
+
+ return client
+
+class FunctionTests(unittest.TestCase):
+
+ def test_0a_increase_revision(self):
+ self.assertEqual(increase_revision("1"), "2")
+ self.assertEqual(increase_revision("1.0"), "1.1")
+ self.assertEqual(increase_revision("1.1.1"), "1.1.2")
+ self.assertEqual(increase_revision("1.1.1.3"), "1.1.1.4")
+ self.assertRaises(ValueError, increase_revision, "1.a")
+ self.assertRaises(ValueError, increase_revision, "1.")
+ self.assertRaises(ValueError, increase_revision, "")
+
+ def test_0b_revision_greater_or_equal(self):
+ self.assertTrue(revision_greater_or_equal("2", "2"))
+ self.assertTrue(revision_greater_or_equal("2", "1"))
+ self.assertTrue(revision_greater_or_equal("10", "2"))
+ self.assertTrue(revision_greater_or_equal("1.10", "1.2"))
+ self.assertFalse(revision_greater_or_equal("1.2", "1.10"))
+ self.assertTrue(revision_greater_or_equal("1.10", "1"))
+ self.assertTrue(revision_greater_or_equal("1.10.1", "1.10"))
+ self.assertFalse(revision_greater_or_equal("1.10.1", "1.10.2"))
+ self.assertTrue(revision_greater_or_equal("1.10.1", "1.10.1"))
+ self.assertTrue(revision_greater_or_equal("1.10.1", "1"))
+
+ # DB tests
+
+ def test_0b_db(self):
+ dbfile = "testtable.sqlite3"
+ if os.path.exists(dbfile):
+ os.remove(dbfile)
+
+ self.db = db.PRData(dbfile)
+ self.table = self.db["PRMAIN"]
+
+ self.table.store_value(version, pkgarch, checksum0, "0")
+ self.table.store_value(version, pkgarch, checksum1, "1")
+ # "No history" mode supports multiple PRs for the same checksum
+ self.table.store_value(version, pkgarch, checksum0, "2")
+ self.table.store_value(version, pkgarch, checksum2, "1.0")
+
+ self.assertTrue(self.table.test_package(version, pkgarch))
+ self.assertFalse(self.table.test_package(version, other_arch))
+
+ self.assertTrue(self.table.test_value(version, pkgarch, "0"))
+ self.assertTrue(self.table.test_value(version, pkgarch, "1"))
+ self.assertTrue(self.table.test_value(version, pkgarch, "2"))
+
+ self.assertEqual(self.table.find_package_max_value(version, pkgarch), "2")
+
+ self.assertEqual(self.table.find_min_value(version, pkgarch, checksum0), "0")
+ self.assertEqual(self.table.find_max_value(version, pkgarch, checksum0), "2")
+
+ # Test history modes
+ self.assertEqual(self.table.find_value(version, pkgarch, checksum0, True), "0")
+ self.assertEqual(self.table.find_value(version, pkgarch, checksum0, False), "2")
+
+ self.assertEqual(self.table.find_new_subvalue(version, pkgarch, "3"), "3.0")
+ self.assertEqual(self.table.find_new_subvalue(version, pkgarch, "1"), "1.1")
+
+class PRBasicTests(PRTestSetup, unittest.TestCase):
+
+ def setUp(self):
+ dbfile = "prtest-basic.sqlite3"
+ if os.path.exists(dbfile):
+ os.remove(dbfile)
+
+ self.server1 = self.start_server("basic", dbfile)
+ self.client1 = self.start_client(self.server1.address)
+
+ def test_1a_basic(self):
+
+ # Checks on non existing configuration
+
+ result = self.client1.test_pr(version, pkgarch, checksum0)
+ self.assertIsNone(result, "test_pr should return 'None' for a non existing PR")
+
+ result = self.client1.test_package(version, pkgarch)
+ self.assertFalse(result, "test_package should return 'False' for a non existing PR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertIsNone(result, "max_package_pr should return 'None' for a non existing PR")
+
+ # Add a first configuration
+
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'")
+
+ result = self.client1.test_pr(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "test_pr should return '0' here, matching the result of getPR")
+
+ result = self.client1.test_package(version, pkgarch)
+ self.assertTrue(result, "test_package should return 'True' for an existing PR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "0", "max_package_pr should return '0' in the current test series")
+
+ # Check that the same request gets the same value
+
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: asking for the same PR a second time in a row should return the same value.")
+
+ # Add new configurations
+
+ result = self.client1.getPR(version, pkgarch, checksum1)
+ self.assertEqual(result, "1", "getPR: second PR of a package should be '1'")
+
+ result = self.client1.test_pr(version, pkgarch, checksum1)
+ self.assertEqual(result, "1", "test_pr should return '1' here, matching the result of getPR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "1", "max_package_pr should return '1' in the current test series")
+
+ result = self.client1.getPR(version, pkgarch, checksum2)
+ self.assertEqual(result, "2", "getPR: second PR of a package should be '2'")
+
+ result = self.client1.test_pr(version, pkgarch, checksum2)
+ self.assertEqual(result, "2", "test_pr should return '2' here, matching the result of getPR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "2", "max_package_pr should return '2' in the current test series")
+
+ result = self.client1.getPR(version, pkgarch, checksum3)
+ self.assertEqual(result, "3", "getPR: second PR of a package should be '3'")
+
+ result = self.client1.test_pr(version, pkgarch, checksum3)
+ self.assertEqual(result, "3", "test_pr should return '3' here, matching the result of getPR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "3", "max_package_pr should return '3' in the current test series")
+
+ # Ask again for the first configuration
+
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "4", "getPR: should return '4' in this configuration")
+
+ # Ask again with explicit "no history" mode
+
+ result = self.client1.getPR(version, pkgarch, checksum0, False)
+ self.assertEqual(result, "4", "getPR: should return '4' in this configuration")
+
+ # Ask again with explicit "history" mode. This should return the first recorded PR for checksum0
+
+ result = self.client1.getPR(version, pkgarch, checksum0, True)
+ self.assertEqual(result, "0", "getPR: should return '0' in this configuration")
+
+ # Check again that another pkgarg resets the counters
+
+ result = self.client1.test_pr(version, other_arch, checksum0)
+ self.assertIsNone(result, "test_pr should return 'None' for a non existing PR")
+
+ result = self.client1.test_package(version, other_arch)
+ self.assertFalse(result, "test_package should return 'False' for a non existing PR")
+
+ result = self.client1.max_package_pr(version, other_arch)
+ self.assertIsNone(result, "max_package_pr should return 'None' for a non existing PR")
+
+ # Now add the configuration
+
+ result = self.client1.getPR(version, other_arch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'")
+
+ result = self.client1.test_pr(version, other_arch, checksum0)
+ self.assertEqual(result, "0", "test_pr should return '0' here, matching the result of getPR")
+
+ result = self.client1.test_package(version, other_arch)
+ self.assertTrue(result, "test_package should return 'True' for an existing PR")
+
+ result = self.client1.max_package_pr(version, other_arch)
+ self.assertEqual(result, "0", "max_package_pr should return '0' in the current test series")
+
+ def test_1b_is_readonly(self):
+ result = self.client1.is_readonly()
+ self.assertFalse(result, "Server should not be described as 'read-only'")
+
+class PRReadOnlyTests(PRTestSetup, unittest.TestCase):
+
+ def setUp(self):
+ # Here we're using the database test table, as for reasons not understood yet,
+ # the other tables we created from the PR server in prior tests, have empty contents!
+ dbfile = "testtable.sqlite3"
+ self.server1 = self.start_server("basic-readonly", dbfile, read_only=True)
+ self.client1 = self.start_client(self.server1.address)
+
+ def test_2a_read_only_server(self):
+
+ result = self.client1.is_readonly()
+ self.assertTrue(result, "Database should be described as 'read-only'")
+
+ # Checks on non existing configuration
+ self.assertIsNone(self.client1.test_pr(version, pkgarch, checksumX))
+ self.assertFalse(self.client1.test_package("unknown", pkgarch))
+
+ # Look up an existing configuration
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum0), "2") # "no history" mode
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum0, True), "0") # "history" mode
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum2), "3")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum2, True), "1.0")
+ self.assertEqual(self.client1.max_package_pr(version, pkgarch), "2")
+
+ # Try to insert a new value. The value should be correct given the read-only dababase, but won't be saved
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum3), "3")
+ # Same of another value
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum3), "3")
+
+class PRUpstreamTests(PRTestSetup, unittest.TestCase):
+
+ def setUp(self):
+
+ dbfile2 = "prtest-upstream2.sqlite3"
+ if os.path.exists(dbfile2):
+ os.remove(dbfile2)
+
+ self.server2 = self.start_server("upstream2", dbfile2)
+ self.client2 = self.start_client(self.server2.address)
+
+ dbfile1 = "prtest-upstream1.sqlite3"
+ if os.path.exists(dbfile1):
+ os.remove(dbfile1)
+
+ self.server1 = self.start_server("upstream1", dbfile1, upstream=self.server2.address)
+ self.client1 = self.start_client(self.server1.address)
+
+ dbfile0 = "prtest-local.sqlite3"
+ if os.path.exists(dbfile0):
+ os.remove(dbfile0)
+
+ self.server0 = self.start_server("local", dbfile0, upstream=self.server1.address)
+ self.client0 = self.start_client(self.server0.address)
+
+ def test_3a_upstream(self):
+
+ # For identical checksums, all servers should return the same PR
+
+ result = self.client2.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'")
+
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0' (same as upstream)")
+
+ result = self.client0.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0' (same as upstream)")
+
+ # Now introduce new checksums on server1 for, same version
+
+ result = self.client1.getPR(version, pkgarch, checksum1)
+ self.assertEqual(result, "0.0", "getPR: first PR of a package which has a different checksum upstream should be '0.0'")
+
+ result = self.client1.getPR(version, pkgarch, checksum2)
+ self.assertEqual(result, "0.1", "getPR: second PR of a package that has a different checksum upstream should be '0.1'")
+
+ # Now introduce checksums on server0 for, same version
+
+ result = self.client1.getPR(version, pkgarch, checksum1)
+ self.assertEqual(result, "0.2", "getPR: can't decrease for known PR")
+
+ result = self.client1.getPR(version, pkgarch, checksum2)
+ self.assertEqual(result, "0.3")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "0.3")
+
+ result = self.client0.getPR(version, pkgarch, checksum3)
+ self.assertEqual(result, "0.3.0", "getPR: first PR of a package that doesn't exist upstream should be '0.3.0'")
+
+ result = self.client0.getPR(version, pkgarch, checksum4)
+ self.assertEqual(result, "0.3.1", "getPR: second PR of a package that doesn't exist upstream should be '0.3.1'")
+
+ result = self.client0.getPR(version, pkgarch, checksum3)
+ self.assertEqual(result, "0.3.2")
+
+ # More upstream updates
+ # Here, we assume no communication between server2 and server0. server2 only impacts server0
+ # after impacting server1
+
+ self.assertEqual(self.client2.getPR(version, pkgarch, checksum5), "1")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum6), "1.0")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum7), "1.1")
+ self.assertEqual(self.client0.getPR(version, pkgarch, checksum8), "1.1.0")
+ self.assertEqual(self.client0.getPR(version, pkgarch, checksum9), "1.1.1")
+
+ # "history" mode tests
+
+ self.assertEqual(self.client2.getPR(version, pkgarch, checksum0, True), "0")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum2, True), "0.1")
+ self.assertEqual(self.client0.getPR(version, pkgarch, checksum3, True), "0.3.0")
+
+ # More "no history" mode tests
+
+ self.assertEqual(self.client2.getPR(version, pkgarch, checksum0), "2")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum0), "2") # Same as upstream
+ self.assertEqual(self.client0.getPR(version, pkgarch, checksum0), "2") # Same as upstream
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum7), "3") # This could be surprising, but since the previous revision was "2", increasing it yields "3".
+ # We don't know how many upstream servers we have
+
+class ScriptTests(unittest.TestCase):
+
+ def test_4a_start_bitbake_prserv(self):
+ try:
+ subprocess.check_call([BIN_DIR / "bitbake-prserv", "--start", "-f", "testtable.sqlite3"])
+ except subprocess.CalledProcessError as e:
+ self.fail("Failed to start bitbake-prserv: %s" % e.returncode)
+
+ def test_4b_stop_bitbake_prserv(self):
+ try:
+ subprocess.check_call([BIN_DIR / "bitbake-prserv", "--stop"])
+ except subprocess.CalledProcessError as e:
+ self.fail("Failed to stop bitbake-prserv: %s" % e.returncode)
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-04-23 9:38 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-23 9:37 [PATCH v4 0/5] prserv: add support for an "upstream" server michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 1/5] prserv: declare "max_package_pr" client hook michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 2/5] prserv: move code from __init__ to bitbake-prserv michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 3/5] prserv: add "upstream" server support michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 4/5] prserv: sync the database after each change michael.opdenacker
2024-04-23 9:37 ` [PATCH v4 5/5] prserv: add bitbake selftests michael.opdenacker
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.