* [PATCH 0/6] toaster patchset: localhost build control
@ 2014-06-12 13:51 ` Alex DAMIAN
0 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:51 UTC (permalink / raw)
To: bitbake-devel, toaster
From: Alexandru DAMIAN <alexandru.damian@intel.com>
Hello,
This toaster patchset contains the core of the build control functionality
for running on a localhost backend. Variants of this patchset have
been reviewed on the toaster mailing list.
Please merge at your convenience,
Alex
The following changes since commit 2e742c03e8dfdfa67899e7f5d579ed14bd87e139:
bb/utils: fix contains_any() (2014-06-11 13:18:20 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib adamian/20140612-submission-bb
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/20140612-submission-bb
Alexandru DAMIAN (6):
toaster: initial bldcontrol application
toaster: create models for bldcontrol and enable it
toaster: add function to get the database url
xmlrpc: add support for token reusing
toastergui: fix built ETA calculation
toaster: build control functionality
bin/bitbake | 8 +-
bin/toaster | 9 +-
lib/bb/server/xmlrpc.py | 19 +-
lib/bb/ui/buildinfohelper.py | 10 +
lib/bb/ui/toasterui.py | 3 +
lib/toaster/bldcontrol/__init__.py | 0
lib/toaster/bldcontrol/bbcontroller.py | 239 +++++++++++++++++++++
lib/toaster/bldcontrol/fixtures/initial_data.json | 1 +
lib/toaster/bldcontrol/management/__init__.py | 0
.../bldcontrol/management/commands/__init__.py | 0
.../bldcontrol/management/commands/runbuilds.py | 85 ++++++++
lib/toaster/bldcontrol/migrations/0001_initial.py | 154 +++++++++++++
lib/toaster/bldcontrol/migrations/__init__.py | 0
lib/toaster/bldcontrol/models.py | 83 +++++++
lib/toaster/bldcontrol/tests.py | 81 +++++++
lib/toaster/bldcontrol/views.py | 1 +
lib/toaster/toastergui/views.py | 7 +-
lib/toaster/toastermain/settings.py | 20 ++
18 files changed, 704 insertions(+), 16 deletions(-)
create mode 100644 lib/toaster/bldcontrol/__init__.py
create mode 100644 lib/toaster/bldcontrol/bbcontroller.py
create mode 100644 lib/toaster/bldcontrol/fixtures/initial_data.json
create mode 100644 lib/toaster/bldcontrol/management/__init__.py
create mode 100644 lib/toaster/bldcontrol/management/commands/__init__.py
create mode 100644 lib/toaster/bldcontrol/management/commands/runbuilds.py
create mode 100644 lib/toaster/bldcontrol/migrations/0001_initial.py
create mode 100644 lib/toaster/bldcontrol/migrations/__init__.py
create mode 100644 lib/toaster/bldcontrol/models.py
create mode 100644 lib/toaster/bldcontrol/tests.py
create mode 100644 lib/toaster/bldcontrol/views.py
--
1.9.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 0/6] toaster patchset: localhost build control
@ 2014-06-12 13:51 ` Alex DAMIAN
0 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:51 UTC (permalink / raw)
To: bitbake-devel, toaster; +Cc: Alexandru DAMIAN
From: Alexandru DAMIAN <alexandru.damian@intel.com>
Hello,
This toaster patchset contains the core of the build control functionality
for running on a localhost backend. Variants of this patchset have
been reviewed on the toaster mailing list.
Please merge at your convenience,
Alex
The following changes since commit 2e742c03e8dfdfa67899e7f5d579ed14bd87e139:
bb/utils: fix contains_any() (2014-06-11 13:18:20 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib adamian/20140612-submission-bb
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/20140612-submission-bb
Alexandru DAMIAN (6):
toaster: initial bldcontrol application
toaster: create models for bldcontrol and enable it
toaster: add function to get the database url
xmlrpc: add support for token reusing
toastergui: fix built ETA calculation
toaster: build control functionality
bin/bitbake | 8 +-
bin/toaster | 9 +-
lib/bb/server/xmlrpc.py | 19 +-
lib/bb/ui/buildinfohelper.py | 10 +
lib/bb/ui/toasterui.py | 3 +
lib/toaster/bldcontrol/__init__.py | 0
lib/toaster/bldcontrol/bbcontroller.py | 239 +++++++++++++++++++++
lib/toaster/bldcontrol/fixtures/initial_data.json | 1 +
lib/toaster/bldcontrol/management/__init__.py | 0
.../bldcontrol/management/commands/__init__.py | 0
.../bldcontrol/management/commands/runbuilds.py | 85 ++++++++
lib/toaster/bldcontrol/migrations/0001_initial.py | 154 +++++++++++++
lib/toaster/bldcontrol/migrations/__init__.py | 0
lib/toaster/bldcontrol/models.py | 83 +++++++
lib/toaster/bldcontrol/tests.py | 81 +++++++
lib/toaster/bldcontrol/views.py | 1 +
lib/toaster/toastergui/views.py | 7 +-
lib/toaster/toastermain/settings.py | 20 ++
18 files changed, 704 insertions(+), 16 deletions(-)
create mode 100644 lib/toaster/bldcontrol/__init__.py
create mode 100644 lib/toaster/bldcontrol/bbcontroller.py
create mode 100644 lib/toaster/bldcontrol/fixtures/initial_data.json
create mode 100644 lib/toaster/bldcontrol/management/__init__.py
create mode 100644 lib/toaster/bldcontrol/management/commands/__init__.py
create mode 100644 lib/toaster/bldcontrol/management/commands/runbuilds.py
create mode 100644 lib/toaster/bldcontrol/migrations/0001_initial.py
create mode 100644 lib/toaster/bldcontrol/migrations/__init__.py
create mode 100644 lib/toaster/bldcontrol/models.py
create mode 100644 lib/toaster/bldcontrol/tests.py
create mode 100644 lib/toaster/bldcontrol/views.py
--
1.9.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 1/6] toaster: initial bldcontrol application
2014-06-12 13:51 ` Alex DAMIAN
@ 2014-06-12 13:51 ` Alex DAMIAN
-1 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:51 UTC (permalink / raw)
To: bitbake-devel, toaster
From: Alexandru DAMIAN <alexandru.damian@intel.com>
The build features of Toaster will be contained into
a separate application, as to modularize the Toaster
project and provide multiple options for deployment.
This patch adds the application as a barebone Django
application in the Toaster project.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
lib/toaster/bldcontrol/__init__.py | 0
lib/toaster/bldcontrol/models.py | 3 +++
lib/toaster/bldcontrol/tests.py | 16 ++++++++++++++++
lib/toaster/bldcontrol/views.py | 1 +
4 files changed, 20 insertions(+)
create mode 100644 lib/toaster/bldcontrol/__init__.py
create mode 100644 lib/toaster/bldcontrol/models.py
create mode 100644 lib/toaster/bldcontrol/tests.py
create mode 100644 lib/toaster/bldcontrol/views.py
diff --git a/lib/toaster/bldcontrol/__init__.py b/lib/toaster/bldcontrol/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/toaster/bldcontrol/models.py b/lib/toaster/bldcontrol/models.py
new file mode 100644
index 0000000..71a8362
--- /dev/null
+++ b/lib/toaster/bldcontrol/models.py
@@ -0,0 +1,3 @@
+from django.db import models
+
+# Create your models here.
diff --git a/lib/toaster/bldcontrol/tests.py b/lib/toaster/bldcontrol/tests.py
new file mode 100644
index 0000000..501deb7
--- /dev/null
+++ b/lib/toaster/bldcontrol/tests.py
@@ -0,0 +1,16 @@
+"""
+This file demonstrates writing tests using the unittest module. These will pass
+when you run "manage.py test".
+
+Replace this with more appropriate tests for your application.
+"""
+
+from django.test import TestCase
+
+
+class SimpleTest(TestCase):
+ def test_basic_addition(self):
+ """
+ Tests that 1 + 1 always equals 2.
+ """
+ self.assertEqual(1 + 1, 2)
diff --git a/lib/toaster/bldcontrol/views.py b/lib/toaster/bldcontrol/views.py
new file mode 100644
index 0000000..60f00ef
--- /dev/null
+++ b/lib/toaster/bldcontrol/views.py
@@ -0,0 +1 @@
+# Create your views here.
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 1/6] toaster: initial bldcontrol application
@ 2014-06-12 13:51 ` Alex DAMIAN
0 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:51 UTC (permalink / raw)
To: bitbake-devel, toaster; +Cc: Alexandru DAMIAN
From: Alexandru DAMIAN <alexandru.damian@intel.com>
The build features of Toaster will be contained into
a separate application, as to modularize the Toaster
project and provide multiple options for deployment.
This patch adds the application as a barebone Django
application in the Toaster project.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
lib/toaster/bldcontrol/__init__.py | 0
lib/toaster/bldcontrol/models.py | 3 +++
lib/toaster/bldcontrol/tests.py | 16 ++++++++++++++++
lib/toaster/bldcontrol/views.py | 1 +
4 files changed, 20 insertions(+)
create mode 100644 lib/toaster/bldcontrol/__init__.py
create mode 100644 lib/toaster/bldcontrol/models.py
create mode 100644 lib/toaster/bldcontrol/tests.py
create mode 100644 lib/toaster/bldcontrol/views.py
diff --git a/lib/toaster/bldcontrol/__init__.py b/lib/toaster/bldcontrol/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/toaster/bldcontrol/models.py b/lib/toaster/bldcontrol/models.py
new file mode 100644
index 0000000..71a8362
--- /dev/null
+++ b/lib/toaster/bldcontrol/models.py
@@ -0,0 +1,3 @@
+from django.db import models
+
+# Create your models here.
diff --git a/lib/toaster/bldcontrol/tests.py b/lib/toaster/bldcontrol/tests.py
new file mode 100644
index 0000000..501deb7
--- /dev/null
+++ b/lib/toaster/bldcontrol/tests.py
@@ -0,0 +1,16 @@
+"""
+This file demonstrates writing tests using the unittest module. These will pass
+when you run "manage.py test".
+
+Replace this with more appropriate tests for your application.
+"""
+
+from django.test import TestCase
+
+
+class SimpleTest(TestCase):
+ def test_basic_addition(self):
+ """
+ Tests that 1 + 1 always equals 2.
+ """
+ self.assertEqual(1 + 1, 2)
diff --git a/lib/toaster/bldcontrol/views.py b/lib/toaster/bldcontrol/views.py
new file mode 100644
index 0000000..60f00ef
--- /dev/null
+++ b/lib/toaster/bldcontrol/views.py
@@ -0,0 +1 @@
+# Create your views here.
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 2/6] toaster: create models for bldcontrol and enable it
2014-06-12 13:51 ` Alex DAMIAN
@ 2014-06-12 13:51 ` Alex DAMIAN
-1 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:51 UTC (permalink / raw)
To: bitbake-devel, toaster
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We create the model classes that store information
about triggering builds, and the available build
environments.
We add a fixture with a default build environment
for build control, using a "build/" directory under
the poky checkout directory.
We enable the bldcontrol in toaster starting script
and in the toaster settings as to allow the actual database
to be kept in sync with the source code.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
bin/toaster | 2 +-
lib/toaster/bldcontrol/fixtures/initial_data.json | 1 +
lib/toaster/bldcontrol/migrations/0001_initial.py | 154 ++++++++++++++++++++++
lib/toaster/bldcontrol/migrations/__init__.py | 0
lib/toaster/bldcontrol/models.py | 80 ++++++++++-
lib/toaster/toastermain/settings.py | 1 +
6 files changed, 236 insertions(+), 2 deletions(-)
create mode 100644 lib/toaster/bldcontrol/fixtures/initial_data.json
create mode 100644 lib/toaster/bldcontrol/migrations/0001_initial.py
create mode 100644 lib/toaster/bldcontrol/migrations/__init__.py
diff --git a/bin/toaster b/bin/toaster
index f469e4f..80bda6d 100755
--- a/bin/toaster
+++ b/bin/toaster
@@ -63,7 +63,7 @@ function webserverStartAll()
retval=0
python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
fi
- # python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
+ python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
if [ $retval -eq 0 ]; then
python $BBBASEDIR/lib/toaster/manage.py runserver 0.0.0.0:8000 </dev/null >${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
diff --git a/lib/toaster/bldcontrol/fixtures/initial_data.json b/lib/toaster/bldcontrol/fixtures/initial_data.json
new file mode 100644
index 0000000..21883ab
--- /dev/null
+++ b/lib/toaster/bldcontrol/fixtures/initial_data.json
@@ -0,0 +1 @@
+[{"pk": 1, "model": "bldcontrol.buildenvironment", "fields": {"updated": "2014-05-20T12:17:30Z", "created": "2014-05-20T12:17:30Z", "lock": 0, "bbstate": 0, "bbaddress": "", "betype": 0, "bbtoken": "", "address": "localhost", "bbport": -1}}]
diff --git a/lib/toaster/bldcontrol/migrations/0001_initial.py b/lib/toaster/bldcontrol/migrations/0001_initial.py
new file mode 100644
index 0000000..9f13d14
--- /dev/null
+++ b/lib/toaster/bldcontrol/migrations/0001_initial.py
@@ -0,0 +1,154 @@
+# -*- coding: utf-8 -*-
+from south.utils import datetime_utils as datetime
+from south.db import db
+from south.v2 import SchemaMigration
+from django.db import models
+
+
+class Migration(SchemaMigration):
+
+ def forwards(self, orm):
+ # Adding model 'BuildEnvironment'
+ db.create_table(u'bldcontrol_buildenvironment', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('address', self.gf('django.db.models.fields.CharField')(max_length=254)),
+ ('betype', self.gf('django.db.models.fields.IntegerField')()),
+ ('bbaddress', self.gf('django.db.models.fields.CharField')(max_length=254, blank=True)),
+ ('bbport', self.gf('django.db.models.fields.IntegerField')(default=-1)),
+ ('bbtoken', self.gf('django.db.models.fields.CharField')(max_length=126, blank=True)),
+ ('bbstate', self.gf('django.db.models.fields.IntegerField')(default=0)),
+ ('lock', self.gf('django.db.models.fields.IntegerField')(default=0)),
+ ('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
+ ('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BuildEnvironment'])
+
+ # Adding model 'BuildRequest'
+ db.create_table(u'bldcontrol_buildrequest', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('project', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['orm.Project'])),
+ ('build', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['orm.Build'], null=True)),
+ ('state', self.gf('django.db.models.fields.IntegerField')(default=0)),
+ ('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
+ ('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BuildRequest'])
+
+ # Adding model 'BRLayer'
+ db.create_table(u'bldcontrol_brlayer', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('req', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['bldcontrol.BuildRequest'])),
+ ('name', self.gf('django.db.models.fields.CharField')(max_length=100)),
+ ('giturl', self.gf('django.db.models.fields.CharField')(max_length=254)),
+ ('commit', self.gf('django.db.models.fields.CharField')(max_length=254)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BRLayer'])
+
+ # Adding model 'BRVariable'
+ db.create_table(u'bldcontrol_brvariable', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('req', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['bldcontrol.BuildRequest'])),
+ ('name', self.gf('django.db.models.fields.CharField')(max_length=100)),
+ ('value', self.gf('django.db.models.fields.TextField')(blank=True)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BRVariable'])
+
+ # Adding model 'BRTarget'
+ db.create_table(u'bldcontrol_brtarget', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('req', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['bldcontrol.BuildRequest'])),
+ ('target', self.gf('django.db.models.fields.CharField')(max_length=100)),
+ ('task', self.gf('django.db.models.fields.CharField')(max_length=100, null=True)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BRTarget'])
+
+
+ def backwards(self, orm):
+ # Deleting model 'BuildEnvironment'
+ db.delete_table(u'bldcontrol_buildenvironment')
+
+ # Deleting model 'BuildRequest'
+ db.delete_table(u'bldcontrol_buildrequest')
+
+ # Deleting model 'BRLayer'
+ db.delete_table(u'bldcontrol_brlayer')
+
+ # Deleting model 'BRVariable'
+ db.delete_table(u'bldcontrol_brvariable')
+
+ # Deleting model 'BRTarget'
+ db.delete_table(u'bldcontrol_brtarget')
+
+
+ models = {
+ u'bldcontrol.brlayer': {
+ 'Meta': {'object_name': 'BRLayer'},
+ 'commit': ('django.db.models.fields.CharField', [], {'max_length': '254'}),
+ 'giturl': ('django.db.models.fields.CharField', [], {'max_length': '254'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'req': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['bldcontrol.BuildRequest']"})
+ },
+ u'bldcontrol.brtarget': {
+ 'Meta': {'object_name': 'BRTarget'},
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'req': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['bldcontrol.BuildRequest']"}),
+ 'target': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'task': ('django.db.models.fields.CharField', [], {'max_length': '100', 'null': 'True'})
+ },
+ u'bldcontrol.brvariable': {
+ 'Meta': {'object_name': 'BRVariable'},
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'req': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['bldcontrol.BuildRequest']"}),
+ 'value': ('django.db.models.fields.TextField', [], {'blank': 'True'})
+ },
+ u'bldcontrol.buildenvironment': {
+ 'Meta': {'object_name': 'BuildEnvironment'},
+ 'address': ('django.db.models.fields.CharField', [], {'max_length': '254'}),
+ 'bbaddress': ('django.db.models.fields.CharField', [], {'max_length': '254', 'blank': 'True'}),
+ 'bbport': ('django.db.models.fields.IntegerField', [], {'default': '-1'}),
+ 'bbstate': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'bbtoken': ('django.db.models.fields.CharField', [], {'max_length': '126', 'blank': 'True'}),
+ 'betype': ('django.db.models.fields.IntegerField', [], {}),
+ 'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'lock': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
+ },
+ u'bldcontrol.buildrequest': {
+ 'Meta': {'object_name': 'BuildRequest'},
+ 'build': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['orm.Build']", 'null': 'True'}),
+ 'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'project': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['orm.Project']"}),
+ 'state': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
+ },
+ u'orm.build': {
+ 'Meta': {'object_name': 'Build'},
+ 'bitbake_version': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
+ 'build_name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'completed_on': ('django.db.models.fields.DateTimeField', [], {}),
+ 'cooker_log_path': ('django.db.models.fields.CharField', [], {'max_length': '500'}),
+ 'distro': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'distro_version': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'errors_no': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'machine': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'outcome': ('django.db.models.fields.IntegerField', [], {'default': '2'}),
+ 'project': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['orm.Project']", 'null': 'True'}),
+ 'started_on': ('django.db.models.fields.DateTimeField', [], {}),
+ 'timespent': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'warnings_no': ('django.db.models.fields.IntegerField', [], {'default': '0'})
+ },
+ u'orm.project': {
+ 'Meta': {'object_name': 'Project'},
+ 'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
+ }
+ }
+
+ complete_apps = ['bldcontrol']
\ No newline at end of file
diff --git a/lib/toaster/bldcontrol/migrations/__init__.py b/lib/toaster/bldcontrol/migrations/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/toaster/bldcontrol/models.py b/lib/toaster/bldcontrol/models.py
index 71a8362..11f487c 100644
--- a/lib/toaster/bldcontrol/models.py
+++ b/lib/toaster/bldcontrol/models.py
@@ -1,3 +1,81 @@
from django.db import models
+from django.core.validators import MaxValueValidator, MinValueValidator
+from orm.models import Project, ProjectLayer, ProjectVariable, ProjectTarget, Build
+
+# a BuildEnvironment is the equivalent of the "build/" directory on the localhost
+class BuildEnvironment(models.Model):
+ SERVER_STOPPED = 0
+ SERVER_STARTED = 1
+ SERVER_STATE = (
+ (SERVER_STOPPED, "stopped"),
+ (SERVER_STARTED, "started"),
+ )
+
+ TYPE_LOCAL = 0
+ TYPE_SSH = 1
+ TYPE = (
+ (TYPE_LOCAL, "local"),
+ (TYPE_SSH, "ssh"),
+ )
+
+ LOCK_FREE = 0
+ LOCK_LOCK = 1
+ LOCK_STATE = (
+ (LOCK_FREE, "free"),
+ (LOCK_LOCK, "lock"),
+ )
+
+ address = models.CharField(max_length = 254)
+ betype = models.IntegerField(choices = TYPE)
+ bbaddress = models.CharField(max_length = 254, blank = True)
+ bbport = models.IntegerField(default = -1)
+ bbtoken = models.CharField(max_length = 126, blank = True)
+ bbstate = models.IntegerField(choices = SERVER_STATE, default = SERVER_STOPPED)
+ lock = models.IntegerField(choices = LOCK_STATE, default = LOCK_FREE)
+ created = models.DateTimeField(auto_now_add = True)
+ updated = models.DateTimeField(auto_now = True)
+
+
+# a BuildRequest is a request that the scheduler will build using a BuildEnvironment
+# the build request queue is the table itself, ordered by state
+
+class BuildRequest(models.Model):
+ REQ_CREATED = 0
+ REQ_QUEUED = 1
+ REQ_INPROGRESS = 2
+ REQ_COMPLETED = 3
+
+ REQUEST_STATE = (
+ (REQ_CREATED, "created"),
+ (REQ_QUEUED, "queued"),
+ (REQ_INPROGRESS, "in progress"),
+ (REQ_COMPLETED, "completed"),
+ )
+
+ project = models.ForeignKey(Project)
+ build = models.ForeignKey(Build, null = True) # TODO: toasterui should set this when Build is created
+ state = models.IntegerField(choices = REQUEST_STATE, default = REQ_CREATED)
+ created = models.DateTimeField(auto_now_add = True)
+ updated = models.DateTimeField(auto_now = True)
+
+
+# These tables specify the settings for running an actual build.
+# They MUST be kept in sync with the tables in orm.models.Project*
+
+class BRLayer(models.Model):
+ req = models.ForeignKey(BuildRequest)
+ name = models.CharField(max_length = 100)
+ giturl = models.CharField(max_length = 254)
+ commit = models.CharField(max_length = 254)
+
+class BRVariable(models.Model):
+ req = models.ForeignKey(BuildRequest)
+ name = models.CharField(max_length=100)
+ value = models.TextField(blank = True)
+
+class BRTarget(models.Model):
+ req = models.ForeignKey(BuildRequest)
+ target = models.CharField(max_length=100)
+ task = models.CharField(max_length=100, null=True)
+
-# Create your models here.
diff --git a/lib/toaster/toastermain/settings.py b/lib/toaster/toastermain/settings.py
index adaa56c..2c52b68 100644
--- a/lib/toaster/toastermain/settings.py
+++ b/lib/toaster/toastermain/settings.py
@@ -221,6 +221,7 @@ INSTALLED_APPS = (
'toastergui',
'bldviewer',
'south',
+ 'bldcontrol',
)
# A sample logging configuration. The only tangible logging
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 2/6] toaster: create models for bldcontrol and enable it
@ 2014-06-12 13:51 ` Alex DAMIAN
0 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:51 UTC (permalink / raw)
To: bitbake-devel, toaster; +Cc: Alexandru DAMIAN
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We create the model classes that store information
about triggering builds, and the available build
environments.
We add a fixture with a default build environment
for build control, using a "build/" directory under
the poky checkout directory.
We enable the bldcontrol in toaster starting script
and in the toaster settings as to allow the actual database
to be kept in sync with the source code.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
bin/toaster | 2 +-
lib/toaster/bldcontrol/fixtures/initial_data.json | 1 +
lib/toaster/bldcontrol/migrations/0001_initial.py | 154 ++++++++++++++++++++++
lib/toaster/bldcontrol/migrations/__init__.py | 0
lib/toaster/bldcontrol/models.py | 80 ++++++++++-
lib/toaster/toastermain/settings.py | 1 +
6 files changed, 236 insertions(+), 2 deletions(-)
create mode 100644 lib/toaster/bldcontrol/fixtures/initial_data.json
create mode 100644 lib/toaster/bldcontrol/migrations/0001_initial.py
create mode 100644 lib/toaster/bldcontrol/migrations/__init__.py
diff --git a/bin/toaster b/bin/toaster
index f469e4f..80bda6d 100755
--- a/bin/toaster
+++ b/bin/toaster
@@ -63,7 +63,7 @@ function webserverStartAll()
retval=0
python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
fi
- # python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
+ python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
if [ $retval -eq 0 ]; then
python $BBBASEDIR/lib/toaster/manage.py runserver 0.0.0.0:8000 </dev/null >${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
diff --git a/lib/toaster/bldcontrol/fixtures/initial_data.json b/lib/toaster/bldcontrol/fixtures/initial_data.json
new file mode 100644
index 0000000..21883ab
--- /dev/null
+++ b/lib/toaster/bldcontrol/fixtures/initial_data.json
@@ -0,0 +1 @@
+[{"pk": 1, "model": "bldcontrol.buildenvironment", "fields": {"updated": "2014-05-20T12:17:30Z", "created": "2014-05-20T12:17:30Z", "lock": 0, "bbstate": 0, "bbaddress": "", "betype": 0, "bbtoken": "", "address": "localhost", "bbport": -1}}]
diff --git a/lib/toaster/bldcontrol/migrations/0001_initial.py b/lib/toaster/bldcontrol/migrations/0001_initial.py
new file mode 100644
index 0000000..9f13d14
--- /dev/null
+++ b/lib/toaster/bldcontrol/migrations/0001_initial.py
@@ -0,0 +1,154 @@
+# -*- coding: utf-8 -*-
+from south.utils import datetime_utils as datetime
+from south.db import db
+from south.v2 import SchemaMigration
+from django.db import models
+
+
+class Migration(SchemaMigration):
+
+ def forwards(self, orm):
+ # Adding model 'BuildEnvironment'
+ db.create_table(u'bldcontrol_buildenvironment', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('address', self.gf('django.db.models.fields.CharField')(max_length=254)),
+ ('betype', self.gf('django.db.models.fields.IntegerField')()),
+ ('bbaddress', self.gf('django.db.models.fields.CharField')(max_length=254, blank=True)),
+ ('bbport', self.gf('django.db.models.fields.IntegerField')(default=-1)),
+ ('bbtoken', self.gf('django.db.models.fields.CharField')(max_length=126, blank=True)),
+ ('bbstate', self.gf('django.db.models.fields.IntegerField')(default=0)),
+ ('lock', self.gf('django.db.models.fields.IntegerField')(default=0)),
+ ('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
+ ('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BuildEnvironment'])
+
+ # Adding model 'BuildRequest'
+ db.create_table(u'bldcontrol_buildrequest', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('project', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['orm.Project'])),
+ ('build', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['orm.Build'], null=True)),
+ ('state', self.gf('django.db.models.fields.IntegerField')(default=0)),
+ ('created', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)),
+ ('updated', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BuildRequest'])
+
+ # Adding model 'BRLayer'
+ db.create_table(u'bldcontrol_brlayer', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('req', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['bldcontrol.BuildRequest'])),
+ ('name', self.gf('django.db.models.fields.CharField')(max_length=100)),
+ ('giturl', self.gf('django.db.models.fields.CharField')(max_length=254)),
+ ('commit', self.gf('django.db.models.fields.CharField')(max_length=254)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BRLayer'])
+
+ # Adding model 'BRVariable'
+ db.create_table(u'bldcontrol_brvariable', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('req', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['bldcontrol.BuildRequest'])),
+ ('name', self.gf('django.db.models.fields.CharField')(max_length=100)),
+ ('value', self.gf('django.db.models.fields.TextField')(blank=True)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BRVariable'])
+
+ # Adding model 'BRTarget'
+ db.create_table(u'bldcontrol_brtarget', (
+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+ ('req', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['bldcontrol.BuildRequest'])),
+ ('target', self.gf('django.db.models.fields.CharField')(max_length=100)),
+ ('task', self.gf('django.db.models.fields.CharField')(max_length=100, null=True)),
+ ))
+ db.send_create_signal(u'bldcontrol', ['BRTarget'])
+
+
+ def backwards(self, orm):
+ # Deleting model 'BuildEnvironment'
+ db.delete_table(u'bldcontrol_buildenvironment')
+
+ # Deleting model 'BuildRequest'
+ db.delete_table(u'bldcontrol_buildrequest')
+
+ # Deleting model 'BRLayer'
+ db.delete_table(u'bldcontrol_brlayer')
+
+ # Deleting model 'BRVariable'
+ db.delete_table(u'bldcontrol_brvariable')
+
+ # Deleting model 'BRTarget'
+ db.delete_table(u'bldcontrol_brtarget')
+
+
+ models = {
+ u'bldcontrol.brlayer': {
+ 'Meta': {'object_name': 'BRLayer'},
+ 'commit': ('django.db.models.fields.CharField', [], {'max_length': '254'}),
+ 'giturl': ('django.db.models.fields.CharField', [], {'max_length': '254'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'req': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['bldcontrol.BuildRequest']"})
+ },
+ u'bldcontrol.brtarget': {
+ 'Meta': {'object_name': 'BRTarget'},
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'req': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['bldcontrol.BuildRequest']"}),
+ 'target': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'task': ('django.db.models.fields.CharField', [], {'max_length': '100', 'null': 'True'})
+ },
+ u'bldcontrol.brvariable': {
+ 'Meta': {'object_name': 'BRVariable'},
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'req': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['bldcontrol.BuildRequest']"}),
+ 'value': ('django.db.models.fields.TextField', [], {'blank': 'True'})
+ },
+ u'bldcontrol.buildenvironment': {
+ 'Meta': {'object_name': 'BuildEnvironment'},
+ 'address': ('django.db.models.fields.CharField', [], {'max_length': '254'}),
+ 'bbaddress': ('django.db.models.fields.CharField', [], {'max_length': '254', 'blank': 'True'}),
+ 'bbport': ('django.db.models.fields.IntegerField', [], {'default': '-1'}),
+ 'bbstate': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'bbtoken': ('django.db.models.fields.CharField', [], {'max_length': '126', 'blank': 'True'}),
+ 'betype': ('django.db.models.fields.IntegerField', [], {}),
+ 'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'lock': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
+ },
+ u'bldcontrol.buildrequest': {
+ 'Meta': {'object_name': 'BuildRequest'},
+ 'build': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['orm.Build']", 'null': 'True'}),
+ 'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'project': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['orm.Project']"}),
+ 'state': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
+ },
+ u'orm.build': {
+ 'Meta': {'object_name': 'Build'},
+ 'bitbake_version': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
+ 'build_name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'completed_on': ('django.db.models.fields.DateTimeField', [], {}),
+ 'cooker_log_path': ('django.db.models.fields.CharField', [], {'max_length': '500'}),
+ 'distro': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'distro_version': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'errors_no': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'machine': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'outcome': ('django.db.models.fields.IntegerField', [], {'default': '2'}),
+ 'project': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['orm.Project']", 'null': 'True'}),
+ 'started_on': ('django.db.models.fields.DateTimeField', [], {}),
+ 'timespent': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'warnings_no': ('django.db.models.fields.IntegerField', [], {'default': '0'})
+ },
+ u'orm.project': {
+ 'Meta': {'object_name': 'Project'},
+ 'created': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'})
+ }
+ }
+
+ complete_apps = ['bldcontrol']
\ No newline at end of file
diff --git a/lib/toaster/bldcontrol/migrations/__init__.py b/lib/toaster/bldcontrol/migrations/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/toaster/bldcontrol/models.py b/lib/toaster/bldcontrol/models.py
index 71a8362..11f487c 100644
--- a/lib/toaster/bldcontrol/models.py
+++ b/lib/toaster/bldcontrol/models.py
@@ -1,3 +1,81 @@
from django.db import models
+from django.core.validators import MaxValueValidator, MinValueValidator
+from orm.models import Project, ProjectLayer, ProjectVariable, ProjectTarget, Build
+
+# a BuildEnvironment is the equivalent of the "build/" directory on the localhost
+class BuildEnvironment(models.Model):
+ SERVER_STOPPED = 0
+ SERVER_STARTED = 1
+ SERVER_STATE = (
+ (SERVER_STOPPED, "stopped"),
+ (SERVER_STARTED, "started"),
+ )
+
+ TYPE_LOCAL = 0
+ TYPE_SSH = 1
+ TYPE = (
+ (TYPE_LOCAL, "local"),
+ (TYPE_SSH, "ssh"),
+ )
+
+ LOCK_FREE = 0
+ LOCK_LOCK = 1
+ LOCK_STATE = (
+ (LOCK_FREE, "free"),
+ (LOCK_LOCK, "lock"),
+ )
+
+ address = models.CharField(max_length = 254)
+ betype = models.IntegerField(choices = TYPE)
+ bbaddress = models.CharField(max_length = 254, blank = True)
+ bbport = models.IntegerField(default = -1)
+ bbtoken = models.CharField(max_length = 126, blank = True)
+ bbstate = models.IntegerField(choices = SERVER_STATE, default = SERVER_STOPPED)
+ lock = models.IntegerField(choices = LOCK_STATE, default = LOCK_FREE)
+ created = models.DateTimeField(auto_now_add = True)
+ updated = models.DateTimeField(auto_now = True)
+
+
+# a BuildRequest is a request that the scheduler will build using a BuildEnvironment
+# the build request queue is the table itself, ordered by state
+
+class BuildRequest(models.Model):
+ REQ_CREATED = 0
+ REQ_QUEUED = 1
+ REQ_INPROGRESS = 2
+ REQ_COMPLETED = 3
+
+ REQUEST_STATE = (
+ (REQ_CREATED, "created"),
+ (REQ_QUEUED, "queued"),
+ (REQ_INPROGRESS, "in progress"),
+ (REQ_COMPLETED, "completed"),
+ )
+
+ project = models.ForeignKey(Project)
+ build = models.ForeignKey(Build, null = True) # TODO: toasterui should set this when Build is created
+ state = models.IntegerField(choices = REQUEST_STATE, default = REQ_CREATED)
+ created = models.DateTimeField(auto_now_add = True)
+ updated = models.DateTimeField(auto_now = True)
+
+
+# These tables specify the settings for running an actual build.
+# They MUST be kept in sync with the tables in orm.models.Project*
+
+class BRLayer(models.Model):
+ req = models.ForeignKey(BuildRequest)
+ name = models.CharField(max_length = 100)
+ giturl = models.CharField(max_length = 254)
+ commit = models.CharField(max_length = 254)
+
+class BRVariable(models.Model):
+ req = models.ForeignKey(BuildRequest)
+ name = models.CharField(max_length=100)
+ value = models.TextField(blank = True)
+
+class BRTarget(models.Model):
+ req = models.ForeignKey(BuildRequest)
+ target = models.CharField(max_length=100)
+ task = models.CharField(max_length=100, null=True)
+
-# Create your models here.
diff --git a/lib/toaster/toastermain/settings.py b/lib/toaster/toastermain/settings.py
index adaa56c..2c52b68 100644
--- a/lib/toaster/toastermain/settings.py
+++ b/lib/toaster/toastermain/settings.py
@@ -221,6 +221,7 @@ INSTALLED_APPS = (
'toastergui',
'bldviewer',
'south',
+ 'bldcontrol',
)
# A sample logging configuration. The only tangible logging
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 3/6] toaster: add function to get the database url
2014-06-12 13:51 ` Alex DAMIAN
@ 2014-06-12 13:51 ` Alex DAMIAN
-1 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:51 UTC (permalink / raw)
To: bitbake-devel, toaster
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We add a function that returns the DATABASE_URL
for the current 'default' database settings. This
is useful to be able to start other toaster instances
with the same database settigns as the currently running
instance.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
lib/toaster/toastermain/settings.py | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/lib/toaster/toastermain/settings.py b/lib/toaster/toastermain/settings.py
index 2c52b68..2ce10c4 100644
--- a/lib/toaster/toastermain/settings.py
+++ b/lib/toaster/toastermain/settings.py
@@ -74,6 +74,25 @@ if 'DATABASE_URL' in os.environ:
else:
raise Exception("FIXME: Please implement missing database url schema for url: %s" % dburl)
+
+# Allows current database settings to be exported as a DATABASE_URL environment variable value
+
+def getDATABASE_URL():
+ d = DATABASES['default']
+ if d['ENGINE'] == 'django.db.backends.sqlite3':
+ if d['NAME'] == ':memory:':
+ return 'sqlite3://:memory:'
+ elif d['NAME'].startswith("/"):
+ return 'sqlite3://' + d['NAME']
+ return "sqlite3://" + os.path.join(os.getcwd(), d['NAME'])
+
+ elif d['ENGINE'] == 'django.db.backends.mysql':
+ return "mysql://" + d['USER'] + ":" + d['PASSWORD'] + "@" + d['HOST'] + ":" + d['PORT'] + "/" + d['NAME']
+
+ raise Exception("FIXME: Please implement missing database url schema for engine: %s" % d['ENGINE'])
+
+
+
# Hosts/domain names that are valid for this site; required if DEBUG is False
# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
ALLOWED_HOSTS = []
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 3/6] toaster: add function to get the database url
@ 2014-06-12 13:51 ` Alex DAMIAN
0 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:51 UTC (permalink / raw)
To: bitbake-devel, toaster; +Cc: Alexandru DAMIAN
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We add a function that returns the DATABASE_URL
for the current 'default' database settings. This
is useful to be able to start other toaster instances
with the same database settigns as the currently running
instance.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
lib/toaster/toastermain/settings.py | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/lib/toaster/toastermain/settings.py b/lib/toaster/toastermain/settings.py
index 2c52b68..2ce10c4 100644
--- a/lib/toaster/toastermain/settings.py
+++ b/lib/toaster/toastermain/settings.py
@@ -74,6 +74,25 @@ if 'DATABASE_URL' in os.environ:
else:
raise Exception("FIXME: Please implement missing database url schema for url: %s" % dburl)
+
+# Allows current database settings to be exported as a DATABASE_URL environment variable value
+
+def getDATABASE_URL():
+ d = DATABASES['default']
+ if d['ENGINE'] == 'django.db.backends.sqlite3':
+ if d['NAME'] == ':memory:':
+ return 'sqlite3://:memory:'
+ elif d['NAME'].startswith("/"):
+ return 'sqlite3://' + d['NAME']
+ return "sqlite3://" + os.path.join(os.getcwd(), d['NAME'])
+
+ elif d['ENGINE'] == 'django.db.backends.mysql':
+ return "mysql://" + d['USER'] + ":" + d['PASSWORD'] + "@" + d['HOST'] + ":" + d['PORT'] + "/" + d['NAME']
+
+ raise Exception("FIXME: Please implement missing database url schema for engine: %s" % d['ENGINE'])
+
+
+
# Hosts/domain names that are valid for this site; required if DEBUG is False
# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
ALLOWED_HOSTS = []
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 4/6] xmlrpc: add support for token reusing
2014-06-12 13:51 ` Alex DAMIAN
@ 2014-06-12 13:52 ` Alex DAMIAN
-1 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:52 UTC (permalink / raw)
To: bitbake-devel, toaster
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We add support to specify a connection token in the command line
and in the environment variable BBTOKEN.
When a client registers to a bitbake server, that client
will have exclusive access to the server. The client is identified
by a server-supplied token. If a client terminates, we cannot
reconnect to the server as the token is lost.
This patch adds the capability to specify the connection token
in the command line for xmlrpc clients. This allows us
to have bitbake work as an already-authenticated
client with the server and resume sending commands to a server.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
bin/bitbake | 8 +++++++-
lib/bb/server/xmlrpc.py | 19 ++++++++++++-------
2 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/bin/bitbake b/bin/bitbake
index f120bac..b3acbe1 100755
--- a/bin/bitbake
+++ b/bin/bitbake
@@ -169,6 +169,9 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
parser.add_option("-t", "--servertype", help = "Choose which server to use, process or xmlrpc.",
action = "store", dest = "servertype")
+ parser.add_option("", "--token", help = "Specify the connection token to be used when connecting to a remote server.",
+ action = "store", dest = "xmlrpctoken")
+
parser.add_option("", "--revisions-changed", help = "Set the exit code depending on whether upstream floating revisions have changed or not.",
action = "store_true", dest = "revisions_changed", default = False)
@@ -200,6 +203,9 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
options.servertype = "xmlrpc"
options.remote_server = os.environ["BBSERVER"]
+ if "BBTOKEN" in os.environ:
+ options.xmlrpctoken = os.environ["BBTOKEN"]
+
# if BBSERVER says to autodetect, let's do that
if options.remote_server:
[host, port] = options.remote_server.split(":", 2)
@@ -332,7 +338,7 @@ def main():
bb.event.ui_queue = []
else:
# we start a stub server that is actually a XMLRPClient that connects to a real server
- server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
+ server = servermodule.BitBakeXMLRPCClient(configParams.observe_only, configParams.xmlrpctoken)
server.saveConnectionDetails(configParams.remote_server)
diff --git a/lib/bb/server/xmlrpc.py b/lib/bb/server/xmlrpc.py
index d6f4338..4205a4c 100644
--- a/lib/bb/server/xmlrpc.py
+++ b/lib/bb/server/xmlrpc.py
@@ -281,13 +281,16 @@ class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
self.observer_only = observer_only
self.featureset = featureset
- def connect(self):
- if not self.observer_only:
- token = self.connection.addClient()
- else:
- token = "observer"
+ def connect(self, token = None):
+ if token is None:
+ if self.observer_only:
+ token = "observer"
+ else:
+ token = self.connection.addClient()
+
if token is None:
return None
+
self.transport.set_connection_token(token)
self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo)
@@ -336,7 +339,9 @@ class BitBakeServer(BitBakeBaseServer):
class BitBakeXMLRPCClient(BitBakeBaseServer):
- def __init__(self, observer_only = False):
+ def __init__(self, observer_only = False, token = None):
+ self.token = token
+
self.observer_only = observer_only
# if we need extra caches, just tell the server to load them all
pass
@@ -366,7 +371,7 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
try:
self.serverImpl = XMLRPCProxyServer(host, port)
self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
- return self.connection.connect()
+ return self.connection.connect(self.token)
except Exception as e:
bb.warn("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
raise e
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 4/6] xmlrpc: add support for token reusing
@ 2014-06-12 13:52 ` Alex DAMIAN
0 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:52 UTC (permalink / raw)
To: bitbake-devel, toaster; +Cc: Alexandru DAMIAN
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We add support to specify a connection token in the command line
and in the environment variable BBTOKEN.
When a client registers to a bitbake server, that client
will have exclusive access to the server. The client is identified
by a server-supplied token. If a client terminates, we cannot
reconnect to the server as the token is lost.
This patch adds the capability to specify the connection token
in the command line for xmlrpc clients. This allows us
to have bitbake work as an already-authenticated
client with the server and resume sending commands to a server.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
bin/bitbake | 8 +++++++-
lib/bb/server/xmlrpc.py | 19 ++++++++++++-------
2 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/bin/bitbake b/bin/bitbake
index f120bac..b3acbe1 100755
--- a/bin/bitbake
+++ b/bin/bitbake
@@ -169,6 +169,9 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
parser.add_option("-t", "--servertype", help = "Choose which server to use, process or xmlrpc.",
action = "store", dest = "servertype")
+ parser.add_option("", "--token", help = "Specify the connection token to be used when connecting to a remote server.",
+ action = "store", dest = "xmlrpctoken")
+
parser.add_option("", "--revisions-changed", help = "Set the exit code depending on whether upstream floating revisions have changed or not.",
action = "store_true", dest = "revisions_changed", default = False)
@@ -200,6 +203,9 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
options.servertype = "xmlrpc"
options.remote_server = os.environ["BBSERVER"]
+ if "BBTOKEN" in os.environ:
+ options.xmlrpctoken = os.environ["BBTOKEN"]
+
# if BBSERVER says to autodetect, let's do that
if options.remote_server:
[host, port] = options.remote_server.split(":", 2)
@@ -332,7 +338,7 @@ def main():
bb.event.ui_queue = []
else:
# we start a stub server that is actually a XMLRPClient that connects to a real server
- server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
+ server = servermodule.BitBakeXMLRPCClient(configParams.observe_only, configParams.xmlrpctoken)
server.saveConnectionDetails(configParams.remote_server)
diff --git a/lib/bb/server/xmlrpc.py b/lib/bb/server/xmlrpc.py
index d6f4338..4205a4c 100644
--- a/lib/bb/server/xmlrpc.py
+++ b/lib/bb/server/xmlrpc.py
@@ -281,13 +281,16 @@ class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
self.observer_only = observer_only
self.featureset = featureset
- def connect(self):
- if not self.observer_only:
- token = self.connection.addClient()
- else:
- token = "observer"
+ def connect(self, token = None):
+ if token is None:
+ if self.observer_only:
+ token = "observer"
+ else:
+ token = self.connection.addClient()
+
if token is None:
return None
+
self.transport.set_connection_token(token)
self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo)
@@ -336,7 +339,9 @@ class BitBakeServer(BitBakeBaseServer):
class BitBakeXMLRPCClient(BitBakeBaseServer):
- def __init__(self, observer_only = False):
+ def __init__(self, observer_only = False, token = None):
+ self.token = token
+
self.observer_only = observer_only
# if we need extra caches, just tell the server to load them all
pass
@@ -366,7 +371,7 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
try:
self.serverImpl = XMLRPCProxyServer(host, port)
self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
- return self.connection.connect()
+ return self.connection.connect(self.token)
except Exception as e:
bb.warn("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
raise e
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 5/6] toastergui: fix built ETA calculation
2014-06-12 13:51 ` Alex DAMIAN
@ 2014-06-12 13:52 ` Alex DAMIAN
-1 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:52 UTC (permalink / raw)
To: bitbake-devel, toaster
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We fix the calculation showing the estimated time
to build completion.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
lib/toaster/toastergui/views.py | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py
index 686cd5c..b38d260 100755
--- a/lib/toaster/toastergui/views.py
+++ b/lib/toaster/toastergui/views.py
@@ -230,11 +230,10 @@ def builds(request):
b.completeper = tf.exclude(order__isnull=True).count()*100/tf.count()
else:
b.completeper = 0
- b.eta = timezone.now()
+
+ b.eta = 0
if b.completeper > 0:
- b.eta += ((timezone.now() - b.started_on)*100/b.completeper)
- else:
- b.eta = 0
+ b.eta = timezone.now() + ((timezone.now() - b.started_on)*(100-b.completeper)/b.completeper)
# set up list of fstypes for each build
fstypes_map = {};
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 5/6] toastergui: fix built ETA calculation
@ 2014-06-12 13:52 ` Alex DAMIAN
0 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:52 UTC (permalink / raw)
To: bitbake-devel, toaster; +Cc: Alexandru DAMIAN
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We fix the calculation showing the estimated time
to build completion.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
lib/toaster/toastergui/views.py | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py
index 686cd5c..b38d260 100755
--- a/lib/toaster/toastergui/views.py
+++ b/lib/toaster/toastergui/views.py
@@ -230,11 +230,10 @@ def builds(request):
b.completeper = tf.exclude(order__isnull=True).count()*100/tf.count()
else:
b.completeper = 0
- b.eta = timezone.now()
+
+ b.eta = 0
if b.completeper > 0:
- b.eta += ((timezone.now() - b.started_on)*100/b.completeper)
- else:
- b.eta = 0
+ b.eta = timezone.now() + ((timezone.now() - b.started_on)*(100-b.completeper)/b.completeper)
# set up list of fstypes for each build
fstypes_map = {};
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 6/6] toaster: build control functionality
2014-06-12 13:51 ` Alex DAMIAN
@ 2014-06-12 13:52 ` Alex DAMIAN
-1 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:52 UTC (permalink / raw)
To: bitbake-devel, toaster
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We add the build control functionality to toaster.
* The bldcontrol application gains bbcontroller classes
that know how to manage a localhost build environment.
* The toaster UI now detects it is running under build
environment controller, and update the build controller
database and will shut down the bitbake server once
the build is complete.
* The toaster script can now run in standalone mode,
launching the build controller and the web interface instead
of just monitoring the build, as in the interactive mode.
* A fixture with the default build controller entry for
localhost is provided.
[YOCTO #5490]
[YOCTO #5491]
[YOCTO #5492]
[YOCTO #5493]
[YOCTO #5494]
[YOCTO #5537]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
bin/toaster | 7 +-
lib/bb/ui/buildinfohelper.py | 10 +
lib/bb/ui/toasterui.py | 3 +
lib/toaster/bldcontrol/bbcontroller.py | 239 +++++++++++++++++++++
lib/toaster/bldcontrol/fixtures/initial_data.json | 2 +-
lib/toaster/bldcontrol/management/__init__.py | 0
.../bldcontrol/management/commands/__init__.py | 0
.../bldcontrol/management/commands/runbuilds.py | 85 ++++++++
lib/toaster/bldcontrol/models.py | 2 +
lib/toaster/bldcontrol/tests.py | 73 ++++++-
10 files changed, 413 insertions(+), 8 deletions(-)
create mode 100644 lib/toaster/bldcontrol/bbcontroller.py
create mode 100644 lib/toaster/bldcontrol/management/__init__.py
create mode 100644 lib/toaster/bldcontrol/management/commands/__init__.py
create mode 100644 lib/toaster/bldcontrol/management/commands/runbuilds.py
diff --git a/bin/toaster b/bin/toaster
index 80bda6d..01ffc7a 100755
--- a/bin/toaster
+++ b/bin/toaster
@@ -139,15 +139,16 @@ if [ -z "$ZSH_NAME" ] && [ `basename \"$0\"` = `basename \"$BASH_SOURCE\"` ]; th
webserverKillAll
RUNNING=0
}
- webserverStartAll || exit 1
+ webserverStartAll || (echo "Fail to start the web server, stopping" 1>&2 && exit 1)
xdg-open http://0.0.0.0:8000/ >/dev/null 2>&1 &
trap trap_ctrlc SIGINT
echo "Running. Stop with Ctrl-C"
while [ $RUNNING -gt 0 ]; do
- wait;
+ python $BBBASEDIR/lib/toaster/manage.py runbuilds
+ sleep 1
done
echo "**** Exit"
- exit 1
+ exit 0
fi
# We make sure we're running in the current shell and in a good environment
diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 58df687..77be7c7 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -916,6 +916,16 @@ class BuildInfoHelper(object):
self.internal_state['recipes'],
)
+ def store_build_done(self, br_id, be_id):
+ from bldcontrol.models import BuildEnvironment, BuildRequest
+ be = BuildEnvironment.objects.get(pk = be_id)
+ be.lock = BuildEnvironment.LOCK_LOCK
+ be.save()
+ br = BuildRequest.objects.get(pk = br_id)
+ br.state = BuildRequest.REQ_COMPLETED
+ br.build = self.internal_state['build']
+ br.save()
+
def _store_log_information(self, level, text):
log_information = {}
log_information['build'] = self.internal_state['build']
diff --git a/lib/bb/ui/toasterui.py b/lib/bb/ui/toasterui.py
index c1696b2..b1c80cc 100644
--- a/lib/bb/ui/toasterui.py
+++ b/lib/bb/ui/toasterui.py
@@ -228,8 +228,11 @@ def main(server, eventHandler, params ):
brbe = server.runCommand(["getVariable", "TOASTER_BRBE"])[0]
+ br_id, be_id = brbe.split(":")
# we start a new build info
if brbe is not None:
+ buildinfohelper.store_build_done(br_id, be_id)
+
print "we are under BuildEnvironment management - after the build, we exit"
server.terminateServer()
else:
diff --git a/lib/toaster/bldcontrol/bbcontroller.py b/lib/toaster/bldcontrol/bbcontroller.py
new file mode 100644
index 0000000..d2b2a23
--- /dev/null
+++ b/lib/toaster/bldcontrol/bbcontroller.py
@@ -0,0 +1,239 @@
+#
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+import os
+import sys
+import re
+from django.db import transaction
+from django.db.models import Q
+from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget
+import subprocess
+
+from toastermain import settings
+
+
+# load Bitbake components
+path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+sys.path.insert(0, path)
+import bb.server.xmlrpc
+
+class BitbakeController(object):
+ """ This is the basic class that controlls a bitbake server.
+ It is outside the scope of this class on how the server is started and aquired
+ """
+
+ def __init__(self, connection):
+ self.connection = connection
+
+ def _runCommand(self, command):
+ result, error = self.connection.connection.runCommand(command)
+ if error:
+ raise Exception(error)
+ return result
+
+ def disconnect(self):
+ return self.connection.removeClient()
+
+ def setVariable(self, name, value):
+ return self._runCommand(["setVariable", name, value])
+
+ def build(self, targets, task = None):
+ if task is None:
+ task = "build"
+ return self._runCommand(["buildTargets", targets, task])
+
+
+
+def getBuildEnvironmentController(**kwargs):
+ """ Gets you a BuildEnvironmentController that encapsulates a build environment,
+ based on the query dictionary sent in.
+
+ This is used to retrieve, for example, the currently running BE from inside
+ the toaster UI, or find a new BE to start a new build in it.
+
+ The return object MUST always be a BuildEnvironmentController.
+ """
+ be = BuildEnvironment.objects.filter(Q(**kwargs))[0]
+ if be.betype == BuildEnvironment.TYPE_LOCAL:
+ return LocalhostBEController(be)
+ elif be.betype == BuildEnvironment.TYPE_SSH:
+ return SSHBEController(be)
+ else:
+ raise Exception("FIXME: Implement BEC for type %s" % str(be.betype))
+
+
+
+class BuildEnvironmentController(object):
+ """ BuildEnvironmentController (BEC) is the abstract class that defines the operations that MUST
+ or SHOULD be supported by a Build Environment. It is used to establish the framework, and must
+ not be instantiated directly by the user.
+
+ Use the "getBuildEnvironmentController()" function to get a working BEC for your remote.
+
+ How the BuildEnvironments are discovered is outside the scope of this class.
+
+ You must derive this class to teach Toaster how to operate in your own infrastructure.
+ We provide some specific BuildEnvironmentController classes that can be used either to
+ directly set-up Toaster infrastructure, or as a model for your own infrastructure set:
+
+ * Localhost controller will run the Toaster BE on the same account as the web server
+ (current user if you are using the the Django development web server)
+ on the local machine, with the "build/" directory under the "poky/" source checkout directory.
+ Bash is expected to be available.
+
+ * SSH controller will run the Toaster BE on a remote machine, where the current user
+ can connect without raise Exception("FIXME: implement")word (set up with either ssh-agent or raise Exception("FIXME: implement")phrase-less key authentication)
+
+ """
+ def __init__(self, be):
+ """ Takes a BuildEnvironment object as parameter that points to the settings of the BE.
+ """
+ self.be = be
+ self.connection = None
+
+ def startBBServer(self):
+ """ Starts a BB server with Toaster toasterui set up to record the builds, an no controlling UI.
+ After this method executes, self.be bbaddress/bbport MUST point to a running and free server,
+ and the bbstate MUST be updated to "started".
+ """
+ raise Exception("Must override in order to actually start the BB server")
+
+ def stopBBServer(self):
+ """ Stops the currently running BB server.
+ The bbstate MUST be updated to "stopped".
+ self.connection must be none.
+ """
+
+ def setLayers(self,ls):
+ """ Sets the layer variables in the config file, after validating local layer paths.
+ The layer paths must be in a list of BRLayer object
+ """
+ raise Exception("Must override setLayers")
+
+
+ def getBBController(self):
+ """ returns a BitbakeController to an already started server; this is the point where the server
+ starts if needed; or reconnects to the server if we can
+ """
+ if not self.connection:
+ self.startBBServer()
+ self.be.lock = BuildEnvironment.LOCK_RUNNING
+ self.be.save()
+
+ server = bb.server.xmlrpc.BitBakeXMLRPCClient()
+ server.initServer()
+ server.saveConnectionDetails("%s:%s" % (self.be.bbaddress, self.be.bbport))
+ self.connection = server.establishConnection([])
+
+ self.be.bbtoken = self.connection.transport.connection_token
+ self.be.save()
+
+ return BitbakeController(self.connection)
+
+ def getArtifact(path):
+ """ This call returns an artifact identified by the 'path'. How 'path' is interpreted as
+ up to the implementing BEC. The return MUST be a REST URL where a GET will actually return
+ the content of the artifact, e.g. for use as a "download link" in a web UI.
+ """
+ raise Exception("Must return the REST URL of the artifact")
+
+ def release(self):
+ """ This stops the server and releases any resources. After this point, all resources
+ are un-available for further reference
+ """
+ raise Exception("Must override BE release")
+
+class ShellCmdException(Exception):
+ pass
+
+class LocalhostBEController(BuildEnvironmentController):
+ """ Implementation of the BuildEnvironmentController for the localhost;
+ this controller manages the default build directory,
+ the server setup and system start and stop for the localhost-type build environment
+
+ The address field is used as working directory; if not set, the build/ directory
+ is created
+ """
+
+ def __init__(self, be):
+ super(LocalhostBEController, self).__init__(be)
+ from os.path import dirname as DN
+ self.cwd = DN(DN(DN(DN(DN(os.path.realpath(__file__))))))
+ if self.be.address is None or len(self.be.address) == 0:
+ self.be.address = "build"
+ self.be.save()
+ self.bwd = self.be.address
+ self.dburl = settings.getDATABASE_URL()
+
+ # transform relative paths to absolute ones
+ if not self.bwd.startswith("/"):
+ self.bwd = os.path.join(self.cwd, self.bwd)
+ self._createBE()
+
+ def _shellcmd(self, command):
+ p = subprocess.Popen(command, cwd=self.cwd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ (out,err) = p.communicate()
+ if p.returncode:
+ if len(err) == 0:
+ err = "command: %s" % command
+ else:
+ err = "command: %s \n%s" % (command, err)
+ raise ShellCmdException(err)
+ else:
+ return out
+
+ def _createBE(self):
+ assert self.cwd and os.path.exists(self.cwd)
+ self._shellcmd("bash -c \"source %s/oe-init-build-env %s\"" % (self.cwd, self.bwd))
+
+ def startBBServer(self):
+ assert self.cwd and os.path.exists(self.cwd)
+ print self._shellcmd("bash -c \"source %s/oe-init-build-env %s && DATABASE_URL=%s source toaster start noweb && sleep 1\"" % (self.cwd, self.bwd, self.dburl))
+ # FIXME unfortunate sleep 1 - we need to make sure that bbserver is started and the toaster ui is connected
+ # but since they start async without any return, we just wait a bit
+ print "Started server"
+ assert self.cwd and os.path.exists(self.bwd)
+ self.be.bbaddress = "localhost"
+ self.be.bbport = "8200"
+ self.be.bbstate = BuildEnvironment.SERVER_STARTED
+ self.be.save()
+
+ def stopBBServer(self):
+ assert self.cwd
+ print self._shellcmd("bash -c \"source %s/oe-init-build-env %s && %s source toaster stop\"" %
+ (self.cwd, self.bwd, (lambda: "" if self.be.bbtoken is None else "BBTOKEN=%s" % self.be.bbtoken)()))
+ self.be.bbstate = BuildEnvironment.SERVER_STOPPED
+ self.be.save()
+ print "Stopped server"
+
+ def setLayers(self, layers):
+ assert self.cwd is not None
+ layerconf = os.path.join(self.bwd, "conf/bblayers.conf")
+ if not os.path.exists(layerconf):
+ raise Exception("BE is not consistent: bblayers.conf file missing at ", layerconf)
+ return True
+
+ def release(self):
+ assert self.cwd and os.path.exists(self.bwd)
+ import shutil
+ shutil.rmtree(os.path.join(self.cwd, "build"))
+ assert not os.path.exists(self.bwd)
diff --git a/lib/toaster/bldcontrol/fixtures/initial_data.json b/lib/toaster/bldcontrol/fixtures/initial_data.json
index 21883ab..b2f12b3 100644
--- a/lib/toaster/bldcontrol/fixtures/initial_data.json
+++ b/lib/toaster/bldcontrol/fixtures/initial_data.json
@@ -1 +1 @@
-[{"pk": 1, "model": "bldcontrol.buildenvironment", "fields": {"updated": "2014-05-20T12:17:30Z", "created": "2014-05-20T12:17:30Z", "lock": 0, "bbstate": 0, "bbaddress": "", "betype": 0, "bbtoken": "", "address": "localhost", "bbport": -1}}]
+[{"pk": 1, "model": "bldcontrol.buildenvironment", "fields": {"updated": "2014-05-20T12:17:30Z", "created": "2014-05-20T12:17:30Z", "lock": 0, "bbstate": 0, "bbaddress": "", "betype": 0, "bbtoken": "", "address": "", "bbport": -1}}]
diff --git a/lib/toaster/bldcontrol/management/__init__.py b/lib/toaster/bldcontrol/management/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/toaster/bldcontrol/management/commands/__init__.py b/lib/toaster/bldcontrol/management/commands/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/toaster/bldcontrol/management/commands/runbuilds.py b/lib/toaster/bldcontrol/management/commands/runbuilds.py
new file mode 100644
index 0000000..dd8f84b
--- /dev/null
+++ b/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -0,0 +1,85 @@
+from django.core.management.base import NoArgsCommand, CommandError
+from django.db import transaction
+from orm.models import Build
+from bldcontrol.bbcontroller import getBuildEnvironmentController, ShellCmdException
+from bldcontrol.models import BuildRequest, BuildEnvironment
+import os
+
+class Command(NoArgsCommand):
+ args = ""
+ help = "Schedules and executes build requests as possible. Does not return (interrupt with Ctrl-C)"
+
+
+ @transaction.commit_on_success
+ def _selectBuildEnvironment(self):
+ bec = getBuildEnvironmentController(lock = BuildEnvironment.LOCK_FREE)
+ bec.be.lock = BuildEnvironment.LOCK_LOCK
+ bec.be.save()
+ return bec
+
+ @transaction.commit_on_success
+ def _selectBuildRequest(self):
+ br = BuildRequest.objects.filter(state = BuildRequest.REQ_QUEUED).order_by('pk')[0]
+ br.state = BuildRequest.REQ_INPROGRESS
+ br.save()
+ return br
+
+ def schedule(self):
+ try:
+ br = None
+ try:
+ # select the build environment and the request to build
+ br = self._selectBuildRequest()
+ except IndexError as e:
+ return
+ try:
+ bec = self._selectBuildEnvironment()
+ except IndexError as e:
+ # we could not find a BEC; postpone the BR
+ br.state = BuildRequest.REQ_QUEUED
+ br.save()
+ return
+
+ # set up the buid environment with the needed layers
+ print "Build %s, Environment %s" % (br, bec.be)
+ bec.setLayers(br.brlayer_set.all())
+
+ # get the bb server running
+ bbctrl = bec.getBBController()
+
+ # let toasterui that this is a managed build
+ bbctrl.setVariable("TOASTER_BRBE", "%d:%d" % (br.pk, bec.be.pk))
+
+ # set the build configuration
+ for variable in br.brvariable_set.all():
+ bbctrl.setVariable(variable.name, variable.value)
+
+ # trigger the build command
+ bbctrl.build(list(map(lambda x:x.target, br.brtarget_set.all())))
+
+ print "Build launched, exiting"
+ # disconnect from the server
+ bbctrl.disconnect()
+
+ # cleanup to be performed by toaster when the deed is done
+
+ except ShellCmdException as e:
+ import traceback
+ print " EE Error executing shell command\n", e
+ traceback.format_exc(e)
+
+ except Exception as e:
+ import traceback
+ traceback.print_exc()
+ raise e
+
+ def cleanup(self):
+ from django.utils import timezone
+ from datetime import timedelta
+ # environments locked for more than 30 seconds - they should be unlocked
+ BuildEnvironment.objects.filter(lock=BuildEnvironment.LOCK_LOCK).filter(updated__lt = timezone.now() - timedelta(seconds = 30)).update(lock = BuildEnvironment.LOCK_FREE)
+
+
+ def handle_noargs(self, **options):
+ self.cleanup()
+ self.schedule()
diff --git a/lib/toaster/bldcontrol/models.py b/lib/toaster/bldcontrol/models.py
index 11f487c..158874f 100644
--- a/lib/toaster/bldcontrol/models.py
+++ b/lib/toaster/bldcontrol/models.py
@@ -20,9 +20,11 @@ class BuildEnvironment(models.Model):
LOCK_FREE = 0
LOCK_LOCK = 1
+ LOCK_RUNNING = 2
LOCK_STATE = (
(LOCK_FREE, "free"),
(LOCK_LOCK, "lock"),
+ (LOCK_RUNNING, "running"),
)
address = models.CharField(max_length = 254)
diff --git a/lib/toaster/bldcontrol/tests.py b/lib/toaster/bldcontrol/tests.py
index 501deb7..ebe477d 100644
--- a/lib/toaster/bldcontrol/tests.py
+++ b/lib/toaster/bldcontrol/tests.py
@@ -7,10 +7,75 @@ Replace this with more appropriate tests for your application.
from django.test import TestCase
+from bldcontrol.bbcontroller import LocalhostBEController, BitbakeController
+from bldcontrol.models import BuildEnvironment, BuildRequest
+from bldcontrol.management.commands.runbuilds import Command
-class SimpleTest(TestCase):
- def test_basic_addition(self):
+import socket
+import subprocess
+
+class LocalhostBEControllerTests(TestCase):
+ def test_StartAndStopServer(self):
+ obe = BuildEnvironment.objects.create(lock = BuildEnvironment.LOCK_FREE, betype = BuildEnvironment.TYPE_LOCAL)
+ lbc = LocalhostBEController(obe)
+
+ # test start server and stop
+ self.assertTrue(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('localhost', 8200)), "Port already occupied")
+ lbc.startBBServer()
+ self.assertFalse(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('localhost', 8200)), "Server not answering")
+
+ lbc.stopBBServer()
+ self.assertTrue(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('localhost', 8200)), "Server not stopped")
+
+ # clean up
+ import subprocess
+ out, err = subprocess.Popen("netstat -tapn 2>/dev/null | grep 8200 | awk '{print $7}' | sort -fu | cut -d \"/\" -f 1 | grep -v -- - | tee /dev/fd/2 | xargs -r kill", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
+
+ self.assertTrue(err == '', "bitbake server pid %s not stopped" % err)
+
+ obe = BuildEnvironment.objects.create(lock = BuildEnvironment.LOCK_FREE, betype = BuildEnvironment.TYPE_LOCAL)
+ lbc = LocalhostBEController(obe)
+
+ bbc = lbc.getBBController()
+ self.assertTrue(isinstance(bbc, BitbakeController))
+ # test set variable
+ try:
+ bbc.setVariable
+ except Exception as e :
+ self.fail("setVariable raised %s", e)
+
+ lbc.stopBBServer()
+ out, err = subprocess.Popen("netstat -tapn 2>/dev/null | grep 8200 | awk '{print $7}' | sort -fu | cut -d \"/\" -f 1 | grep -v -- - | tee /dev/fd/2 | xargs -r kill", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
+ self.assertTrue(err == '', "bitbake server pid %s not stopped" % err)
+
+
+class RunBuildsCommandTests(TestCase):
+ def test_bec_select(self):
"""
- Tests that 1 + 1 always equals 2.
+ Tests that we can find and lock a build environment
"""
- self.assertEqual(1 + 1, 2)
+
+ obe = BuildEnvironment.objects.create(lock = BuildEnvironment.LOCK_FREE, betype = BuildEnvironment.TYPE_LOCAL)
+ command = Command()
+ bec = command._selectBuildEnvironment()
+
+ # make sure we select the object we've just built
+ self.assertTrue(bec.be.id == obe.id, "Environment is not properly selected")
+ # we have a locked environment
+ self.assertTrue(bec.be.lock == BuildEnvironment.LOCK_LOCK, "Environment is not locked")
+ # no more selections possible here
+ self.assertRaises(IndexError, command._selectBuildEnvironment)
+
+ def test_br_select(self):
+ from orm.models import Project
+ p, created = Project.objects.get_or_create(pk=1)
+ obr = BuildRequest.objects.create(state = BuildRequest.REQ_QUEUED, project = p)
+ command = Command()
+ br = command._selectBuildRequest()
+
+ # make sure we select the object we've just built
+ self.assertTrue(obr.id == br.id, "Request is not properly selected")
+ # we have a locked environment
+ self.assertTrue(br.state == BuildRequest.REQ_INPROGRESS, "Request is not updated")
+ # no more selections possible here
+ self.assertRaises(IndexError, command._selectBuildRequest)
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 6/6] toaster: build control functionality
@ 2014-06-12 13:52 ` Alex DAMIAN
0 siblings, 0 replies; 14+ messages in thread
From: Alex DAMIAN @ 2014-06-12 13:52 UTC (permalink / raw)
To: bitbake-devel, toaster; +Cc: Alexandru DAMIAN
From: Alexandru DAMIAN <alexandru.damian@intel.com>
We add the build control functionality to toaster.
* The bldcontrol application gains bbcontroller classes
that know how to manage a localhost build environment.
* The toaster UI now detects it is running under build
environment controller, and update the build controller
database and will shut down the bitbake server once
the build is complete.
* The toaster script can now run in standalone mode,
launching the build controller and the web interface instead
of just monitoring the build, as in the interactive mode.
* A fixture with the default build controller entry for
localhost is provided.
[YOCTO #5490]
[YOCTO #5491]
[YOCTO #5492]
[YOCTO #5493]
[YOCTO #5494]
[YOCTO #5537]
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
bin/toaster | 7 +-
lib/bb/ui/buildinfohelper.py | 10 +
lib/bb/ui/toasterui.py | 3 +
lib/toaster/bldcontrol/bbcontroller.py | 239 +++++++++++++++++++++
lib/toaster/bldcontrol/fixtures/initial_data.json | 2 +-
lib/toaster/bldcontrol/management/__init__.py | 0
.../bldcontrol/management/commands/__init__.py | 0
.../bldcontrol/management/commands/runbuilds.py | 85 ++++++++
lib/toaster/bldcontrol/models.py | 2 +
lib/toaster/bldcontrol/tests.py | 73 ++++++-
10 files changed, 413 insertions(+), 8 deletions(-)
create mode 100644 lib/toaster/bldcontrol/bbcontroller.py
create mode 100644 lib/toaster/bldcontrol/management/__init__.py
create mode 100644 lib/toaster/bldcontrol/management/commands/__init__.py
create mode 100644 lib/toaster/bldcontrol/management/commands/runbuilds.py
diff --git a/bin/toaster b/bin/toaster
index 80bda6d..01ffc7a 100755
--- a/bin/toaster
+++ b/bin/toaster
@@ -139,15 +139,16 @@ if [ -z "$ZSH_NAME" ] && [ `basename \"$0\"` = `basename \"$BASH_SOURCE\"` ]; th
webserverKillAll
RUNNING=0
}
- webserverStartAll || exit 1
+ webserverStartAll || (echo "Fail to start the web server, stopping" 1>&2 && exit 1)
xdg-open http://0.0.0.0:8000/ >/dev/null 2>&1 &
trap trap_ctrlc SIGINT
echo "Running. Stop with Ctrl-C"
while [ $RUNNING -gt 0 ]; do
- wait;
+ python $BBBASEDIR/lib/toaster/manage.py runbuilds
+ sleep 1
done
echo "**** Exit"
- exit 1
+ exit 0
fi
# We make sure we're running in the current shell and in a good environment
diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 58df687..77be7c7 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -916,6 +916,16 @@ class BuildInfoHelper(object):
self.internal_state['recipes'],
)
+ def store_build_done(self, br_id, be_id):
+ from bldcontrol.models import BuildEnvironment, BuildRequest
+ be = BuildEnvironment.objects.get(pk = be_id)
+ be.lock = BuildEnvironment.LOCK_LOCK
+ be.save()
+ br = BuildRequest.objects.get(pk = br_id)
+ br.state = BuildRequest.REQ_COMPLETED
+ br.build = self.internal_state['build']
+ br.save()
+
def _store_log_information(self, level, text):
log_information = {}
log_information['build'] = self.internal_state['build']
diff --git a/lib/bb/ui/toasterui.py b/lib/bb/ui/toasterui.py
index c1696b2..b1c80cc 100644
--- a/lib/bb/ui/toasterui.py
+++ b/lib/bb/ui/toasterui.py
@@ -228,8 +228,11 @@ def main(server, eventHandler, params ):
brbe = server.runCommand(["getVariable", "TOASTER_BRBE"])[0]
+ br_id, be_id = brbe.split(":")
# we start a new build info
if brbe is not None:
+ buildinfohelper.store_build_done(br_id, be_id)
+
print "we are under BuildEnvironment management - after the build, we exit"
server.terminateServer()
else:
diff --git a/lib/toaster/bldcontrol/bbcontroller.py b/lib/toaster/bldcontrol/bbcontroller.py
new file mode 100644
index 0000000..d2b2a23
--- /dev/null
+++ b/lib/toaster/bldcontrol/bbcontroller.py
@@ -0,0 +1,239 @@
+#
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+import os
+import sys
+import re
+from django.db import transaction
+from django.db.models import Q
+from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget
+import subprocess
+
+from toastermain import settings
+
+
+# load Bitbake components
+path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+sys.path.insert(0, path)
+import bb.server.xmlrpc
+
+class BitbakeController(object):
+ """ This is the basic class that controlls a bitbake server.
+ It is outside the scope of this class on how the server is started and aquired
+ """
+
+ def __init__(self, connection):
+ self.connection = connection
+
+ def _runCommand(self, command):
+ result, error = self.connection.connection.runCommand(command)
+ if error:
+ raise Exception(error)
+ return result
+
+ def disconnect(self):
+ return self.connection.removeClient()
+
+ def setVariable(self, name, value):
+ return self._runCommand(["setVariable", name, value])
+
+ def build(self, targets, task = None):
+ if task is None:
+ task = "build"
+ return self._runCommand(["buildTargets", targets, task])
+
+
+
+def getBuildEnvironmentController(**kwargs):
+ """ Gets you a BuildEnvironmentController that encapsulates a build environment,
+ based on the query dictionary sent in.
+
+ This is used to retrieve, for example, the currently running BE from inside
+ the toaster UI, or find a new BE to start a new build in it.
+
+ The return object MUST always be a BuildEnvironmentController.
+ """
+ be = BuildEnvironment.objects.filter(Q(**kwargs))[0]
+ if be.betype == BuildEnvironment.TYPE_LOCAL:
+ return LocalhostBEController(be)
+ elif be.betype == BuildEnvironment.TYPE_SSH:
+ return SSHBEController(be)
+ else:
+ raise Exception("FIXME: Implement BEC for type %s" % str(be.betype))
+
+
+
+class BuildEnvironmentController(object):
+ """ BuildEnvironmentController (BEC) is the abstract class that defines the operations that MUST
+ or SHOULD be supported by a Build Environment. It is used to establish the framework, and must
+ not be instantiated directly by the user.
+
+ Use the "getBuildEnvironmentController()" function to get a working BEC for your remote.
+
+ How the BuildEnvironments are discovered is outside the scope of this class.
+
+ You must derive this class to teach Toaster how to operate in your own infrastructure.
+ We provide some specific BuildEnvironmentController classes that can be used either to
+ directly set-up Toaster infrastructure, or as a model for your own infrastructure set:
+
+ * Localhost controller will run the Toaster BE on the same account as the web server
+ (current user if you are using the the Django development web server)
+ on the local machine, with the "build/" directory under the "poky/" source checkout directory.
+ Bash is expected to be available.
+
+ * SSH controller will run the Toaster BE on a remote machine, where the current user
+ can connect without raise Exception("FIXME: implement")word (set up with either ssh-agent or raise Exception("FIXME: implement")phrase-less key authentication)
+
+ """
+ def __init__(self, be):
+ """ Takes a BuildEnvironment object as parameter that points to the settings of the BE.
+ """
+ self.be = be
+ self.connection = None
+
+ def startBBServer(self):
+ """ Starts a BB server with Toaster toasterui set up to record the builds, an no controlling UI.
+ After this method executes, self.be bbaddress/bbport MUST point to a running and free server,
+ and the bbstate MUST be updated to "started".
+ """
+ raise Exception("Must override in order to actually start the BB server")
+
+ def stopBBServer(self):
+ """ Stops the currently running BB server.
+ The bbstate MUST be updated to "stopped".
+ self.connection must be none.
+ """
+
+ def setLayers(self,ls):
+ """ Sets the layer variables in the config file, after validating local layer paths.
+ The layer paths must be in a list of BRLayer object
+ """
+ raise Exception("Must override setLayers")
+
+
+ def getBBController(self):
+ """ returns a BitbakeController to an already started server; this is the point where the server
+ starts if needed; or reconnects to the server if we can
+ """
+ if not self.connection:
+ self.startBBServer()
+ self.be.lock = BuildEnvironment.LOCK_RUNNING
+ self.be.save()
+
+ server = bb.server.xmlrpc.BitBakeXMLRPCClient()
+ server.initServer()
+ server.saveConnectionDetails("%s:%s" % (self.be.bbaddress, self.be.bbport))
+ self.connection = server.establishConnection([])
+
+ self.be.bbtoken = self.connection.transport.connection_token
+ self.be.save()
+
+ return BitbakeController(self.connection)
+
+ def getArtifact(path):
+ """ This call returns an artifact identified by the 'path'. How 'path' is interpreted as
+ up to the implementing BEC. The return MUST be a REST URL where a GET will actually return
+ the content of the artifact, e.g. for use as a "download link" in a web UI.
+ """
+ raise Exception("Must return the REST URL of the artifact")
+
+ def release(self):
+ """ This stops the server and releases any resources. After this point, all resources
+ are un-available for further reference
+ """
+ raise Exception("Must override BE release")
+
+class ShellCmdException(Exception):
+ pass
+
+class LocalhostBEController(BuildEnvironmentController):
+ """ Implementation of the BuildEnvironmentController for the localhost;
+ this controller manages the default build directory,
+ the server setup and system start and stop for the localhost-type build environment
+
+ The address field is used as working directory; if not set, the build/ directory
+ is created
+ """
+
+ def __init__(self, be):
+ super(LocalhostBEController, self).__init__(be)
+ from os.path import dirname as DN
+ self.cwd = DN(DN(DN(DN(DN(os.path.realpath(__file__))))))
+ if self.be.address is None or len(self.be.address) == 0:
+ self.be.address = "build"
+ self.be.save()
+ self.bwd = self.be.address
+ self.dburl = settings.getDATABASE_URL()
+
+ # transform relative paths to absolute ones
+ if not self.bwd.startswith("/"):
+ self.bwd = os.path.join(self.cwd, self.bwd)
+ self._createBE()
+
+ def _shellcmd(self, command):
+ p = subprocess.Popen(command, cwd=self.cwd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ (out,err) = p.communicate()
+ if p.returncode:
+ if len(err) == 0:
+ err = "command: %s" % command
+ else:
+ err = "command: %s \n%s" % (command, err)
+ raise ShellCmdException(err)
+ else:
+ return out
+
+ def _createBE(self):
+ assert self.cwd and os.path.exists(self.cwd)
+ self._shellcmd("bash -c \"source %s/oe-init-build-env %s\"" % (self.cwd, self.bwd))
+
+ def startBBServer(self):
+ assert self.cwd and os.path.exists(self.cwd)
+ print self._shellcmd("bash -c \"source %s/oe-init-build-env %s && DATABASE_URL=%s source toaster start noweb && sleep 1\"" % (self.cwd, self.bwd, self.dburl))
+ # FIXME unfortunate sleep 1 - we need to make sure that bbserver is started and the toaster ui is connected
+ # but since they start async without any return, we just wait a bit
+ print "Started server"
+ assert self.cwd and os.path.exists(self.bwd)
+ self.be.bbaddress = "localhost"
+ self.be.bbport = "8200"
+ self.be.bbstate = BuildEnvironment.SERVER_STARTED
+ self.be.save()
+
+ def stopBBServer(self):
+ assert self.cwd
+ print self._shellcmd("bash -c \"source %s/oe-init-build-env %s && %s source toaster stop\"" %
+ (self.cwd, self.bwd, (lambda: "" if self.be.bbtoken is None else "BBTOKEN=%s" % self.be.bbtoken)()))
+ self.be.bbstate = BuildEnvironment.SERVER_STOPPED
+ self.be.save()
+ print "Stopped server"
+
+ def setLayers(self, layers):
+ assert self.cwd is not None
+ layerconf = os.path.join(self.bwd, "conf/bblayers.conf")
+ if not os.path.exists(layerconf):
+ raise Exception("BE is not consistent: bblayers.conf file missing at ", layerconf)
+ return True
+
+ def release(self):
+ assert self.cwd and os.path.exists(self.bwd)
+ import shutil
+ shutil.rmtree(os.path.join(self.cwd, "build"))
+ assert not os.path.exists(self.bwd)
diff --git a/lib/toaster/bldcontrol/fixtures/initial_data.json b/lib/toaster/bldcontrol/fixtures/initial_data.json
index 21883ab..b2f12b3 100644
--- a/lib/toaster/bldcontrol/fixtures/initial_data.json
+++ b/lib/toaster/bldcontrol/fixtures/initial_data.json
@@ -1 +1 @@
-[{"pk": 1, "model": "bldcontrol.buildenvironment", "fields": {"updated": "2014-05-20T12:17:30Z", "created": "2014-05-20T12:17:30Z", "lock": 0, "bbstate": 0, "bbaddress": "", "betype": 0, "bbtoken": "", "address": "localhost", "bbport": -1}}]
+[{"pk": 1, "model": "bldcontrol.buildenvironment", "fields": {"updated": "2014-05-20T12:17:30Z", "created": "2014-05-20T12:17:30Z", "lock": 0, "bbstate": 0, "bbaddress": "", "betype": 0, "bbtoken": "", "address": "", "bbport": -1}}]
diff --git a/lib/toaster/bldcontrol/management/__init__.py b/lib/toaster/bldcontrol/management/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/toaster/bldcontrol/management/commands/__init__.py b/lib/toaster/bldcontrol/management/commands/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/toaster/bldcontrol/management/commands/runbuilds.py b/lib/toaster/bldcontrol/management/commands/runbuilds.py
new file mode 100644
index 0000000..dd8f84b
--- /dev/null
+++ b/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -0,0 +1,85 @@
+from django.core.management.base import NoArgsCommand, CommandError
+from django.db import transaction
+from orm.models import Build
+from bldcontrol.bbcontroller import getBuildEnvironmentController, ShellCmdException
+from bldcontrol.models import BuildRequest, BuildEnvironment
+import os
+
+class Command(NoArgsCommand):
+ args = ""
+ help = "Schedules and executes build requests as possible. Does not return (interrupt with Ctrl-C)"
+
+
+ @transaction.commit_on_success
+ def _selectBuildEnvironment(self):
+ bec = getBuildEnvironmentController(lock = BuildEnvironment.LOCK_FREE)
+ bec.be.lock = BuildEnvironment.LOCK_LOCK
+ bec.be.save()
+ return bec
+
+ @transaction.commit_on_success
+ def _selectBuildRequest(self):
+ br = BuildRequest.objects.filter(state = BuildRequest.REQ_QUEUED).order_by('pk')[0]
+ br.state = BuildRequest.REQ_INPROGRESS
+ br.save()
+ return br
+
+ def schedule(self):
+ try:
+ br = None
+ try:
+ # select the build environment and the request to build
+ br = self._selectBuildRequest()
+ except IndexError as e:
+ return
+ try:
+ bec = self._selectBuildEnvironment()
+ except IndexError as e:
+ # we could not find a BEC; postpone the BR
+ br.state = BuildRequest.REQ_QUEUED
+ br.save()
+ return
+
+ # set up the buid environment with the needed layers
+ print "Build %s, Environment %s" % (br, bec.be)
+ bec.setLayers(br.brlayer_set.all())
+
+ # get the bb server running
+ bbctrl = bec.getBBController()
+
+ # let toasterui that this is a managed build
+ bbctrl.setVariable("TOASTER_BRBE", "%d:%d" % (br.pk, bec.be.pk))
+
+ # set the build configuration
+ for variable in br.brvariable_set.all():
+ bbctrl.setVariable(variable.name, variable.value)
+
+ # trigger the build command
+ bbctrl.build(list(map(lambda x:x.target, br.brtarget_set.all())))
+
+ print "Build launched, exiting"
+ # disconnect from the server
+ bbctrl.disconnect()
+
+ # cleanup to be performed by toaster when the deed is done
+
+ except ShellCmdException as e:
+ import traceback
+ print " EE Error executing shell command\n", e
+ traceback.format_exc(e)
+
+ except Exception as e:
+ import traceback
+ traceback.print_exc()
+ raise e
+
+ def cleanup(self):
+ from django.utils import timezone
+ from datetime import timedelta
+ # environments locked for more than 30 seconds - they should be unlocked
+ BuildEnvironment.objects.filter(lock=BuildEnvironment.LOCK_LOCK).filter(updated__lt = timezone.now() - timedelta(seconds = 30)).update(lock = BuildEnvironment.LOCK_FREE)
+
+
+ def handle_noargs(self, **options):
+ self.cleanup()
+ self.schedule()
diff --git a/lib/toaster/bldcontrol/models.py b/lib/toaster/bldcontrol/models.py
index 11f487c..158874f 100644
--- a/lib/toaster/bldcontrol/models.py
+++ b/lib/toaster/bldcontrol/models.py
@@ -20,9 +20,11 @@ class BuildEnvironment(models.Model):
LOCK_FREE = 0
LOCK_LOCK = 1
+ LOCK_RUNNING = 2
LOCK_STATE = (
(LOCK_FREE, "free"),
(LOCK_LOCK, "lock"),
+ (LOCK_RUNNING, "running"),
)
address = models.CharField(max_length = 254)
diff --git a/lib/toaster/bldcontrol/tests.py b/lib/toaster/bldcontrol/tests.py
index 501deb7..ebe477d 100644
--- a/lib/toaster/bldcontrol/tests.py
+++ b/lib/toaster/bldcontrol/tests.py
@@ -7,10 +7,75 @@ Replace this with more appropriate tests for your application.
from django.test import TestCase
+from bldcontrol.bbcontroller import LocalhostBEController, BitbakeController
+from bldcontrol.models import BuildEnvironment, BuildRequest
+from bldcontrol.management.commands.runbuilds import Command
-class SimpleTest(TestCase):
- def test_basic_addition(self):
+import socket
+import subprocess
+
+class LocalhostBEControllerTests(TestCase):
+ def test_StartAndStopServer(self):
+ obe = BuildEnvironment.objects.create(lock = BuildEnvironment.LOCK_FREE, betype = BuildEnvironment.TYPE_LOCAL)
+ lbc = LocalhostBEController(obe)
+
+ # test start server and stop
+ self.assertTrue(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('localhost', 8200)), "Port already occupied")
+ lbc.startBBServer()
+ self.assertFalse(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('localhost', 8200)), "Server not answering")
+
+ lbc.stopBBServer()
+ self.assertTrue(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('localhost', 8200)), "Server not stopped")
+
+ # clean up
+ import subprocess
+ out, err = subprocess.Popen("netstat -tapn 2>/dev/null | grep 8200 | awk '{print $7}' | sort -fu | cut -d \"/\" -f 1 | grep -v -- - | tee /dev/fd/2 | xargs -r kill", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
+
+ self.assertTrue(err == '', "bitbake server pid %s not stopped" % err)
+
+ obe = BuildEnvironment.objects.create(lock = BuildEnvironment.LOCK_FREE, betype = BuildEnvironment.TYPE_LOCAL)
+ lbc = LocalhostBEController(obe)
+
+ bbc = lbc.getBBController()
+ self.assertTrue(isinstance(bbc, BitbakeController))
+ # test set variable
+ try:
+ bbc.setVariable
+ except Exception as e :
+ self.fail("setVariable raised %s", e)
+
+ lbc.stopBBServer()
+ out, err = subprocess.Popen("netstat -tapn 2>/dev/null | grep 8200 | awk '{print $7}' | sort -fu | cut -d \"/\" -f 1 | grep -v -- - | tee /dev/fd/2 | xargs -r kill", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
+ self.assertTrue(err == '', "bitbake server pid %s not stopped" % err)
+
+
+class RunBuildsCommandTests(TestCase):
+ def test_bec_select(self):
"""
- Tests that 1 + 1 always equals 2.
+ Tests that we can find and lock a build environment
"""
- self.assertEqual(1 + 1, 2)
+
+ obe = BuildEnvironment.objects.create(lock = BuildEnvironment.LOCK_FREE, betype = BuildEnvironment.TYPE_LOCAL)
+ command = Command()
+ bec = command._selectBuildEnvironment()
+
+ # make sure we select the object we've just built
+ self.assertTrue(bec.be.id == obe.id, "Environment is not properly selected")
+ # we have a locked environment
+ self.assertTrue(bec.be.lock == BuildEnvironment.LOCK_LOCK, "Environment is not locked")
+ # no more selections possible here
+ self.assertRaises(IndexError, command._selectBuildEnvironment)
+
+ def test_br_select(self):
+ from orm.models import Project
+ p, created = Project.objects.get_or_create(pk=1)
+ obr = BuildRequest.objects.create(state = BuildRequest.REQ_QUEUED, project = p)
+ command = Command()
+ br = command._selectBuildRequest()
+
+ # make sure we select the object we've just built
+ self.assertTrue(obr.id == br.id, "Request is not properly selected")
+ # we have a locked environment
+ self.assertTrue(br.state == BuildRequest.REQ_INPROGRESS, "Request is not updated")
+ # no more selections possible here
+ self.assertRaises(IndexError, command._selectBuildRequest)
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
end of thread, other threads:[~2014-06-12 13:52 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-06-12 13:51 [PATCH 0/6] toaster patchset: localhost build control Alex DAMIAN
2014-06-12 13:51 ` Alex DAMIAN
2014-06-12 13:51 ` [PATCH 1/6] toaster: initial bldcontrol application Alex DAMIAN
2014-06-12 13:51 ` Alex DAMIAN
2014-06-12 13:51 ` [PATCH 2/6] toaster: create models for bldcontrol and enable it Alex DAMIAN
2014-06-12 13:51 ` Alex DAMIAN
2014-06-12 13:51 ` [PATCH 3/6] toaster: add function to get the database url Alex DAMIAN
2014-06-12 13:51 ` Alex DAMIAN
2014-06-12 13:52 ` [PATCH 4/6] xmlrpc: add support for token reusing Alex DAMIAN
2014-06-12 13:52 ` Alex DAMIAN
2014-06-12 13:52 ` [PATCH 5/6] toastergui: fix built ETA calculation Alex DAMIAN
2014-06-12 13:52 ` Alex DAMIAN
2014-06-12 13:52 ` [PATCH 6/6] toaster: build control functionality Alex DAMIAN
2014-06-12 13:52 ` Alex DAMIAN
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.