From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FA22C4167B for ; Mon, 11 Dec 2023 16:47:29 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.587.1702313244767291255 for ; Mon, 11 Dec 2023 08:47:24 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=VTDRUYDN; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: marlon.rodriguez-garcia@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 28F3E9C284E; Mon, 11 Dec 2023 11:47:24 -0500 (EST) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id w9RFCbyEe9IR; Mon, 11 Dec 2023 11:47:18 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id C1EE69C3679; Mon, 11 Dec 2023 11:47:18 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com C1EE69C3679 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1702313238; bh=9D0lARxUCAt19PcRkMKwojs70ROw5nGxVNpK4PSB4dc=; h=From:To:Date:Message-Id:MIME-Version; b=VTDRUYDNJwkYjBmifpt69TGQQtq3e+pKE+DOda398luqByAoojIi6ovMMzgrP2YCs yW89vGAp//quGuqolozitDF8J8O77/SLyodIplzQJkql7mEZmleF0u1+BUCJHKMxBI dA3GkzWfpohPDknGnkQX98UvA6594N4+8g6RCsxtAVKd01NJLj+NU2iYkRmc7GJxaU 1m/zGHHD4D8bSOSV1FN3sRqyDURCtdLF24+57t+xEu8kQYzi7+xxDO1+Pmrg8eRyQ/ ckhHiTOvHufu3lZdv+5TrTAViEbynOq499c9AqC6zpBrENqbXA2Fn2pV48UhDY8e2n lr/hZf34ia2aQ== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id uzJLIEVOb-sD; Mon, 11 Dec 2023 11:47:18 -0500 (EST) Received: from savoirfairelinux.ht.home (modemcable141.201-58-74.mc.videotron.ca [74.58.201.141]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 8A4749C3502; Mon, 11 Dec 2023 11:47:18 -0500 (EST) From: Marlon Rodriguez Garcia To: bitbake-devel@lists.openembedded.org, toaster@lists.yoctoproject.org Cc: Marlon Rodriguez Garcia Subject: [toaster][PATCHv6 1/1] toaster: Added new feature to import eventlogs from command line into toaster using replay functionality Date: Mon, 11 Dec 2023 11:47:05 -0500 Message-Id: <20231211164705.34450-2-marlon.rodriguez-garcia@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231211164705.34450-1-marlon.rodriguez-garcia@savoirfairelinux.com> References: <20231211164705.34450-1-marlon.rodriguez-garcia@savoirfairelinux.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 11 Dec 2023 16:47:29 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/toaster/message/6077 Added a new button on the base template to access a new template. Added a model register the information on the builds and generate access = links Added a form to include the option to load specific files Added jquery and ajax functions to block screen and redirect to build pag= e when import eventlogs is trigger Added a new button on landing page linked to import build page, and set m= in-height of buttons in landing page for uniformity Removed test assertion to check command line build in content, because ne= w button contains text Updated toaster_eventreplay to use library Fix test in test_layerdetails_page Rebased from master This feature uses the value from the variable BB_DEFAULT_EVENTLOG to read= the files created by bitbake Exclude listing of files that don't contain the allvariables definitions = used to replay builds This part of the feature should be revisited. Over a long period of time,= the BB_DEFAULT_EVENTLOG will exponentially increase the size of the log file and cause bottleneck= s when importing. Signed-off-by: Marlon Rodriguez Garcia --- bin/toaster-eventreplay | 74 +------ lib/bb/ui/eventreplay.py | 86 ++++++++ lib/bb/ui/toasterui.py | 2 +- .../orm/migrations/0021_eventlogsimports.py | 22 ++ lib/toaster/orm/models.py | 9 + .../tests/browser/test_landing_page.py | 2 - .../tests/browser/test_layerdetails_page.py | 2 +- lib/toaster/toastergui/forms.py | 14 ++ lib/toaster/toastergui/static/css/default.css | 28 +++ lib/toaster/toastergui/templates/base.html | 3 +- .../templates/command_line_builds.html | 198 ++++++++++++++++++ lib/toaster/toastergui/templates/landing.html | 10 +- lib/toaster/toastergui/urls.py | 1 + lib/toaster/toastergui/views.py | 173 ++++++++++++++- 14 files changed, 543 insertions(+), 81 deletions(-) create mode 100644 lib/bb/ui/eventreplay.py create mode 100644 lib/toaster/orm/migrations/0021_eventlogsimports.py create mode 100644 lib/toaster/toastergui/forms.py create mode 100644 lib/toaster/toastergui/templates/command_line_builds.= html diff --git a/bin/toaster-eventreplay b/bin/toaster-eventreplay index f137c71d..74a31932 100755 --- a/bin/toaster-eventreplay +++ b/bin/toaster-eventreplay @@ -30,77 +30,7 @@ sys.path.insert(0, join(dirname(dirname(abspath(__file= __))), 'lib')) =20 import bb.cooker from bb.ui import toasterui - -class EventPlayer: - """Emulate a connection to a bitbake server.""" - - def __init__(self, eventfile, variables): - self.eventfile =3D eventfile - self.variables =3D variables - self.eventmask =3D [] - - def waitEvent(self, _timeout): - """Read event from the file.""" - line =3D self.eventfile.readline().strip() - if not line: - return - try: - decodedline =3D json.loads(line) - if 'allvariables' in decodedline: - self.variables =3D decodedline['allvariables'] - return - if not 'vars' in decodedline: - raise ValueError - event_str =3D decodedline['vars'].encode('utf-8') - event =3D pickle.loads(codecs.decode(event_str, 'base64')) - event_name =3D "%s.%s" % (event.__module__, event.__class__.= __name__) - if event_name not in self.eventmask: - return - return event - except ValueError as err: - print("Failed loading ", line) - raise err - - def runCommand(self, command_line): - """Emulate running a command on the server.""" - name =3D command_line[0] - - if name =3D=3D "getVariable": - var_name =3D command_line[1] - variable =3D self.variables.get(var_name) - if variable: - return variable['v'], None - return None, "Missing variable %s" % var_name - - elif name =3D=3D "getAllKeysWithFlags": - dump =3D {} - flaglist =3D command_line[1] - for key, val in self.variables.items(): - try: - if not key.startswith("__"): - dump[key] =3D { - 'v': val['v'], - 'history' : val['history'], - } - for flag in flaglist: - dump[key][flag] =3D val[flag] - except Exception as err: - print(err) - return (dump, None) - - elif name =3D=3D 'setEventMask': - self.eventmask =3D command_line[-1] - return True, None - - else: - raise Exception("Command %s not implemented" % command_line[= 0]) - - def getEventHandle(self): - """ - This method is called by toasterui. - The return value is passed to self.runCommand but not used there= . - """ - pass +from bb.ui import eventreplay =20 def main(argv): with open(argv[-1]) as eventfile: @@ -116,7 +46,7 @@ def main(argv): sys.exit("Cannot find allvariables entry in event log file %= s" % argv[-1]) eventfile.seek(0) params =3D namedtuple('ConfigParams', ['observe_only'])(True) - player =3D EventPlayer(eventfile, variables) + player =3D eventreplay.EventPlayer(eventfile, variables) =20 return toasterui.main(player, player, params) =20 diff --git a/lib/bb/ui/eventreplay.py b/lib/bb/ui/eventreplay.py new file mode 100644 index 00000000..d62ecbfa --- /dev/null +++ b/lib/bb/ui/eventreplay.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-only +# +# This file re-uses code spread throughout other Bitbake source files. +# As such, all other copyrights belong to their own right holders. +# + + +import os +import sys +import json +import pickle +import codecs + + +class EventPlayer: + """Emulate a connection to a bitbake server.""" + + def __init__(self, eventfile, variables): + self.eventfile =3D eventfile + self.variables =3D variables + self.eventmask =3D [] + + def waitEvent(self, _timeout): + """Read event from the file.""" + line =3D self.eventfile.readline().strip() + if not line: + return + try: + decodedline =3D json.loads(line) + if 'allvariables' in decodedline: + self.variables =3D decodedline['allvariables'] + return + if not 'vars' in decodedline: + raise ValueError + event_str =3D decodedline['vars'].encode('utf-8') + event =3D pickle.loads(codecs.decode(event_str, 'base64')) + event_name =3D "%s.%s" % (event.__module__, event.__class__.= __name__) + if event_name not in self.eventmask: + return + return event + except ValueError as err: + print("Failed loading ", line) + raise err + + def runCommand(self, command_line): + """Emulate running a command on the server.""" + name =3D command_line[0] + + if name =3D=3D "getVariable": + var_name =3D command_line[1] + variable =3D self.variables.get(var_name) + if variable: + return variable['v'], None + return None, "Missing variable %s" % var_name + + elif name =3D=3D "getAllKeysWithFlags": + dump =3D {} + flaglist =3D command_line[1] + for key, val in self.variables.items(): + try: + if not key.startswith("__"): + dump[key] =3D { + 'v': val['v'], + 'history' : val['history'], + } + for flag in flaglist: + dump[key][flag] =3D val[flag] + except Exception as err: + print(err) + return (dump, None) + + elif name =3D=3D 'setEventMask': + self.eventmask =3D command_line[-1] + return True, None + + else: + raise Exception("Command %s not implemented" % command_line[= 0]) + + def getEventHandle(self): + """ + This method is called by toasterui. + The return value is passed to self.runCommand but not used there= . + """ + pass diff --git a/lib/bb/ui/toasterui.py b/lib/bb/ui/toasterui.py index ec5bd4f1..6bd21f18 100644 --- a/lib/bb/ui/toasterui.py +++ b/lib/bb/ui/toasterui.py @@ -385,7 +385,7 @@ def main(server, eventHandler, params): main.shutdown =3D 1 =20 logger.info("ToasterUI build done, brbe: %s", brbe) - continue + break =20 if isinstance(event, (bb.command.CommandCompleted, bb.command.CommandFailed, diff --git a/lib/toaster/orm/migrations/0021_eventlogsimports.py b/lib/to= aster/orm/migrations/0021_eventlogsimports.py new file mode 100644 index 00000000..328eb575 --- /dev/null +++ b/lib/toaster/orm/migrations/0021_eventlogsimports.py @@ -0,0 +1,22 @@ +# Generated by Django 4.2.5 on 2023-11-23 18:44 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('orm', '0020_models_bigautofield'), + ] + + operations =3D [ + migrations.CreateModel( + name=3D'EventLogsImports', + fields=3D[ + ('id', models.BigAutoField(auto_created=3DTrue, primary_= key=3DTrue, serialize=3DFalse, verbose_name=3D'ID')), + ('name', models.CharField(max_length=3D255)), + ('imported', models.BooleanField(default=3DFalse)), + ('build_id', models.IntegerField(blank=3DTrue, null=3DTr= ue)), + ], + ), + ] diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py index 1098ad3f..19c96862 100644 --- a/lib/toaster/orm/models.py +++ b/lib/toaster/orm/models.py @@ -1868,6 +1868,15 @@ class Distro(models.Model): def __unicode__(self): return "Distro " + self.name + "(" + self.description + ")" =20 +class EventLogsImports(models.Model): + name =3D models.CharField(max_length=3D255) + imported =3D models.BooleanField(default=3DFalse) + build_id =3D models.IntegerField(blank=3DTrue, null=3DTrue) + + def __str__(self): + return self.name + + django.db.models.signals.post_save.connect(invalidate_cache) django.db.models.signals.post_delete.connect(invalidate_cache) django.db.models.signals.m2m_changed.connect(invalidate_cache) diff --git a/lib/toaster/tests/browser/test_landing_page.py b/lib/toaster= /tests/browser/test_landing_page.py index f6649a3d..8fe5fea4 100644 --- a/lib/toaster/tests/browser/test_landing_page.py +++ b/lib/toaster/tests/browser/test_landing_page.py @@ -219,5 +219,3 @@ class TestLandingPage(SeleniumTestCase): content =3D self.get_page_source() self.assertTrue(self.PROJECT_NAME in content, 'should show builds for project %s' % self.PROJE= CT_NAME) - self.assertFalse(self.CLI_BUILDS_PROJECT_NAME in content, - 'should not show builds for cli project') diff --git a/lib/toaster/tests/browser/test_layerdetails_page.py b/lib/to= aster/tests/browser/test_layerdetails_page.py index 367c6179..05ee88b0 100644 --- a/lib/toaster/tests/browser/test_layerdetails_page.py +++ b/lib/toaster/tests/browser/test_layerdetails_page.py @@ -108,7 +108,7 @@ class TestLayerDetailsPage(SeleniumTestCase): =20 self.wait_until_visible("#save-changes-for-switch", poll=3D3) btn_save_chg_for_switch =3D self.find("#save-changes-for-switch"= ) - btn_save_chg_for_switch.click() + self.driver.execute_script("arguments[0].click();", btn_save_chg= _for_switch) self.wait_until_visible("#edit-layer-source") =20 # Refresh the page to see if the new values are returned diff --git a/lib/toaster/toastergui/forms.py b/lib/toaster/toastergui/for= ms.py new file mode 100644 index 00000000..10c7ac40 --- /dev/null +++ b/lib/toaster/toastergui/forms.py @@ -0,0 +1,14 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# BitBake Toaster UI tests implementation +# +# Copyright (C) 2023 Savoir-faire Linux +# +# SPDX-License-Identifier: GPL-2.0-only +# + +from django import forms +from django.core.validators import FileExtensionValidator + =20 +class LoadFileForm(forms.Form): + eventlog_file =3D forms.FileField(widget=3Dforms.FileInput(attrs=3D{= 'accept': '.json'})) diff --git a/lib/toaster/toastergui/static/css/default.css b/lib/toaster/= toastergui/static/css/default.css index 5cd7e211..284355e7 100644 --- a/lib/toaster/toastergui/static/css/default.css +++ b/lib/toaster/toastergui/static/css/default.css @@ -367,3 +367,31 @@ h2.panel-title { font-size: 30px; } } } /* End copied in from newer version of Font-Awesome 4.3.0 */ + + +#overlay { + display: flex; + position: fixed; + top: 0; + left: 0; + width: 100%; + height: 100%; + background-color: rgba(0, 0, 0, 0.7); + align-items: center; + justify-content: center; + z-index: 999; +} + +.spinner { + border: 6px solid rgba(255, 255, 255, 0.3); + border-radius: 50%; + border-top: 6px solid #3498db; + width: 50px; + height: 50px; + animation: spin 1s linear infinite; +} + +@keyframes spin { + 0% { transform: rotate(0deg); } + 100% { transform: rotate(360deg); } +} diff --git a/lib/toaster/toastergui/templates/base.html b/lib/toaster/toa= stergui/templates/base.html index 041448d1..e90be696 100644 --- a/lib/toaster/toastergui/templates/base.html +++ b/lib/toaster/toastergui/templates/base.html @@ -132,7 +132,8 @@ {% if project_enable %} New project {% endif %} - + Import command line builds + =20 diff --git a/lib/toaster/toastergui/templates/command_line_builds.html b/= lib/toaster/toastergui/templates/command_line_builds.html new file mode 100644 index 00000000..95944c74 --- /dev/null +++ b/lib/toaster/toastergui/templates/command_line_builds.html @@ -0,0 +1,198 @@ +{% extends "base.html" %} +{% load projecttags %} +{% load humanize %} + +{% block title %} Import Builds from eventlogs - Toaster {% endblock %} + +{% block pagecontent %} + +
+
+
+
+
+
+
+
+
+
+
+
+

Import command line builds

+
+ {% if import_all %} +
+ +
=20 + {% endif %} =20 +
+
+ {% if messages %} +
+ {% for message in messages %} +
{{message}}
+ {%endfor%} +
+ {% endif %} +
+

Import eventlog= file

+
=20 + {% csrf_token %}=20 +
+ +
{{ form.eventlog_fi= le}}
=20 +
+
+
=20 + +
+
+
+ +
+ +
+
+

Eventlogs from existing build directory:= + + + + + + +

+ {% if files %} +
+ + + + + + + + + + {% for file in files %} + + {{file.name}} + + + + + {% endfor%} + +
NameSizeAction
{{file.size|filesizeformat}} + {% if file.imported =3D=3D True = and file.build_id is not None %} + Build Details + {% elif request.session.file =3D= =3D file.name or request.session.all_builds %} + + + + {%else%} + + + + {%endif%} +
+
+ {% else %} +
+
Sorry - no files= found
+
+ {%endif%} +
+
+
+
+ + + + +{% endblock %} diff --git a/lib/toaster/toastergui/templates/landing.html b/lib/toaster/= toastergui/templates/landing.html index 22bbed69..589ee226 100644 --- a/lib/toaster/toastergui/templates/landing.html +++ b/lib/toaster/toastergui/templates/landing.html @@ -15,7 +15,7 @@

A web interface to OpenEmbedded and BitBake, the Yocto P= roject build system.

=20

- + Toaster is ready to capture your command line builds

@@ -23,7 +23,7 @@ {% if lvs_nos %} {% if project_enable %}

- + Create your first Toaster project to run manage builds

@@ -42,6 +42,12 @@ {% endif %} =20 +

+ + Import command line event logs from build directory + +

+