[PATCH 0/6] Update the web interface to the latest versions of foundation components

[PATCH 0/6] Update the web interface to the latest versions of foundation components

[Lucas Meneghel Rodrigues]

Update the web interface to use Django 1.3, fixed the remaining problems with
custom SQL injection code, resolving problems with the scheduler. Also, tested
the latest versions of gwt and gwt incubator, bumping the requirements on the
build_externals code, making the web interface to run on the latest stable
versions of each foundation component (Django, gwt and gwt-incubator).

Lucas Meneghel Rodrigues (6):
  Update admin site URL dispatcher to Django 1.3
  afe/readonly connection: Catch AttributeError exceptions
  Adding afe custom database engine
  Change web app to use new DB engine
  Removing get_from_clause() from custom query class
  external_packages: Bump version number requirements for web deps

 frontend/afe/model_logic.py               |   15 -------------
 frontend/afe/readonly_connection.py       |   10 ++++++++-
 frontend/db/backends/afe/base.py          |   22 +++++++++++++++++++
 frontend/db/backends/afe/compiler.py      |   32 +++++++++++++++++++++++++++++
 frontend/db/backends/afe/creation.py      |    1 +
 frontend/db/backends/afe/introspection.py |    1 +
 frontend/db/backends/afe/validation.py    |    1 +
 frontend/settings.py                      |    3 +-
 frontend/urls.py                          |    2 +-
 utils/external_packages.py                |   12 +++++-----
 10 files changed, 75 insertions(+), 24 deletions(-)
 create mode 100644 frontend/db/__init__.py
 create mode 100644 frontend/db/backends/__init__.py
 create mode 100644 frontend/db/backends/afe/__init__.py
 create mode 100644 frontend/db/backends/afe/base.py
 create mode 100644 frontend/db/backends/afe/compiler.py
 create mode 100644 frontend/db/backends/afe/creation.py
 create mode 100644 frontend/db/backends/afe/introspection.py
 create mode 100644 frontend/db/backends/afe/validation.py

-- 
1.7.5.2

[PATCH 1/6] Update admin site URL dispatcher to Django 1.3

[Lucas Meneghel Rodrigues]

admin.site.root does not exist under Django 1.3 anymore.
Use, therefore, the current idiom to specify the admin URL.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 frontend/urls.py |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/frontend/urls.py b/frontend/urls.py
index 7f18fec..5c50f51 100644
--- a/frontend/urls.py
+++ b/frontend/urls.py
@@ -15,7 +15,7 @@ handler500 = 'frontend.afe.views.handler500'
 
 urlpatterns = defaults.patterns(
         '',
-        (RE_PREFIX + r'admin/(.*)', admin.site.root),
+        (RE_PREFIX + r'admin/', defaults.include(admin.site.urls)),
         (RE_PREFIX, defaults.include('frontend.afe.urls')),
         (TKO_RE_PREFIX, defaults.include('frontend.tko.urls')),
         (PLANNER_RE_PREFIX, defaults.include('frontend.planner.urls')),
-- 
1.7.5.2

[PATCH 2/6] afe/readonly connection: Catch AttributeError exceptions

[Lucas Meneghel Rodrigues]

On Django 1.3, the DB connection object does not have some
of the attributes we are trying to compare on the __neq__
method, resulting on an AttributeError. Let's catch that
error so the application works under 1.3.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 frontend/afe/readonly_connection.py |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/frontend/afe/readonly_connection.py b/frontend/afe/readonly_connection.py
index 7d59aae..68af56c 100644
--- a/frontend/afe/readonly_connection.py
+++ b/frontend/afe/readonly_connection.py
@@ -89,7 +89,15 @@ class ReadOnlyConnection(object):
 
     def close(self):
         if self._connection is not None:
-            assert django_connection != self._connection
+            # Here we are checking if connection can be compared with
+            # the django DB class, but the connection class under django
+            # 1.3 doesn't have a settings_dict attribute, generating an
+            # AttributeError. So if that error is raised, supress it and
+            # only let the AssertionError pass.
+            try:
+                assert django_connection != self._connection
+            except AttributeError:
+                pass
             self._connection.close()
             self._connection = None
 
-- 
1.7.5.2

[PATCH 3/6] Adding afe custom database engine

[Lucas Meneghel Rodrigues]

Turns out that we rely on custom code to generate
joins that does some trickery with Django ORM system,
and was using a method calling get_from_clause, that
was present on the Query class on Django 1.1. Now,
the method was moved to a class upper in the hierarchy,
called SQLCompiler, and SQLCompiler is tied to the db
backend being used. After some research, seems like there's
no solution cleaner than implementing one db backend
special for autotest, just to accomodate this function.

So created one db engine that inherits pretty much everything
from the mysql one, except for the compiler classes.

Tested, and it works fairly well. Eventually more work
needs to be put in getting the web app into better shape.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 frontend/db/backends/afe/base.py          |   22 +++++++++++++++++++
 frontend/db/backends/afe/compiler.py      |   32 +++++++++++++++++++++++++++++
 frontend/db/backends/afe/creation.py      |    1 +
 frontend/db/backends/afe/introspection.py |    1 +
 frontend/db/backends/afe/validation.py    |    1 +
 5 files changed, 57 insertions(+), 0 deletions(-)
 create mode 100644 frontend/db/__init__.py
 create mode 100644 frontend/db/backends/__init__.py
 create mode 100644 frontend/db/backends/afe/__init__.py
 create mode 100644 frontend/db/backends/afe/base.py
 create mode 100644 frontend/db/backends/afe/compiler.py
 create mode 100644 frontend/db/backends/afe/creation.py
 create mode 100644 frontend/db/backends/afe/introspection.py
 create mode 100644 frontend/db/backends/afe/validation.py

diff --git a/frontend/db/__init__.py b/frontend/db/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/frontend/db/backends/__init__.py b/frontend/db/backends/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/frontend/db/backends/afe/__init__.py b/frontend/db/backends/afe/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/frontend/db/backends/afe/base.py b/frontend/db/backends/afe/base.py
new file mode 100644
index 0000000..6d617d2
--- /dev/null
+++ b/frontend/db/backends/afe/base.py
@@ -0,0 +1,22 @@
+from django.db.backends.mysql.base import DatabaseCreation as MySQLCreation
+from django.db.backends.mysql.base import DatabaseOperations as MySQLOperations
+from django.db.backends.mysql.base import DatabaseWrapper as MySQLDatabaseWrapper
+from django.db.backends.mysql.base import DatabaseIntrospection as MySQLIntrospection
+
+try:
+    import MySQLdb as Database
+except ImportError, e:
+    from django.core.exceptions import ImproperlyConfigured
+    raise ImproperlyConfigured("Error loading MySQLdb module: %s" % e)
+
+
+class DatabaseOperations(MySQLOperations):
+    compiler_module = "autotest_lib.frontend.db.backends.afe.compiler"
+
+
+class DatabaseWrapper(MySQLDatabaseWrapper):
+    def __init__(self, *args, **kwargs):
+        super(DatabaseWrapper, self).__init__(*args, **kwargs)
+        self.creation = MySQLCreation(self)
+        self.ops = DatabaseOperations()
+        self.introspection = MySQLIntrospection(self)
diff --git a/frontend/db/backends/afe/compiler.py b/frontend/db/backends/afe/compiler.py
new file mode 100644
index 0000000..9971415
--- /dev/null
+++ b/frontend/db/backends/afe/compiler.py
@@ -0,0 +1,32 @@
+from django.db.backends.mysql import compiler as mysql_compiler
+from autotest_lib.frontend.afe.model_logic import _quote_name
+
+class SQLCompiler(mysql_compiler.SQLCompiler):
+    def get_from_clause(self):
+        from_, params = super(SQLCompiler, self).get_from_clause()
+
+        if hasattr(self.query, "_custom_joins"):
+            for join_dict in self.query._custom_joins:
+                from_.append('%s %s AS %s ON (%s)'
+                             % (join_dict['join_type'],
+                                _quote_name(join_dict['table']),
+                                _quote_name(join_dict['alias']),
+                                join_dict['condition']))
+                params.extend(join_dict['condition_values'])
+
+        return from_, params
+
+class SQLInsertCompiler(mysql_compiler.SQLInsertCompiler, SQLCompiler):
+    pass
+
+class SQLDeleteCompiler(mysql_compiler.SQLDeleteCompiler, SQLCompiler):
+    pass
+
+class SQLUpdateCompiler(mysql_compiler.SQLUpdateCompiler, SQLCompiler):
+    pass
+
+class SQLAggregateCompiler(mysql_compiler.SQLAggregateCompiler, SQLCompiler):
+    pass
+
+class SQLDateCompiler(mysql_compiler.SQLDateCompiler, SQLCompiler):
+    pass
diff --git a/frontend/db/backends/afe/creation.py b/frontend/db/backends/afe/creation.py
new file mode 100644
index 0000000..955a11e
--- /dev/null
+++ b/frontend/db/backends/afe/creation.py
@@ -0,0 +1 @@
+from django.db.backends.mysql.creation import *
diff --git a/frontend/db/backends/afe/introspection.py b/frontend/db/backends/afe/introspection.py
new file mode 100644
index 0000000..222bc9b
--- /dev/null
+++ b/frontend/db/backends/afe/introspection.py
@@ -0,0 +1 @@
+from django.db.backends.mysql.introspection import *
diff --git a/frontend/db/backends/afe/validation.py b/frontend/db/backends/afe/validation.py
new file mode 100644
index 0000000..818b34a
--- /dev/null
+++ b/frontend/db/backends/afe/validation.py
@@ -0,0 +1 @@
+from django.db.backends.mysql.validation import *
-- 
1.7.5.2

[PATCH 4/6] Change web app to use new DB engine

[Lucas Meneghel Rodrigues]

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 frontend/settings.py |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/frontend/settings.py b/frontend/settings.py
index 3ac7f80..79b2a70 100644
--- a/frontend/settings.py
+++ b/frontend/settings.py
@@ -15,7 +15,8 @@ ADMINS = (
 
 MANAGERS = ADMINS
 
-DATABASE_ENGINE = 'mysql'      # 'postgresql_psycopg2', 'postgresql',
+DATABASE_ENGINE = 'autotest_lib.frontend.db.backends.afe'
+                               # 'postgresql_psycopg2', 'postgresql',
                                # 'mysql', 'sqlite3' or 'ado_mssql'.
 DATABASE_PORT = ''             # Set to empty string for default.
                                # Not used with sqlite3.
-- 
1.7.5.2

[PATCH 5/6] Removing get_from_clause() from custom query class

[Lucas Meneghel Rodrigues]

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 frontend/afe/model_logic.py |   15 ---------------
 1 files changed, 0 insertions(+), 15 deletions(-)

diff --git a/frontend/afe/model_logic.py b/frontend/afe/model_logic.py
index 0ddcefe..38bcabe 100644
--- a/frontend/afe/model_logic.py
+++ b/frontend/afe/model_logic.py
@@ -124,21 +124,6 @@ class ExtendedManager(dbmodels.Manager):
             self._custom_joins.append(join_dict)
 
 
-        def get_from_clause(self):
-            from_, params = (super(ExtendedManager.CustomQuery, self)
-                             .get_from_clause())
-
-            for join_dict in self._custom_joins:
-                from_.append('%s %s AS %s ON (%s)'
-                             % (join_dict['join_type'],
-                                _quote_name(join_dict['table']),
-                                _quote_name(join_dict['alias']),
-                                join_dict['condition']))
-                params.extend(join_dict['condition_values'])
-
-            return from_, params
-
-
         @classmethod
         def convert_query(self, query_set):
             """
-- 
1.7.5.2

[PATCH 6/6] external_packages: Bump version number requirements for web deps

[Lucas Meneghel Rodrigues]

With the previous patchset that fixes the frontend running under
Django 1.3, tested the frontend code compilation with the
latest versions of gwt and gwt-incubator with success. The interface
looks pretty good, and therefore, let's bump version numbers for:

* Django
* gwt
* gwt-incubator

On build_externals.py all at once. This is important so we can
keep the code working under new versions of the foundation components
of the web interface.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 utils/external_packages.py |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/utils/external_packages.py b/utils/external_packages.py
index bcfd15d..34d62b6 100755
--- a/utils/external_packages.py
+++ b/utils/external_packages.py
@@ -552,10 +552,10 @@ class MySQLdbPackage(ExternalPackage):
 
 
 class DjangoPackage(ExternalPackage):
-    version = '1.1.1'
+    version = '1.3'
     local_filename = 'Django-%s.tar.gz' % version
     urls = ('http://www.djangoproject.com/download/%s/tarball/' % version,)
-    hex_sum = '441c54f0e90730bf4a55432b64519169b1e6ef20'
+    hex_sum = 'f8814d5e1412bb932318db5130260da5bf053ff7'
 
     _build_and_install = ExternalPackage._build_and_install_from_package
     _build_and_install_current_dir = (
@@ -680,10 +680,10 @@ class Httplib2Package(ExternalPackage):
 class GwtPackage(ExternalPackage):
     """Fetch and extract a local copy of GWT used to build the frontend."""
 
-    version = '2.0.3'
+    version = '2.3.0'
     local_filename = 'gwt-%s.zip' % version
     urls = ('http://google-web-toolkit.googlecode.com/files/' + local_filename,)
-    hex_sum = '1dabd25a02b9299f6fa84c51c97210a3373a663e'
+    hex_sum = 'd51fce9166e6b31349659ffca89baf93e39bc84b'
     name = 'gwt'
     about_filename = 'about.txt'
     module_name = None  # Not a Python module.
@@ -724,12 +724,12 @@ class GwtPackage(ExternalPackage):
 # This requires GWT to already be installed, so it must be declared after
 # GwtPackage
 class GwtIncubatorPackage(ExternalPackage):
-    version = '20100204-r1747'
+    version = '20101117-r1766'
     local_filename = 'gwt-incubator-%s.jar' % version
     symlink_name = 'gwt-incubator.jar'
     urls = ('http://google-web-toolkit-incubator.googlecode.com/files/'
             + local_filename,)
-    hex_sum = '0c9495634f0627d0b4de0d78a50a3aefebf67f8c'
+    hex_sum = '3aa16d4c7c00edad4719092669d820a34e10ef0a'
     module_name = None  # Not a Python module
 
 
-- 
1.7.5.2

Re: [PATCH 1/6] Update admin site URL dispatcher to Django 1.3

[Lucas Meneghel Rodrigues]

On Fri, 2011-05-27 at 10:53 -0400, vyanktesh yadav wrote:
> Hi ,
> 
> [Please see screen shot too..] 
> 
> I am facing Template syntax error while trying to do add hosts in
> admin
> interface. 

Yeah, some of the custom template logic present on the application is
being slashed by the internal changes of the admin module. And I thought
1.3 was supposed to be mostly compatible API wise with 1.1. What a
mistake...

I've fixed this problem on my local git repo, but there are other things
that came up to address. I will work more on the port and publish
another patchset, after proper testing is made.

Thanks for trying and sorry for the hassle,

Lucas

> Facing this error with the latest autotest.git and latest Django's
> syncdb.
> 
> TemplateSyntaxError at /afe/server/admin/afe/host/add/
> Caught an exception while rendering: 
> 
> In
> template 
> /usr/localtest/autotest/site-packages/django/contrib/admin/templates/admin/includes/fieldset.html,
>  error at line 12
> 
> also In
> 
> TemplateSyntaxError at /afe/server/admin/afe/aclgroup/add/
> Caught an exception while rendering:
> 
> 
> 
> Please let me know any patch is available for this issue?
> 
> Thanks & Regards,
> Vyanktesh Yadav
>