All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] createrepo: Implement --dbpath command line option
@ 2015-04-01 12:09 Ed Bartosh
  0 siblings, 0 replies; 6+ messages in thread
From: Ed Bartosh @ 2015-04-01 12:09 UTC (permalink / raw)
  To: openembedded-core

--dbpath option can be used in cases where users don't want
createrepo to use system rpm db to avoid possible collisiouns
with other programs.

For bitbake builds it would be possible to specify different
databases even for every createrepo run. Considering that rootfs
builds can run multiple createrepo in parallel, it can help to avoid
race conditions caused by accessing or creating the same rpm database
by multiple createrepo instances at the same time.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 .../createrepo/createrepo/createrepo-dbpath.patch  | 61 ++++++++++++++++++++++
 .../createrepo/createrepo_0.4.11.bb                |  1 +
 2 files changed, 62 insertions(+)
 create mode 100644 meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch

diff --git a/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
new file mode 100644
index 0000000..7275598
--- /dev/null
+++ b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
@@ -0,0 +1,61 @@
+createrepo: Implement --dbpath command line option
+
+--dbpath option allows to specify path to the directory
+with rpm database. By default createrepo uses or creates
+rpm database in /var/lib/rpm/
+
+Upstream-Status: Pending
+
+Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
+
+--- createrepo-0.4.11.orig/genpkgmetadata.py 2015-03-30 22:18:19.904000000 +0300
++++ createrepo-0.4.11/genpkgmetadata.py 2015-03-30 22:28:49.208000000 +0300
+@@ -65,6 +65,7 @@
+      -p, --pretty = output xml files in pretty format.
+      --update = update existing metadata (if present)
+      -d, --database = generate the sqlite databases.
++     --dbpath <dir> = specify path to rpm db directory.
+     """)
+ 
+     sys.exit(retval)
+@@ -72,10 +73,13 @@
+ class MetaDataGenerator:
+     def __init__(self, cmds):
+         self.cmds = cmds
+-        self.ts = rpm.TransactionSet()
+         self.pkgcount = 0
+         self.files = []
+ 
++        if self.cmds['dbpath']:
++            rpm.addMacro("_dbpath", self.cmds['dbpath'])
++        self.ts = rpm.TransactionSet()
++
+     def _os_path_walk(self, top, func, arg):
+         """Directory tree walk with callback function.
+          copy of os.path.walk, fixes the link/stating problem
+@@ -435,6 +439,7 @@
+     cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*']
+     cmds['skip-symlinks'] = False
+     cmds['pkglist'] = []
++    cmds['dbpath'] = None
+ 
+     try:
+         gopts, argsleft = getopt.getopt(args, 'phqVvndg:s:x:u:c:o:CSi:', ['help', 'exclude=',
+@@ -442,7 +447,7 @@
+                                                                   'baseurl=', 'groupfile=', 'checksum=',
+                                                                   'version', 'pretty', 'split', 'outputdir=',
+                                                                   'noepoch', 'checkts', 'database', 'update',
+-                                                                  'skip-symlinks', 'pkglist='])
++                                                                  'skip-symlinks', 'pkglist=', 'dbpath='])
+     except getopt.error, e:
+         errorprint(_('Options Error: %s.') % e)
+         usage()
+@@ -516,6 +521,8 @@
+                 cmds['skip-symlinks'] = True
+             elif arg in ['-i', '--pkglist']:
+                 cmds['pkglist'] = a
++            elif arg == '--dbpath':
++                cmds['dbpath'] = os.path.realpath(a)
+                                 
+     except ValueError, e:
+         errorprint(_('Options Error: %s') % e)
diff --git a/meta/recipes-support/createrepo/createrepo_0.4.11.bb b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
index 49b45fc..adc193e 100644
--- a/meta/recipes-support/createrepo/createrepo_0.4.11.bb
+++ b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
@@ -14,6 +14,7 @@ SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \
           file://python-scripts-should-use-interpreter-from-env.patch \
 	  file://createrepo-rpm549.patch \
 	  file://recommends.patch \
+	  file://createrepo-dbpath.patch \
 	  file://rpm-createsolvedb.py \
          "
 
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 6+ messages in thread
* Re: [PATCH] rootfs.py: Remove rpm database from staging area
@ 2015-03-30 22:47 Ed Bartosh
  2015-03-30 23:49 ` [PATCH] createrepo: Implement --dbpath command line option Ed Bartosh
  0 siblings, 1 reply; 6+ messages in thread
From: Ed Bartosh @ 2015-03-30 22:47 UTC (permalink / raw)
  To: Mark Hatle; +Cc: openembedded-core

On Mon, Mar 30, 2015 at 10:14:36AM -0500, Mark Hatle wrote:
> On 3/30/15 3:53 AM, Richard Purdie wrote:
> > On Mon, 2015-03-30 at 11:30 +0300, Ed Bartosh wrote:
> >> Rpm database in staging area is used only by createrepo.
> >> createrepo fails with the error
> >> "rpmdb: BDB0060 PANIC: fatal region error detected"
> >> if rpm database is broken from the previous run of createrepo.
> >>
> >> Removing the databae before running createrepo can hopefully
> >> prevent this failure to happen.
> >>
> >> [YOCTO #6571]
> >>
> >> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> >> ---
> >>  meta/lib/oe/rootfs.py | 3 +++
> >>  1 file changed, 3 insertions(+)
> >>
> >> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> >> index 4e4e6eb..9f7dc65 100644
> >> --- a/meta/lib/oe/rootfs.py
> >> +++ b/meta/lib/oe/rootfs.py
> >> @@ -306,6 +306,9 @@ class RpmRootfs(Rootfs):
> >>              bb.utils.remove(self.image_rootfs, True)
> >>          else:
> >>              self.pm.recovery_packaging_data()
> >> +        dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True),
> >> +                              'var/lib/rpm/*')
> >> +        bb.utils.remove(dbpath, recurse=True)
> >>          bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
> >>  
> >>          self.pm.crea
> > 
> > This patch helps me see the problem a lot. I'd never realised there was
> > an rpm database in the native sysroot that was getting corrupted, I'd
> > always assumed it was the target rootfs one.
> > 
> > I'm a little worried about what happens if you have multiple images
> > generating at the same time as this change as above may delete something
> > being worked on by another process.
> > 
> > I'm wondering why is it getting into the sysroot at all? Rather than
> > delete it here, could we either not generate it at all, or place it
> > somewhere in WORKDIR (so that other tasks can't see it or race against
> > it)?
> 
> I did some quick looking.  It appears (at least on first glance) that the call to:
> 
> rpm.TransactionSet()
> 
> is what is opening the DB.  It appears to me that we can change the call
> slightly, and it should do what is being requested:
> 
>     { "TransactionSet", (PyCFunction) rpmts_Create, METH_VARARGS|METH_KEYWORDS,
> "rpm.TransactionSet([rootDir, [db]]) -> ts\n\
> - Create a transaction set.\n" },
> 
> So transaction set takes two arguments, the rootDir of the thing we're working
> against, and a DB path.  It -should- be as simply as setting a rootDir to the
> WORKDIR.  If that doesn't work, then both arguments will be needed.
> 
> I'd suggest just adding a "--root" option to genpkgmetadata.py in the
> createrepo, and maybe a "--dbpath" option as well (second argument).  Then only
> call the function with their values IF they were defined. I.e.
> 
> if root:
>   if dbpath:
>     self.ts = rpm.TransactionSet(root, dbpath)
>   else:
>     self.ts = rpm.TransactionSet(root)
> else:
>   self.ts = rpm.TransactionSet()
> 
> (If there is a better way to do that in python, fine.. but passing in "None" or
> "" won't result in the desired behavior.. since RPM is actually parsing
> arguments and appears like it will use the values if passed in, whatever they are.)
>

Second parameter is not a path. Integer is required there, so I guess it's a db mode something like that.
Defining _dbpath macro looks better from my point of view as it allows to specify full path to the db directory.

Regards,
Ed

 


^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-04-01 12:12 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-04-01 12:09 [PATCH] createrepo: Implement --dbpath command line option Ed Bartosh
  -- strict thread matches above, loose matches on Subject: below --
2015-03-30 22:47 [PATCH] rootfs.py: Remove rpm database from staging area Ed Bartosh
2015-03-30 23:49 ` [PATCH] createrepo: Implement --dbpath command line option Ed Bartosh
2015-03-31 21:25   ` Burton, Ross
2015-04-01 11:19     ` Ed Bartosh
2015-04-01 12:06     ` Ed Bartosh
2015-04-01 12:12       ` Ed Bartosh

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.