[PATCH] rootfs.py: Remove rpm database from staging area

[PATCH] rootfs.py: Remove rpm database from staging area

[Ed Bartosh]

Rpm database in staging area is used only by createrepo.
createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected"
if rpm database is broken from the previous run of createrepo.

Removing the databae before running createrepo can hopefully
prevent this failure to happen.

[YOCTO #6571]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oe/rootfs.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 4e4e6eb..9f7dc65 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -306,6 +306,9 @@ class RpmRootfs(Rootfs):
             bb.utils.remove(self.image_rootfs, True)
         else:
             self.pm.recovery_packaging_data()
+        dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True),
+                              'var/lib/rpm/*')
+        bb.utils.remove(dbpath, recurse=True)
         bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
 
         self.pm.create_configs()
-- 
2.1.4

Re: [PATCH] rootfs.py: Remove rpm database from staging area

[Richard Purdie]

On Mon, 2015-03-30 at 11:30 +0300, Ed Bartosh wrote:
> Rpm database in staging area is used only by createrepo.
> createrepo fails with the error
> "rpmdb: BDB0060 PANIC: fatal region error detected"
> if rpm database is broken from the previous run of createrepo.
> 
> Removing the databae before running createrepo can hopefully
> prevent this failure to happen.
> 
> [YOCTO #6571]
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/lib/oe/rootfs.py | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> index 4e4e6eb..9f7dc65 100644
> --- a/meta/lib/oe/rootfs.py
> +++ b/meta/lib/oe/rootfs.py
> @@ -306,6 +306,9 @@ class RpmRootfs(Rootfs):
>              bb.utils.remove(self.image_rootfs, True)
>          else:
>              self.pm.recovery_packaging_data()
> +        dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True),
> +                              'var/lib/rpm/*')
> +        bb.utils.remove(dbpath, recurse=True)
>          bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
>  
>          self.pm.crea

This patch helps me see the problem a lot. I'd never realised there was
an rpm database in the native sysroot that was getting corrupted, I'd
always assumed it was the target rootfs one.

I'm a little worried about what happens if you have multiple images
generating at the same time as this change as above may delete something
being worked on by another process.

I'm wondering why is it getting into the sysroot at all? Rather than
delete it here, could we either not generate it at all, or place it
somewhere in WORKDIR (so that other tasks can't see it or race against
it)?

Cheers,

Richard

Re: [PATCH] rootfs.py: Remove rpm database from staging area

[Mark Hatle]

On 3/30/15 3:53 AM, Richard Purdie wrote:
> On Mon, 2015-03-30 at 11:30 +0300, Ed Bartosh wrote:
>> Rpm database in staging area is used only by createrepo.
>> createrepo fails with the error
>> "rpmdb: BDB0060 PANIC: fatal region error detected"
>> if rpm database is broken from the previous run of createrepo.
>>
>> Removing the databae before running createrepo can hopefully
>> prevent this failure to happen.
>>
>> [YOCTO #6571]
>>
>> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>> ---
>>  meta/lib/oe/rootfs.py | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
>> index 4e4e6eb..9f7dc65 100644
>> --- a/meta/lib/oe/rootfs.py
>> +++ b/meta/lib/oe/rootfs.py
>> @@ -306,6 +306,9 @@ class RpmRootfs(Rootfs):
>>              bb.utils.remove(self.image_rootfs, True)
>>          else:
>>              self.pm.recovery_packaging_data()
>> +        dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True),
>> +                              'var/lib/rpm/*')
>> +        bb.utils.remove(dbpath, recurse=True)
>>          bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
>>  
>>          self.pm.crea
> 
> This patch helps me see the problem a lot. I'd never realised there was
> an rpm database in the native sysroot that was getting corrupted, I'd
> always assumed it was the target rootfs one.
> 
> I'm a little worried about what happens if you have multiple images
> generating at the same time as this change as above may delete something
> being worked on by another process.
> 
> I'm wondering why is it getting into the sysroot at all? Rather than
> delete it here, could we either not generate it at all, or place it
> somewhere in WORKDIR (so that other tasks can't see it or race against
> it)?

My guess is that either createrepo or other rpm commands are doing a lazy init
on the DB.  As you mentioned deleting the DB is a little worrying because if
it's open in another process you -are- going to cause other failures.

It would be nice if we could find the creation points and redirect them to the
current workdir.  (Pseudo and it's logging might help us be able to do that.)
The other option could be to create an empty DB, and simply set it and the
directory containing it to read-only, no write.... (but I'm not sure that the
filesystem is actually going to enforce that for the current user.)

--Mark

> Cheers,
> 
> Richard
> 

Re: [PATCH] rootfs.py: Remove rpm database from staging area

[Mark Hatle]

On 3/30/15 3:53 AM, Richard Purdie wrote:
> On Mon, 2015-03-30 at 11:30 +0300, Ed Bartosh wrote:
>> Rpm database in staging area is used only by createrepo.
>> createrepo fails with the error
>> "rpmdb: BDB0060 PANIC: fatal region error detected"
>> if rpm database is broken from the previous run of createrepo.
>>
>> Removing the databae before running createrepo can hopefully
>> prevent this failure to happen.
>>
>> [YOCTO #6571]
>>
>> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>> ---
>>  meta/lib/oe/rootfs.py | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
>> index 4e4e6eb..9f7dc65 100644
>> --- a/meta/lib/oe/rootfs.py
>> +++ b/meta/lib/oe/rootfs.py
>> @@ -306,6 +306,9 @@ class RpmRootfs(Rootfs):
>>              bb.utils.remove(self.image_rootfs, True)
>>          else:
>>              self.pm.recovery_packaging_data()
>> +        dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True),
>> +                              'var/lib/rpm/*')
>> +        bb.utils.remove(dbpath, recurse=True)
>>          bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
>>  
>>          self.pm.crea
> 
> This patch helps me see the problem a lot. I'd never realised there was
> an rpm database in the native sysroot that was getting corrupted, I'd
> always assumed it was the target rootfs one.
> 
> I'm a little worried about what happens if you have multiple images
> generating at the same time as this change as above may delete something
> being worked on by another process.
> 
> I'm wondering why is it getting into the sysroot at all? Rather than
> delete it here, could we either not generate it at all, or place it
> somewhere in WORKDIR (so that other tasks can't see it or race against
> it)?

I did some quick looking.  It appears (at least on first glance) that the call to:

rpm.TransactionSet()

is what is opening the DB.  It appears to me that we can change the call
slightly, and it should do what is being requested:

    { "TransactionSet", (PyCFunction) rpmts_Create, METH_VARARGS|METH_KEYWORDS,
"rpm.TransactionSet([rootDir, [db]]) -> ts\n\
- Create a transaction set.\n" },

So transaction set takes two arguments, the rootDir of the thing we're working
against, and a DB path.  It -should- be as simply as setting a rootDir to the
WORKDIR.  If that doesn't work, then both arguments will be needed.

I'd suggest just adding a "--root" option to genpkgmetadata.py in the
createrepo, and maybe a "--dbpath" option as well (second argument).  Then only
call the function with their values IF they were defined. I.e.

if root:
  if dbpath:
    self.ts = rpm.TransactionSet(root, dbpath)
  else:
    self.ts = rpm.TransactionSet(root)
else:
  self.ts = rpm.TransactionSet()

(If there is a better way to do that in python, fine.. but passing in "None" or
"" won't result in the desired behavior.. since RPM is actually parsing
arguments and appears like it will use the values if passed in, whatever they are.)

--Mark

> Cheers,
> 
> Richard
> 

Re: [PATCH] rootfs.py: Remove rpm database from staging area

[Ed Bartosh]

On Mon, Mar 30, 2015 at 10:14:36AM -0500, Mark Hatle wrote:
> On 3/30/15 3:53 AM, Richard Purdie wrote:
> > On Mon, 2015-03-30 at 11:30 +0300, Ed Bartosh wrote:
> >> Rpm database in staging area is used only by createrepo.
> >> createrepo fails with the error
> >> "rpmdb: BDB0060 PANIC: fatal region error detected"
> >> if rpm database is broken from the previous run of createrepo.
> >>
> >> Removing the databae before running createrepo can hopefully
> >> prevent this failure to happen.
> >>
> >> [YOCTO #6571]
> >>
> >> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> >> ---
> >>  meta/lib/oe/rootfs.py | 3 +++
> >>  1 file changed, 3 insertions(+)
> >>
> >> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> >> index 4e4e6eb..9f7dc65 100644
> >> --- a/meta/lib/oe/rootfs.py
> >> +++ b/meta/lib/oe/rootfs.py
> >> @@ -306,6 +306,9 @@ class RpmRootfs(Rootfs):
> >>              bb.utils.remove(self.image_rootfs, True)
> >>          else:
> >>              self.pm.recovery_packaging_data()
> >> +        dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True),
> >> +                              'var/lib/rpm/*')
> >> +        bb.utils.remove(dbpath, recurse=True)
> >>          bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
> >>  
> >>          self.pm.crea
> > 
> > This patch helps me see the problem a lot. I'd never realised there was
> > an rpm database in the native sysroot that was getting corrupted, I'd
> > always assumed it was the target rootfs one.
> > 
> > I'm a little worried about what happens if you have multiple images
> > generating at the same time as this change as above may delete something
> > being worked on by another process.
> > 
> > I'm wondering why is it getting into the sysroot at all? Rather than
> > delete it here, could we either not generate it at all, or place it
> > somewhere in WORKDIR (so that other tasks can't see it or race against
> > it)?
> 
> I did some quick looking.  It appears (at least on first glance) that the call to:
> 
> rpm.TransactionSet()
> 
> is what is opening the DB.  It appears to me that we can change the call
> slightly, and it should do what is being requested:
> 
>     { "TransactionSet", (PyCFunction) rpmts_Create, METH_VARARGS|METH_KEYWORDS,
> "rpm.TransactionSet([rootDir, [db]]) -> ts\n\
> - Create a transaction set.\n" },
> 
> So transaction set takes two arguments, the rootDir of the thing we're working
> against, and a DB path.  It -should- be as simply as setting a rootDir to the
> WORKDIR.  If that doesn't work, then both arguments will be needed.
> 
> I'd suggest just adding a "--root" option to genpkgmetadata.py in the
> createrepo, and maybe a "--dbpath" option as well (second argument).  Then only
> call the function with their values IF they were defined. I.e.
> 
> if root:
>   if dbpath:
>     self.ts = rpm.TransactionSet(root, dbpath)
>   else:
>     self.ts = rpm.TransactionSet(root)
> else:
>   self.ts = rpm.TransactionSet()
> 
> (If there is a better way to do that in python, fine.. but passing in "None" or
> "" won't result in the desired behavior.. since RPM is actually parsing
> arguments and appears like it will use the values if passed in, whatever they are.)
>

I was going to do something like this:

if dbpath:
    rpm.addMacro("_dbpath", dbpath)

self.ts = rpm.TransactionSet()

Brief test showed that this approach works just fine.

Regards,
Ed

Re: [PATCH] rootfs.py: Remove rpm database from staging area

[Mark Hatle]

On 3/30/15 2:00 PM, Ed Bartosh wrote:
> On Mon, Mar 30, 2015 at 10:14:36AM -0500, Mark Hatle wrote:
>> On 3/30/15 3:53 AM, Richard Purdie wrote:
>>> On Mon, 2015-03-30 at 11:30 +0300, Ed Bartosh wrote:
>>>> Rpm database in staging area is used only by createrepo.
>>>> createrepo fails with the error
>>>> "rpmdb: BDB0060 PANIC: fatal region error detected"
>>>> if rpm database is broken from the previous run of createrepo.
>>>>
>>>> Removing the databae before running createrepo can hopefully
>>>> prevent this failure to happen.
>>>>
>>>> [YOCTO #6571]
>>>>
>>>> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>>>> ---
>>>>  meta/lib/oe/rootfs.py | 3 +++
>>>>  1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
>>>> index 4e4e6eb..9f7dc65 100644
>>>> --- a/meta/lib/oe/rootfs.py
>>>> +++ b/meta/lib/oe/rootfs.py
>>>> @@ -306,6 +306,9 @@ class RpmRootfs(Rootfs):
>>>>              bb.utils.remove(self.image_rootfs, True)
>>>>          else:
>>>>              self.pm.recovery_packaging_data()
>>>> +        dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True),
>>>> +                              'var/lib/rpm/*')
>>>> +        bb.utils.remove(dbpath, recurse=True)
>>>>          bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
>>>>  
>>>>          self.pm.crea
>>>
>>> This patch helps me see the problem a lot. I'd never realised there was
>>> an rpm database in the native sysroot that was getting corrupted, I'd
>>> always assumed it was the target rootfs one.
>>>
>>> I'm a little worried about what happens if you have multiple images
>>> generating at the same time as this change as above may delete something
>>> being worked on by another process.
>>>
>>> I'm wondering why is it getting into the sysroot at all? Rather than
>>> delete it here, could we either not generate it at all, or place it
>>> somewhere in WORKDIR (so that other tasks can't see it or race against
>>> it)?
>>
>> I did some quick looking.  It appears (at least on first glance) that the call to:
>>
>> rpm.TransactionSet()
>>
>> is what is opening the DB.  It appears to me that we can change the call
>> slightly, and it should do what is being requested:
>>
>>     { "TransactionSet", (PyCFunction) rpmts_Create, METH_VARARGS|METH_KEYWORDS,
>> "rpm.TransactionSet([rootDir, [db]]) -> ts\n\
>> - Create a transaction set.\n" },
>>
>> So transaction set takes two arguments, the rootDir of the thing we're working
>> against, and a DB path.  It -should- be as simply as setting a rootDir to the
>> WORKDIR.  If that doesn't work, then both arguments will be needed.
>>
>> I'd suggest just adding a "--root" option to genpkgmetadata.py in the
>> createrepo, and maybe a "--dbpath" option as well (second argument).  Then only
>> call the function with their values IF they were defined. I.e.
>>
>> if root:
>>   if dbpath:
>>     self.ts = rpm.TransactionSet(root, dbpath)
>>   else:
>>     self.ts = rpm.TransactionSet(root)
>> else:
>>   self.ts = rpm.TransactionSet()
>>
>> (If there is a better way to do that in python, fine.. but passing in "None" or
>> "" won't result in the desired behavior.. since RPM is actually parsing
>> arguments and appears like it will use the values if passed in, whatever they are.)
>>
> 
> I was going to do something like this:
> 
> if dbpath:
>     rpm.addMacro("_dbpath", dbpath)
> 
> self.ts = rpm.TransactionSet()
> 
> Brief test showed that this approach works just fine.

Yes, that can work as well..  but the transactionSet actually has arguments for
root and dbpath built in.  It's a good idea to select the WORKDIR as the 'root',
and the dbpath as somewhere "in the WORKDIR".

(This will prevent rpm for intentionally writing anything into the execution
sysroot, and restrict it to the WORKDIR.)

--Mark

> Regards,
> Ed
> 

Re: [PATCH] rootfs.py: Remove rpm database from staging area

[Ed Bartosh]

On Mon, Mar 30, 2015 at 10:14:36AM -0500, Mark Hatle wrote:
> On 3/30/15 3:53 AM, Richard Purdie wrote:
> > On Mon, 2015-03-30 at 11:30 +0300, Ed Bartosh wrote:
> >> Rpm database in staging area is used only by createrepo.
> >> createrepo fails with the error
> >> "rpmdb: BDB0060 PANIC: fatal region error detected"
> >> if rpm database is broken from the previous run of createrepo.
> >>
> >> Removing the databae before running createrepo can hopefully
> >> prevent this failure to happen.
> >>
> >> [YOCTO #6571]
> >>
> >> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> >> ---
> >>  meta/lib/oe/rootfs.py | 3 +++
> >>  1 file changed, 3 insertions(+)
> >>
> >> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> >> index 4e4e6eb..9f7dc65 100644
> >> --- a/meta/lib/oe/rootfs.py
> >> +++ b/meta/lib/oe/rootfs.py
> >> @@ -306,6 +306,9 @@ class RpmRootfs(Rootfs):
> >>              bb.utils.remove(self.image_rootfs, True)
> >>          else:
> >>              self.pm.recovery_packaging_data()
> >> +        dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True),
> >> +                              'var/lib/rpm/*')
> >> +        bb.utils.remove(dbpath, recurse=True)
> >>          bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
> >>  
> >>          self.pm.crea
> > 
> > This patch helps me see the problem a lot. I'd never realised there was
> > an rpm database in the native sysroot that was getting corrupted, I'd
> > always assumed it was the target rootfs one.
> > 
> > I'm a little worried about what happens if you have multiple images
> > generating at the same time as this change as above may delete something
> > being worked on by another process.
> > 
> > I'm wondering why is it getting into the sysroot at all? Rather than
> > delete it here, could we either not generate it at all, or place it
> > somewhere in WORKDIR (so that other tasks can't see it or race against
> > it)?
> 
> I did some quick looking.  It appears (at least on first glance) that the call to:
> 
> rpm.TransactionSet()
> 
> is what is opening the DB.  It appears to me that we can change the call
> slightly, and it should do what is being requested:
> 
>     { "TransactionSet", (PyCFunction) rpmts_Create, METH_VARARGS|METH_KEYWORDS,
> "rpm.TransactionSet([rootDir, [db]]) -> ts\n\
> - Create a transaction set.\n" },
> 
> So transaction set takes two arguments, the rootDir of the thing we're working
> against, and a DB path.  It -should- be as simply as setting a rootDir to the
> WORKDIR.  If that doesn't work, then both arguments will be needed.
> 
> I'd suggest just adding a "--root" option to genpkgmetadata.py in the
> createrepo, and maybe a "--dbpath" option as well (second argument).  Then only
> call the function with their values IF they were defined. I.e.
> 
> if root:
>   if dbpath:
>     self.ts = rpm.TransactionSet(root, dbpath)
>   else:
>     self.ts = rpm.TransactionSet(root)
> else:
>   self.ts = rpm.TransactionSet()
> 
> (If there is a better way to do that in python, fine.. but passing in "None" or
> "" won't result in the desired behavior.. since RPM is actually parsing
> arguments and appears like it will use the values if passed in, whatever they are.)
>

Second parameter is not a path. Integer is required there, so I guess it's a db mode something like that.
Defining _dbpath macro looks better from my point of view as it allows to specify full path to the db directory.

Regards,
Ed

 

[PATCH] createrepo: Implement --dbpath command line option

[Ed Bartosh]

--dbpath option allows to specify path to the directory
with rpm database. By default createrepo uses or creates
rpm database in /var/lib/rpm/.

Upstream-Status: Pending

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 .../createrepo/createrepo/createrepo-dbpath.patch  | 51 ++++++++++++++++++++++
 .../createrepo/createrepo_0.4.11.bb                |  1 +
 2 files changed, 52 insertions(+)
 create mode 100644 meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch

diff --git a/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
new file mode 100644
index 0000000..2b113bf
--- /dev/null
+++ b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
@@ -0,0 +1,51 @@
+--- createrepo-0.4.11.orig/genpkgmetadata.py	2015-03-30 22:18:19.904000000 +0300
++++ createrepo-0.4.11.orig/genpkgmetadata.py	2015-03-30 22:28:49.208000000 +0300
+@@ -65,6 +65,7 @@
+      -p, --pretty = output xml files in pretty format.
+      --update = update existing metadata (if present)
+      -d, --database = generate the sqlite databases.
++     --dbpath <dir> = specify path to rpm db directory.
+     """)
+ 
+     sys.exit(retval)
+@@ -72,10 +73,13 @@
+ class MetaDataGenerator:
+     def __init__(self, cmds):
+         self.cmds = cmds
+-        self.ts = rpm.TransactionSet()
+         self.pkgcount = 0
+         self.files = []
+ 
++        if self.cmds['dbpath']:
++            rpm.addMacro("_dbpath", self.cmds['dbpath'])
++        self.ts = rpm.TransactionSet()
++
+     def _os_path_walk(self, top, func, arg):
+         """Directory tree walk with callback function.
+          copy of os.path.walk, fixes the link/stating problem
+@@ -435,6 +439,7 @@
+     cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*']
+     cmds['skip-symlinks'] = False
+     cmds['pkglist'] = []
++    cmds['dbpath'] = None
+ 
+     try:
+         gopts, argsleft = getopt.getopt(args, 'phqVvndg:s:x:u:c:o:CSi:', ['help', 'exclude=',
+@@ -442,7 +447,7 @@
+                                                                   'baseurl=', 'groupfile=', 'checksum=',
+                                                                   'version', 'pretty', 'split', 'outputdir=',
+                                                                   'noepoch', 'checkts', 'database', 'update',
+-                                                                  'skip-symlinks', 'pkglist='])
++                                                                  'skip-symlinks', 'pkglist=', 'dbpath='])
+     except getopt.error, e:
+         errorprint(_('Options Error: %s.') % e)
+         usage()
+@@ -516,6 +521,8 @@
+                 cmds['skip-symlinks'] = True
+             elif arg in ['-i', '--pkglist']:
+                 cmds['pkglist'] = a
++            elif arg == '--dbpath':
++                cmds['dbpath'] = os.path.realpath(a)
+                                 
+     except ValueError, e:
+         errorprint(_('Options Error: %s') % e)
diff --git a/meta/recipes-support/createrepo/createrepo_0.4.11.bb b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
index 49b45fc..adc193e 100644
--- a/meta/recipes-support/createrepo/createrepo_0.4.11.bb
+++ b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
@@ -14,6 +14,7 @@ SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \
           file://python-scripts-should-use-interpreter-from-env.patch \
 	  file://createrepo-rpm549.patch \
 	  file://recommends.patch \
+	  file://createrepo-dbpath.patch \
 	  file://rpm-createsolvedb.py \
          "
 
-- 
2.1.4

[PATCH] package_manager: call createrepo with --dbpath pointing inside WORKDIR

[Ed Bartosh]

Rpm database in staging area is used only by createrepo.
createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected"
if rpm database is broken during previous run of createrepo.

Made createrepo to create rpm db in $WORKDIR/rpmdb/ from scratch
for every build. This should potentially fix the failure.

[YOCTO #6571]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oe/package_manager.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 743c7cb..2461acd 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -110,12 +110,16 @@ class RpmIndexer(Indexer):
         rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
         index_cmds = []
         rpm_dirs_found = False
+        dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb')
+        if os.path.exists(dbpath):
+            bb.utils.remove(dbpath, True)
         for arch in archs:
             arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
                 continue
 
-            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
+            index_cmds.append("%s --dbpath %s --update -q %s" % \
+                             (rpm_createrepo, dbpath, arch_dir))
 
             rpm_dirs_found = True
 
-- 
2.1.4

Re: [PATCH] package_manager: call createrepo with --dbpath pointing inside WORKDIR

[Richard Purdie]

On Tue, 2015-03-31 at 03:16 +0300, Ed Bartosh wrote:
> Rpm database in staging area is used only by createrepo.
> createrepo fails with the error
> "rpmdb: BDB0060 PANIC: fatal region error detected"
> if rpm database is broken during previous run of createrepo.
> 
> Made createrepo to create rpm db in $WORKDIR/rpmdb/ from scratch
> for every build. This should potentially fix the failure.
> 
> [YOCTO #6571]
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/lib/oe/package_manager.py | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index 743c7cb..2461acd 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -110,12 +110,16 @@ class RpmIndexer(Indexer):
>          rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
>          index_cmds = []
>          rpm_dirs_found = False
> +        dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb')
> +        if os.path.exists(dbpath):
> +            bb.utils.remove(dbpath, True)
>          for arch in archs:
>              arch_dir = os.path.join(self.deploy_dir, arch)
>              if not os.path.isdir(arch_dir):
>                  continue
>  
> -            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
> +            index_cmds.append("%s --dbpath %s --update -q %s" % \
> +                             (rpm_createrepo, dbpath, arch_dir))
>  
>              rpm_dirs_found = True

Do we need to add arch to dbpath here to avoid races between the
separate createrepo processes?

Cheers,

Richard

Re: [PATCH] package_manager: call createrepo with --dbpath pointing inside WORKDIR

[Mark Hatle]

On 3/30/15 7:16 PM, Ed Bartosh wrote:
> Rpm database in staging area is used only by createrepo.
> createrepo fails with the error
> "rpmdb: BDB0060 PANIC: fatal region error detected"
> if rpm database is broken during previous run of createrepo.
> 
> Made createrepo to create rpm db in $WORKDIR/rpmdb/ from scratch
> for every build. This should potentially fix the failure.
> 
> [YOCTO #6571]
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/lib/oe/package_manager.py | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index 743c7cb..2461acd 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -110,12 +110,16 @@ class RpmIndexer(Indexer):
>          rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
>          index_cmds = []
>          rpm_dirs_found = False
> +        dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb')
> +        if os.path.exists(dbpath):
> +            bb.utils.remove(dbpath, True)
>          for arch in archs:
>              arch_dir = os.path.join(self.deploy_dir, arch)
>              if not os.path.isdir(arch_dir):
>                  continue
>  
> -            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
> +            index_cmds.append("%s --dbpath %s --update -q %s" % \
> +                             (rpm_createrepo, dbpath, arch_dir))

I'd suggest that the dbpath be made arch specific.... this would eliminate any
serialization that may occur with even simple DB locks.

--dbpath %s/%s ....

--Mark

>  
>              rpm_dirs_found = True
>  
> 

Re: [PATCH] createrepo: Implement --dbpath command line option

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

Hi Ed,

On 31 March 2015 at 00:49, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:

> Upstream-Status: Pending
>

The commit log needs just a Signed-off-by; the patch needs an explanation,
Signed-off-by, and Upstream-Status.  The goal is that the patch itself is
self-explanatory, so should explain what it does and why, whether it went
upstream and why (not), and who admits creating it in the first place.

Ross

[-- Attachment #2: Type: text/html, Size: 877 bytes --]

[PATCH] package_manager: call createrepo with --dbpath pointing inside WORKDIR

[Ed Bartosh]

Rpm database in staging area is used only by createrepo.
createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected"
if rpm database is broken during previous run of createrepo.

Made createrepo to create rpm db in $WORKDIR/rpmdb/ from scratch
for every build and architecture. This should potentially fix the
failure as every run of createrepo will be using separate db.

[YOCTO #6571]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oe/package_manager.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 743c7cb..ed0c9a7 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -111,11 +111,15 @@ class RpmIndexer(Indexer):
         index_cmds = []
         rpm_dirs_found = False
         for arch in archs:
+            dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb', arch)
+            if os.path.exists(dbpath):
+                bb.utils.remove(dbpath, True)
             arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
                 continue
 
-            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
+            index_cmds.append("%s --dbpath %s --update -q %s" % \
+                             (rpm_createrepo, dbpath, arch_dir))
 
             rpm_dirs_found = True
 
-- 
2.1.4

Re: [PATCH] createrepo: Implement --dbpath command line option

[Ed Bartosh]

On Tue, Mar 31, 2015 at 10:25:11PM +0100, Burton, Ross wrote:
> Hi Ed,
> 
> On 31 March 2015 at 00:49, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> 
> > Upstream-Status: Pending
> >
> 
> The commit log needs just a Signed-off-by; the patch needs an explanation,
> Signed-off-by, and Upstream-Status.  The goal is that the patch itself is
> self-explanatory, so should explain what it does and why, whether it went
> upstream and why (not), and who admits creating it in the first place.
>
Thanks for the explanations. I'll update the patch.

Regards,
Ed

[PATCH] createrepo: Implement --dbpath command line option

[Ed Bartosh]

--dbpath option can be used in cases where users don't want
createrepo to use system rpm db to avoid possible collisiouns
with other programs.

For bitbake builds it would be possible to specify different
databases even for every createrepo run. Considering that rootfs
builds can run multiple createrepo in parallel, it can help to avoid
race conditions caused by accessing or creating the same rpm database
by multiple createrepo instances at the same time.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 .../createrepo/createrepo/createrepo-dbpath.patch  | 51 ++++++++++++++++++++++
 .../createrepo/createrepo_0.4.11.bb                |  1 +
 2 files changed, 52 insertions(+)
 create mode 100644 meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch

diff --git a/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
new file mode 100644
index 0000000..2b113bf
--- /dev/null
+++ b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
@@ -0,0 +1,51 @@
+--- createrepo-0.4.11.orig/genpkgmetadata.py	2015-03-30 22:18:19.904000000 +0300
++++ createrepo-0.4.11.orig/genpkgmetadata.py	2015-03-30 22:28:49.208000000 +0300
+@@ -65,6 +65,7 @@
+      -p, --pretty = output xml files in pretty format.
+      --update = update existing metadata (if present)
+      -d, --database = generate the sqlite databases.
++     --dbpath <dir> = specify path to rpm db directory.
+     """)
+ 
+     sys.exit(retval)
+@@ -72,10 +73,13 @@
+ class MetaDataGenerator:
+     def __init__(self, cmds):
+         self.cmds = cmds
+-        self.ts = rpm.TransactionSet()
+         self.pkgcount = 0
+         self.files = []
+ 
++        if self.cmds['dbpath']:
++            rpm.addMacro("_dbpath", self.cmds['dbpath'])
++        self.ts = rpm.TransactionSet()
++
+     def _os_path_walk(self, top, func, arg):
+         """Directory tree walk with callback function.
+          copy of os.path.walk, fixes the link/stating problem
+@@ -435,6 +439,7 @@
+     cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*']
+     cmds['skip-symlinks'] = False
+     cmds['pkglist'] = []
++    cmds['dbpath'] = None
+ 
+     try:
+         gopts, argsleft = getopt.getopt(args, 'phqVvndg:s:x:u:c:o:CSi:', ['help', 'exclude=',
+@@ -442,7 +447,7 @@
+                                                                   'baseurl=', 'groupfile=', 'checksum=',
+                                                                   'version', 'pretty', 'split', 'outputdir=',
+                                                                   'noepoch', 'checkts', 'database', 'update',
+-                                                                  'skip-symlinks', 'pkglist='])
++                                                                  'skip-symlinks', 'pkglist=', 'dbpath='])
+     except getopt.error, e:
+         errorprint(_('Options Error: %s.') % e)
+         usage()
+@@ -516,6 +521,8 @@
+                 cmds['skip-symlinks'] = True
+             elif arg in ['-i', '--pkglist']:
+                 cmds['pkglist'] = a
++            elif arg == '--dbpath':
++                cmds['dbpath'] = os.path.realpath(a)
+                                 
+     except ValueError, e:
+         errorprint(_('Options Error: %s') % e)
diff --git a/meta/recipes-support/createrepo/createrepo_0.4.11.bb b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
index 49b45fc..adc193e 100644
--- a/meta/recipes-support/createrepo/createrepo_0.4.11.bb
+++ b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
@@ -14,6 +14,7 @@ SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \
           file://python-scripts-should-use-interpreter-from-env.patch \
 	  file://createrepo-rpm549.patch \
 	  file://recommends.patch \
+	  file://createrepo-dbpath.patch \
 	  file://rpm-createsolvedb.py \
          "
 
-- 
2.1.4

Re: [PATCH] createrepo: Implement --dbpath command line option

[Ed Bartosh]

On Wed, Apr 01, 2015 at 03:06:52PM +0300, Ed Bartosh wrote:
> --dbpath option can be used in cases where users don't want
> createrepo to use system rpm db to avoid possible collisiouns
> with other programs.
> 
> For bitbake builds it would be possible to specify different
> databases even for every createrepo run. Considering that rootfs
> builds can run multiple createrepo in parallel, it can help to avoid
> race conditions caused by accessing or creating the same rpm database
> by multiple createrepo instances at the same time.
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  .../createrepo/createrepo/createrepo-dbpath.patch  | 51 ++++++++++++++++++++++
>  .../createrepo/createrepo_0.4.11.bb                |  1 +
>  2 files changed, 52 insertions(+)
>  create mode 100644 meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
> 
> diff --git a/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
> new file mode 100644
> index 0000000..2b113bf
> --- /dev/null
> +++ b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
> @@ -0,0 +1,51 @@
> +--- createrepo-0.4.11.orig/genpkgmetadata.py	2015-03-30 22:18:19.904000000 +0300
> ++++ createrepo-0.4.11.orig/genpkgmetadata.py	2015-03-30 22:28:49.208000000 +0300
> +@@ -65,6 +65,7 @@
> +      -p, --pretty = output xml files in pretty format.
> +      --update = update existing metadata (if present)
> +      -d, --database = generate the sqlite databases.
> ++     --dbpath <dir> = specify path to rpm db directory.
> +     """)
> + 
> +     sys.exit(retval)
> +@@ -72,10 +73,13 @@
> + class MetaDataGenerator:
> +     def __init__(self, cmds):
> +         self.cmds = cmds
> +-        self.ts = rpm.TransactionSet()
> +         self.pkgcount = 0
> +         self.files = []
> + 
> ++        if self.cmds['dbpath']:
> ++            rpm.addMacro("_dbpath", self.cmds['dbpath'])
> ++        self.ts = rpm.TransactionSet()
> ++
> +     def _os_path_walk(self, top, func, arg):
> +         """Directory tree walk with callback function.
> +          copy of os.path.walk, fixes the link/stating problem
> +@@ -435,6 +439,7 @@
> +     cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*']
> +     cmds['skip-symlinks'] = False
> +     cmds['pkglist'] = []
> ++    cmds['dbpath'] = None
> + 
> +     try:
> +         gopts, argsleft = getopt.getopt(args, 'phqVvndg:s:x:u:c:o:CSi:', ['help', 'exclude=',
> +@@ -442,7 +447,7 @@
> +                                                                   'baseurl=', 'groupfile=', 'checksum=',
> +                                                                   'version', 'pretty', 'split', 'outputdir=',
> +                                                                   'noepoch', 'checkts', 'database', 'update',
> +-                                                                  'skip-symlinks', 'pkglist='])
> ++                                                                  'skip-symlinks', 'pkglist=', 'dbpath='])
> +     except getopt.error, e:
> +         errorprint(_('Options Error: %s.') % e)
> +         usage()
> +@@ -516,6 +521,8 @@
> +                 cmds['skip-symlinks'] = True
> +             elif arg in ['-i', '--pkglist']:
> +                 cmds['pkglist'] = a
> ++            elif arg == '--dbpath':
> ++                cmds['dbpath'] = os.path.realpath(a)
> +                                 
> +     except ValueError, e:
> +         errorprint(_('Options Error: %s') % e)
> diff --git a/meta/recipes-support/createrepo/createrepo_0.4.11.bb b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
> index 49b45fc..adc193e 100644
> --- a/meta/recipes-support/createrepo/createrepo_0.4.11.bb
> +++ b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
> @@ -14,6 +14,7 @@ SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \
>            file://python-scripts-should-use-interpreter-from-env.patch \
>  	  file://createrepo-rpm549.patch \
>  	  file://recommends.patch \
> +	  file://createrepo-dbpath.patch \
>  	  file://rpm-createsolvedb.py \
>           "
>  

Please, don't apply this patch. I've just sent updated one. Sorry for the confusion.


-- 
Regards,
Ed

[PATCH] createrepo: Implement --dbpath command line option

[Ed Bartosh]

--dbpath option can be used in cases where users don't want
createrepo to use system rpm db to avoid possible collisiouns
with other programs.

For bitbake builds it would be possible to specify different
databases even for every createrepo run. Considering that rootfs
builds can run multiple createrepo in parallel, it can help to avoid
race conditions caused by accessing or creating the same rpm database
by multiple createrepo instances at the same time.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 .../createrepo/createrepo/createrepo-dbpath.patch  | 61 ++++++++++++++++++++++
 .../createrepo/createrepo_0.4.11.bb                |  1 +
 2 files changed, 62 insertions(+)
 create mode 100644 meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch

diff --git a/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
new file mode 100644
index 0000000..7275598
--- /dev/null
+++ b/meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
@@ -0,0 +1,61 @@
+createrepo: Implement --dbpath command line option
+
+--dbpath option allows to specify path to the directory
+with rpm database. By default createrepo uses or creates
+rpm database in /var/lib/rpm/
+
+Upstream-Status: Pending
+
+Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
+
+--- createrepo-0.4.11.orig/genpkgmetadata.py 2015-03-30 22:18:19.904000000 +0300
++++ createrepo-0.4.11/genpkgmetadata.py 2015-03-30 22:28:49.208000000 +0300
+@@ -65,6 +65,7 @@
+      -p, --pretty = output xml files in pretty format.
+      --update = update existing metadata (if present)
+      -d, --database = generate the sqlite databases.
++     --dbpath <dir> = specify path to rpm db directory.
+     """)
+ 
+     sys.exit(retval)
+@@ -72,10 +73,13 @@
+ class MetaDataGenerator:
+     def __init__(self, cmds):
+         self.cmds = cmds
+-        self.ts = rpm.TransactionSet()
+         self.pkgcount = 0
+         self.files = []
+ 
++        if self.cmds['dbpath']:
++            rpm.addMacro("_dbpath", self.cmds['dbpath'])
++        self.ts = rpm.TransactionSet()
++
+     def _os_path_walk(self, top, func, arg):
+         """Directory tree walk with callback function.
+          copy of os.path.walk, fixes the link/stating problem
+@@ -435,6 +439,7 @@
+     cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*']
+     cmds['skip-symlinks'] = False
+     cmds['pkglist'] = []
++    cmds['dbpath'] = None
+ 
+     try:
+         gopts, argsleft = getopt.getopt(args, 'phqVvndg:s:x:u:c:o:CSi:', ['help', 'exclude=',
+@@ -442,7 +447,7 @@
+                                                                   'baseurl=', 'groupfile=', 'checksum=',
+                                                                   'version', 'pretty', 'split', 'outputdir=',
+                                                                   'noepoch', 'checkts', 'database', 'update',
+-                                                                  'skip-symlinks', 'pkglist='])
++                                                                  'skip-symlinks', 'pkglist=', 'dbpath='])
+     except getopt.error, e:
+         errorprint(_('Options Error: %s.') % e)
+         usage()
+@@ -516,6 +521,8 @@
+                 cmds['skip-symlinks'] = True
+             elif arg in ['-i', '--pkglist']:
+                 cmds['pkglist'] = a
++            elif arg == '--dbpath':
++                cmds['dbpath'] = os.path.realpath(a)
+                                 
+     except ValueError, e:
+         errorprint(_('Options Error: %s') % e)
diff --git a/meta/recipes-support/createrepo/createrepo_0.4.11.bb b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
index 49b45fc..adc193e 100644
--- a/meta/recipes-support/createrepo/createrepo_0.4.11.bb
+++ b/meta/recipes-support/createrepo/createrepo_0.4.11.bb
@@ -14,6 +14,7 @@ SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \
           file://python-scripts-should-use-interpreter-from-env.patch \
 	  file://createrepo-rpm549.patch \
 	  file://recommends.patch \
+	  file://createrepo-dbpath.patch \
 	  file://rpm-createsolvedb.py \
          "
 
-- 
2.1.4