what is the state of meta-cloud-services, re: chef/puppet/ruby recipes?

what is the state of meta-cloud-services, re: chef/puppet/ruby recipes?

[Robert P. J. Day]

  followup to earlier post of mine on main yocto mailing list, now
that i realize this is the right place.

  using gatesgarth version of layers, i want to test build puppet,
chef and a number of ruby recipes, then dig into how to extend/create
my own ruby recipes, but i want to first verify what shape the
meta-cloud-services layer is in.

  if i use "qemux86-64" as my target, i started off with "bitbake
puppet" and that seemed to work (it didn't in earlier versions, but i
see no point rehashing that here).

  i then moved on to "bitbake chef", hoping to verify that an
absolutely generic build would work -- customization would come
later. now, even though chef doesn't need "yard", it does depend on
yard-native, and i have other needs for "yard" later so i tried:

  $ bitbake yard

and got:

ERROR: yard-0.8.7.6-r0 do_fetch: Fetcher failure: Unable to find
revision d83194e1a09098ec5be28b616cde3b9a15380873 in branch master
even from upstream
ERROR: yard-0.8.7.6-r0 do_fetch: Fetcher failure for URL:
'git://github.com/lsegal/yard.git'. Unable to fetch URL from any
source.
ERROR: Logfile of failure stored in:
/home/rpjday/oe/builds/puppet/build/tmp/work/core2-64-poky-linux/yard/0.8.7.6-r0/temp/log.do_fetch.1418067
ERROR: Task
(/home/rpjday/oe/dist/layers/meta-cloud-services/meta-openstack/recipes-devtools/ruby/yard_git.bb:do_fetch)
failed with exit code '1'

well, there's an obvious reason for that -- yard is one of the many
recipes that succumbed to political correctness and renamed "master"
to "main", so a quick patch:

   SRC_URI = " \
  -    git://github.com/lsegal/yard.git \
  +    git://github.com/lsegal/yard.git;branch=main \
       "

resolved that issue and i got a build for yard, but that's a bit
worrisome -- that seems like a really blatant error in that it clearly
won't even allow yard to build. obviously, i can submit a patch, but
it makes me wonder what other trivial gotchas are waiting for me.

  with that patch in place, i figured i'd try the following, which i
would need for chef, anyway:

  $ bitbake yard-native

ERROR: yard-native-0.8.7.6-r0 do_compile: Execution of
'/home/rpjday/oe/builds/puppet/build/tmp/work/x86_64-linux/yard-native/0.8.7.6-r0/temp/run.do_compile.1419759'
failed with exit code 1:
ERROR:  Gemspec file not found: yard-native.gemspec.gemspec

  i'm still digging into gemspec files, but it looks really weird that
the file suffix is ".gemspec.gemspec", as if the code somewhere is
accidentally adding one of the suffixes, not realizing something else
will be adding it a second time.

  am i investing my time wisely? has anyone else got a working
puppet/chef/general ruby build environment up and running? sorry for
not being more specific, but i don't want to spend hours only to find
that no one is looking after this layer.

rday

Re: [meta-virtualization] what is the state of meta-cloud-services, re: chef/puppet/ruby recipes?

[Bruce Ashfield]

On Sat, Dec 26, 2020 at 1:28 PM Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
>
>   followup to earlier post of mine on main yocto mailing list, now
> that i realize this is the right place.
>
>   using gatesgarth version of layers, i want to test build puppet,
> chef and a number of ruby recipes, then dig into how to extend/create
> my own ruby recipes, but i want to first verify what shape the
> meta-cloud-services layer is in.
>
>   if i use "qemux86-64" as my target, i started off with "bitbake
> puppet" and that seemed to work (it didn't in earlier versions, but i
> see no point rehashing that here).
>
>   i then moved on to "bitbake chef", hoping to verify that an
> absolutely generic build would work -- customization would come
> later. now, even though chef doesn't need "yard", it does depend on
> yard-native, and i have other needs for "yard" later so i tried:
>
>   $ bitbake yard
>
> and got:
>
> ERROR: yard-0.8.7.6-r0 do_fetch: Fetcher failure: Unable to find
> revision d83194e1a09098ec5be28b616cde3b9a15380873 in branch master
> even from upstream
> ERROR: yard-0.8.7.6-r0 do_fetch: Fetcher failure for URL:
> 'git://github.com/lsegal/yard.git'. Unable to fetch URL from any
> source.
> ERROR: Logfile of failure stored in:
> /home/rpjday/oe/builds/puppet/build/tmp/work/core2-64-poky-linux/yard/0.8.7.6-r0/temp/log.do_fetch.1418067
> ERROR: Task
> (/home/rpjday/oe/dist/layers/meta-cloud-services/meta-openstack/recipes-devtools/ruby/yard_git.bb:do_fetch)
> failed with exit code '1'
>
> well, there's an obvious reason for that -- yard is one of the many
> recipes that succumbed to political correctness and renamed "master"
> to "main", so a quick patch:
>
>    SRC_URI = " \
>   -    git://github.com/lsegal/yard.git \
>   +    git://github.com/lsegal/yard.git;branch=main \
>        "
>
> resolved that issue and i got a build for yard, but that's a bit
> worrisome -- that seems like a really blatant error in that it clearly
> won't even allow yard to build. obviously, i can submit a patch, but
> it makes me wonder what other trivial gotchas are waiting for me.
>
>   with that patch in place, i figured i'd try the following, which i
> would need for chef, anyway:
>
>   $ bitbake yard-native
>
> ERROR: yard-native-0.8.7.6-r0 do_compile: Execution of
> '/home/rpjday/oe/builds/puppet/build/tmp/work/x86_64-linux/yard-native/0.8.7.6-r0/temp/run.do_compile.1419759'
> failed with exit code 1:
> ERROR:  Gemspec file not found: yard-native.gemspec.gemspec
>
>   i'm still digging into gemspec files, but it looks really weird that
> the file suffix is ".gemspec.gemspec", as if the code somewhere is
> accidentally adding one of the suffixes, not realizing something else
> will be adding it a second time.
>
>   am i investing my time wisely? has anyone else got a working
> puppet/chef/general ruby build environment up and running? sorry for
> not being more specific, but i don't want to spend hours only to find
> that no one is looking after this layer.

meta-cloud-services itself is maintained, but it doesn't go through
global builds or extensive system level testing.

We are keeping the recipes that different users care about, up to
date, and building against the specified yocto release. But other
layers (like meta-openstack) are in a state of waiting for interest
and contributors to become fully functional again.

I'm also migrating some parts of meta-cloud-services to meta-virt
(i.e. I just pulled cloud-init over), when they get broader interest
or have a more generic use case.

As for the chef/puppet parts, the original implementation was
problematic, so I had to undo parts of its recipe changes, and we
haven't had a call for using it as a deployment method since. So the
base support is still around, but as for using it to deploy an image,
that work is custom and typically (if at all) done in other layers.
Just like other parts, I'd spruce it up (or move recipes around to
other layers), if we get more interest.

No one has done much with the ruby support in quite some time, so it
would need work as well. I thought there was a meta-ruby floating
around, but I couldn't find it on the layerindex, so it looks like
just oe-core + some work would be required.

Bruce

>
> rday
>
>
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] what is the state of meta-cloud-services, re: chef/puppet/ruby recipes?

[Robert P. J. Day]

On Sun, 27 Dec 2020, Bruce Ashfield wrote:

... snip ...

> No one has done much with the ruby support in quite some time, so it
> would need work as well. I thought there was a meta-ruby floating
> around, but I couldn't find it on the layerindex, so it looks like
> just oe-core + some work would be required.

  dumb question alert: i'm currently going through all of the ruby
recipes in meta-openstack, doing the occasional fixup to get them to
build on the master branch, and i can see that all(?) of those recipes
define a SRC_URI that downloads a proper .gemspec file for each
recipe, which allows the recipe to build.

  *however*, even though a number of those recipes BBCLASSEXTEND to
class "native", it's not possible to build the native variants as the
error is that there is no gemspec file for the native variant. which
is why i get:

  $ bitbake yard-native
  ...
  Log data follows:
  | DEBUG: Executing shell function do_compile
  | ERROR:  Gemspec file not found: yard-native.gemspec
  ... snip ...

what's the magic to build native ruby recipes? does one actually have
to define a distinct native version of the gemspec file for each
recipe?

rday

Re: [meta-virtualization] what is the state of meta-cloud-services, re: chef/puppet/ruby recipes?

[Bruce Ashfield]

On Sat, Jan 2, 2021 at 9:20 AM Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
> On Sun, 27 Dec 2020, Bruce Ashfield wrote:
>
> ... snip ...
>
> > No one has done much with the ruby support in quite some time, so it
> > would need work as well. I thought there was a meta-ruby floating
> > around, but I couldn't find it on the layerindex, so it looks like
> > just oe-core + some work would be required.
>
>   dumb question alert: i'm currently going through all of the ruby
> recipes in meta-openstack, doing the occasional fixup to get them to
> build on the master branch, and i can see that all(?) of those recipes
> define a SRC_URI that downloads a proper .gemspec file for each
> recipe, which allows the recipe to build.
>
>   *however*, even though a number of those recipes BBCLASSEXTEND to
> class "native", it's not possible to build the native variants as the
> error is that there is no gemspec file for the native variant. which
> is why i get:
>
>   $ bitbake yard-native
>   ...
>   Log data follows:
>   | DEBUG: Executing shell function do_compile
>   | ERROR:  Gemspec file not found: yard-native.gemspec
>   ... snip ...
>
> what's the magic to build native ruby recipes? does one actually have
> to define a distinct native version of the gemspec file for each
> recipe?

I honestly can't say. I did little but review and merge the ruby
changes. There was a primary author for the changes, and I haven't
heard from them for some time.

I don't recall them really being used in the -native variant, so it
could / should probably just be a mass removal.

Do you have a use case on the -native side of things ?

Bruce

>
> rday



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] what is the state of meta-cloud-services, re: chef/puppet/ruby recipes?

[Robert P. J. Day]

On Sun, 27 Dec 2020, Bruce Ashfield wrote:

> No one has done much with the ruby support in quite some time, so it
> would need work as well. I thought there was a meta-ruby floating
> around, but I couldn't find it on the layerindex, so it looks like
> just oe-core + some work would be required.

  random thoughts after messing with various aspects of ruby for the
last couple days.

  first, would it not make sense to move the ruby.bbclass file to
oe-core? given the fundamental nature of ruby.bbclass, it seems
inappropriate to have it in a specialized layer like
meta-cloud-services. if oe-core has class files related to things like
perl and python, it would seem like ruby.bbclass belongs there as
well.

  also, i had to hack ruby.bbclass given the python 3 distinction
between strings and bytes in order to get some ruby variables to be
set properly:

diff --git a/classes/ruby.bbclass b/classes/ruby.bbclass
index 15ac9f4b..585d128d 100644
--- a/classes/ruby.bbclass
+++ b/classes/ruby.bbclass
@@ -26,7 +26,7 @@ def get_rubyversion(p):
     version = subprocess.Popen([cmd, "--version"],
stdout=subprocess.PIPE).communicate()[0]

     r = re.compile("ruby ([0-9]+\.[0-9]+\.[0-9]+)*")
-    m = r.match(version)
+    m = r.match(version.decode("utf-8"))
     if m:
         found_version = m.group(1)

@@ -44,6 +44,7 @@ def get_rubygemslocation(p):
        return found_loc

     loc = subprocess.Popen([cmd, "env"], stdout=subprocess.PIPE).communicate()[0]
+    loc = loc.decode("utf-8")

     r = re.compile(".*\- (/usr.*/ruby/gems/.*)")
     for line in loc.split('\n'):

  i'm sure there's a cleaner way to do that ... i can submit that, but
i'm not sure which ML to give it to.

  in any event, i'm still beating up on ruby, and it would be nice to
formalize how to get this organized.

rday

Re: [meta-virtualization] what is the state of meta-cloud-services, re: chef/puppet/ruby recipes?

[Bruce Ashfield]

On Sat, Jan 2, 2021 at 3:07 PM Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
> On Sun, 27 Dec 2020, Bruce Ashfield wrote:
>
> > No one has done much with the ruby support in quite some time, so it
> > would need work as well. I thought there was a meta-ruby floating
> > around, but I couldn't find it on the layerindex, so it looks like
> > just oe-core + some work would be required.
>
>   random thoughts after messing with various aspects of ruby for the
> last couple days.
>
>   first, would it not make sense to move the ruby.bbclass file to
> oe-core? given the fundamental nature of ruby.bbclass, it seems
> inappropriate to have it in a specialized layer like
> meta-cloud-services. if oe-core has class files related to things like
> perl and python, it would seem like ruby.bbclass belongs there as
> well.

It is a possibility, and was considered in the past. But oe-core
doesn't currently have any ruby recipes (just the toolchain), so it
isn't common enough for core. We had a lot of ruby in the openstack
world, so the extended support was created there.

>
>   also, i had to hack ruby.bbclass given the python 3 distinction
> between strings and bytes in order to get some ruby variables to be
> set properly:
>
> diff --git a/classes/ruby.bbclass b/classes/ruby.bbclass
> index 15ac9f4b..585d128d 100644
> --- a/classes/ruby.bbclass
> +++ b/classes/ruby.bbclass
> @@ -26,7 +26,7 @@ def get_rubyversion(p):
>      version = subprocess.Popen([cmd, "--version"],
> stdout=subprocess.PIPE).communicate()[0]
>
>      r = re.compile("ruby ([0-9]+\.[0-9]+\.[0-9]+)*")
> -    m = r.match(version)
> +    m = r.match(version.decode("utf-8"))
>      if m:
>          found_version = m.group(1)
>
> @@ -44,6 +44,7 @@ def get_rubygemslocation(p):
>         return found_loc
>
>      loc = subprocess.Popen([cmd, "env"], stdout=subprocess.PIPE).communicate()[0]
> +    loc = loc.decode("utf-8")
>
>      r = re.compile(".*\- (/usr.*/ruby/gems/.*)")
>      for line in loc.split('\n'):
>
>   i'm sure there's a cleaner way to do that ... i can submit that, but
> i'm not sure which ML to give it to.

The change should be good enough, that's the typical python3.x technique.

This is the right mailing list for any changes.

Bruce

>
>   in any event, i'm still beating up on ruby, and it would be nice to
> formalize how to get this organized.
>
> rday



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II