Building on MacOS X

Building on MacOS X

[Roger Smith]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

Is there any documentation for running the Yocto build system on Mac OS X or macOS as Apple now calls it? I am working with the Intel Aero board. Before I go down the rabbit hole of fixing issues like this one (and I am using the bash shell), I’d like to know if anyone has build it on os x before.

thanks


source oe-init-build-env
readlink: illegal option -- f
usage: readlink [-n] [file ...]
-bash: /scripts/oe-buildenv-internal: No such file or directory

[-- Attachment #2: Type: text/html, Size: 1428 bytes --]

Re: Building on MacOS X

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com> wrote:

> Is there any documentation for running the Yocto build system on Mac OS X
> or macOS as Apple now calls it? I am working with the Intel Aero board.
> Before I go down the rabbit hole of fixing issues like this one (and I am
> using the bash shell), I’d like to know if anyone has build it on os x
> before.
>

If you install all of the GNU tools using brew or similar and put them
first on $PATH then you can get bitbake started.  Then you need to stub out
the linux-specific bits in bitbake.  I've previously started on this work
already (
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin).
The next step is figuring out how to configure OE to build and link
natively on OSX using LLVM instead of GCC.

However all of this is mostly academic because in Sierra (iirc) onwards
there is tighter security on processes, which means that pseudo won't work
even if you port it to macOS.

So unless you fancy some non-trivial engineering the short version is just
use something like Docker to run a Linux system on your Mac.

Ross

[-- Attachment #2: Type: text/html, Size: 1814 bytes --]

Re: Building on MacOS X

[Mark Hatle]

On 1/12/17 9:14 AM, Roger Smith wrote:
> Is there any documentation for running the Yocto build system on Mac OS X or
> macOS as Apple now calls it? I am working with the Intel Aero board. Before I go
> down the rabbit hole of fixing issues like this one (and I am using the bash
> shell), I’d like to know if anyone has build it on os x before.

As far as I am aware, nobody has ever finished this work.

There are numerous places (scripts primarily) that will expect GNU
util-linux/coreutils (and similar) extensions.

In addition, the pseudo program, used to emulate filesystem permissions and
related items only partially works on MacOS.  (I know it was working at one
point, but Apple changed some of the system properties [preloaded library] for
security reasons, and ended up making pseudo no longer work.  You may want to
look into this before getting too far in.  Without pseudo, there is no way to
perform a build/package.)

Some -very- old work:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib-archive/refs/heads

search for macosx

(I'd love to see a native bitbake/oe-core work on MacOS X... but as far as I
know the work stalled a while back due to lack of interest by people.)

Most of the people I know using MacOS X for development are using virtual box
(or similar) and a Linux based VM.

--Mark

> thanks
> 
> 
> source oe-init-build-env
> readlink: illegal option -- f
> usage: readlink [-n] [file ...]
> -bash: /scripts/oe-buildenv-internal: No such file or directory
> 
> 

Re: Building on MacOS X

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]

You can also build using Docker containers:
https://github.com/crops/docker-win-mac-docs/wiki <https://github.com/crops/docker-win-mac-docs/wiki>

> On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com> wrote:
> 
> 
> On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com <mailto:roger@sentientblue.com>> wrote:
> Is there any documentation for running the Yocto build system on Mac OS X or macOS as Apple now calls it? I am working with the Intel Aero board. Before I go down the rabbit hole of fixing issues like this one (and I am using the bash shell), I’d like to know if anyone has build it on os x before.
> 
> If you install all of the GNU tools using brew or similar and put them first on $PATH then you can get bitbake started.  Then you need to stub out the linux-specific bits in bitbake.  I've previously started on this work already (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin <http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin>).  The next step is figuring out how to configure OE to build and link natively on OSX using LLVM instead of GCC.
> 
> However all of this is mostly academic because in Sierra (iirc) onwards there is tighter security on processes, which means that pseudo won't work even if you port it to macOS.
> 
> So unless you fancy some non-trivial engineering the short version is just use something like Docker to run a Linux system on your Mac.
> 
> Ross
> -- 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto


[-- Attachment #2: Type: text/html, Size: 2885 bytes --]

Re: Building on MacOS X

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

On 12 January 2017 at 15:39, Tim Orling <timothy.t.orling@linux.intel.com>
wrote:

> You can also build using Docker containers:
> https://github.com/crops/docker-win-mac-docs/wiki
>

Yes, this is the link I was failing to find, thanks Tim.  This is basically
the official way of using OE on Windows or Mac.

Ross

[-- Attachment #2: Type: text/html, Size: 796 bytes --]

Re: Building on MacOS X

[Belisko Marek]

On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
<timothy.t.orling@linux.intel.com> wrote:
> You can also build using Docker containers:
> https://github.com/crops/docker-win-mac-docs/wiki
Well the re is other limitation about slow filesystem access from
docker on osx. There is workaround to use nfs but it's not possible to
use nfs for building yocto - so it's kind of chicken-egg problem ;)
>
> On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com> wrote:
>
>
> On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com> wrote:
>>
>> Is there any documentation for running the Yocto build system on Mac OS X
>> or macOS as Apple now calls it? I am working with the Intel Aero board.
>> Before I go down the rabbit hole of fixing issues like this one (and I am
>> using the bash shell), I’d like to know if anyone has build it on os x
>> before.
>
>
> If you install all of the GNU tools using brew or similar and put them first
> on $PATH then you can get bitbake started.  Then you need to stub out the
> linux-specific bits in bitbake.  I've previously started on this work
> already
> (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin).
> The next step is figuring out how to configure OE to build and link natively
> on OSX using LLVM instead of GCC.
>
> However all of this is mostly academic because in Sierra (iirc) onwards
> there is tighter security on processes, which means that pseudo won't work
> even if you port it to macOS.
>
> So unless you fancy some non-trivial engineering the short version is just
> use something like Docker to run a Linux system on your Mac.
>
> Ross
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

marek

-- 
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com

Re: Building on MacOS X

[Khem Raj]

On Thu, Jan 12, 2017 at 8:21 AM, Belisko Marek <marek.belisko@gmail.com> wrote:
> On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
> <timothy.t.orling@linux.intel.com> wrote:
>> You can also build using Docker containers:
>> https://github.com/crops/docker-win-mac-docs/wiki
> Well the re is other limitation about slow filesystem access from
> docker on osx. There is workaround to use nfs but it's not possible to
> use nfs for building yocto - so it's kind of chicken-egg problem ;)

virtualbox works well unless you passionately dont want to use linux
on build host
>>
>> On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com> wrote:
>>
>>
>> On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com> wrote:
>>>
>>> Is there any documentation for running the Yocto build system on Mac OS X
>>> or macOS as Apple now calls it? I am working with the Intel Aero board.
>>> Before I go down the rabbit hole of fixing issues like this one (and I am
>>> using the bash shell), I’d like to know if anyone has build it on os x
>>> before.
>>
>>
>> If you install all of the GNU tools using brew or similar and put them first
>> on $PATH then you can get bitbake started.  Then you need to stub out the
>> linux-specific bits in bitbake.  I've previously started on this work
>> already
>> (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin).
>> The next step is figuring out how to configure OE to build and link natively
>> on OSX using LLVM instead of GCC.
>>
>> However all of this is mostly academic because in Sierra (iirc) onwards
>> there is tighter security on processes, which means that pseudo won't work
>> even if you port it to macOS.
>>
>> So unless you fancy some non-trivial engineering the short version is just
>> use something like Docker to run a Linux system on your Mac.
>>
>> Ross
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>>
>>
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>
> marek
>
> --
> as simple and primitive as possible
> -------------------------------------------------
> Marek Belisko - OPEN-NANDRA
> Freelance Developer
>
> Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
> Tel: +421 915 052 184
> skype: marekwhite
> twitter: #opennandra
> web: http://open-nandra.com
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: Building on MacOS X

[Andrea Galbusera]

[-- Attachment #1: Type: text/plain, Size: 4234 bytes --]

On Thu, Jan 12, 2017 at 5:21 PM, Belisko Marek <marek.belisko@gmail.com>
wrote:

> On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
> <timothy.t.orling@linux.intel.com> wrote:
> > You can also build using Docker containers:
> > https://github.com/crops/docker-win-mac-docs/wiki
> Well the re is other limitation about slow filesystem access from
> docker on osx. There is workaround to use nfs but it's not possible to
> use nfs for building yocto - so it's kind of chicken-egg problem ;)
>

I shortly tested the CROPS docker-based setup after watching some
presentation at ELCE 2016 in Berlin. It basically worked but I experienced
the filesystem slowness your are talking about. I ended up waiting hours to
see a simple core-image-minimal build complete (even after giving more
cores to docker). One more point is that slightly more complex build
scenarios, i.e. building resin.os, also required tweaking docker run
parameters for the build container in order to give bitbake access to
features like loop devices it needed (not always easily debuggable issues
indeed). Turned out I decided to stick with more canonical linux based
environments for the moment.

Anyway, the technology behind CROPS is *very* interesting to me, and I'd
like to hear from people closely involved (Tim?) what the state of the art
is and what we can expect to see in the near future. IIRC, the roadmap for
Yocto 2.3 release was supposed to resurrect the Eclipse plugin and adopt
CROPS as an alternative for running eSDK in a seamless way on different
development host OSs. Beside from the images on docker hub and the github
projects that didn't have high activity in the latest months, I hardly find
discussions and documentation on the whole approach. Isn't this hot enough
anymore or are there big issues that will prevent this technology from
taking off. I often manage SDKs for Windows-minded developers and I
strongly yearn to find a better approach to help them feel at home while
building stuff for OE/Yocto based systems...



> >
> > On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com> wrote:
> >
> >
> > On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com> wrote:
> >>
> >> Is there any documentation for running the Yocto build system on Mac OS
> X
> >> or macOS as Apple now calls it? I am working with the Intel Aero board.
> >> Before I go down the rabbit hole of fixing issues like this one (and I
> am
> >> using the bash shell), I’d like to know if anyone has build it on os x
> >> before.
> >
> >
> > If you install all of the GNU tools using brew or similar and put them
> first
> > on $PATH then you can get bitbake started.  Then you need to stub out the
> > linux-specific bits in bitbake.  I've previously started on this work
> > already
> > (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/
> log/?h=ross/darwin).
> > The next step is figuring out how to configure OE to build and link
> natively
> > on OSX using LLVM instead of GCC.
> >
> > However all of this is mostly academic because in Sierra (iirc) onwards
> > there is tighter security on processes, which means that pseudo won't
> work
> > even if you port it to macOS.
> >
> > So unless you fancy some non-trivial engineering the short version is
> just
> > use something like Docker to run a Linux system on your Mac.
> >
> > Ross
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
> >
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
>
> marek
>
> --
> as simple and primitive as possible
> -------------------------------------------------
> Marek Belisko - OPEN-NANDRA
> Freelance Developer
>
> Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
> Tel: +421 915 052 184
> skype: marekwhite
> twitter: #opennandra
> web: http://open-nandra.com
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 6139 bytes --]

Re: Building on MacOS X

[Roger Smith]

[-- Attachment #1: Type: text/plain, Size: 6715 bytes --]

 I have Parallels (running on El Capitan the one before Sierra)  and ubuntu 14  running my current build environment on a MacBook Pro, but boy is the build slow… I also worked at Apple for 19 years on drivers inside MacOS X/iOS, so I am more than motivated to have this working natively rather than inside any container or disk space hogging environment.  As I mentioned I am working with the Intel Aero compute board, so slogging though all this fat to build an image is a productivity killer.

I think most of the  incompatibilities between Linux and os x (which btw is coming from the ios side of the fence unfortunately), can be mitigated with boot args or via the command line . Apple’s compiler team had to make llvm compatible with gcc, so I am surprised if in 2017, there are compiler issues to building for an x86_64 platform with llvm on the Mac . That’s the kind of bug Apple likes to fix promptly.. 

As I mentioned, I tried to simply source the oe-init-build-env, and got an error that the readlink command that yocto is using is incompatible with the bsd version of readlink built into os x. 

i.e when I run .

source oe-init-build-env

I get the error

readlink: illegal option -- f

Which is because on OS X readlink doesn’t specify -f

YNOPSIS
     stat [-FLnq] [-f format | -l | -r | -s | -x] [-t timefmt] [file ...]
     readlink [-n] [file ...]

I didn’t want to go through this level of change in the Yocto sources if (1) people don’t care to take changes or (2) it had already been done before.. I was curious how far down this rabbit hole people had gone before..

Roger


> On Jan 12, 2017, at 8:50 AM, Andrea Galbusera <gizero@gmail.com> wrote:
> 
> On Thu, Jan 12, 2017 at 5:21 PM, Belisko Marek <marek.belisko@gmail.com <mailto:marek.belisko@gmail.com>> wrote:
> On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
> <timothy.t.orling@linux.intel.com <mailto:timothy.t.orling@linux.intel.com>> wrote:
> > You can also build using Docker containers:
> > https://github.com/crops/docker-win-mac-docs/wiki <https://github.com/crops/docker-win-mac-docs/wiki>
> Well the re is other limitation about slow filesystem access from
> docker on osx. There is workaround to use nfs but it's not possible to
> use nfs for building yocto - so it's kind of chicken-egg problem ;)
> 
> I shortly tested the CROPS docker-based setup after watching some presentation at ELCE 2016 in Berlin. It basically worked but I experienced the filesystem slowness your are talking about. I ended up waiting hours to see a simple core-image-minimal build complete (even after giving more cores to docker). One more point is that slightly more complex build scenarios, i.e. building resin.os, also required tweaking docker run parameters for the build container in order to give bitbake access to features like loop devices it needed (not always easily debuggable issues indeed). Turned out I decided to stick with more canonical linux based environments for the moment.
> 
> Anyway, the technology behind CROPS is *very* interesting to me, and I'd like to hear from people closely involved (Tim?) what the state of the art is and what we can expect to see in the near future. IIRC, the roadmap for Yocto 2.3 release was supposed to resurrect the Eclipse plugin and adopt CROPS as an alternative for running eSDK in a seamless way on different development host OSs. Beside from the images on docker hub and the github projects that didn't have high activity in the latest months, I hardly find discussions and documentation on the whole approach. Isn't this hot enough anymore or are there big issues that will prevent this technology from taking off. I often manage SDKs for Windows-minded developers and I strongly yearn to find a better approach to help them feel at home while building stuff for OE/Yocto based systems... 
> 
>  
> >
> > On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com <mailto:ross.burton@intel.com>> wrote:
> >
> >
> > On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com <mailto:roger@sentientblue.com>> wrote:
> >>
> >> Is there any documentation for running the Yocto build system on Mac OS X
> >> or macOS as Apple now calls it? I am working with the Intel Aero board.
> >> Before I go down the rabbit hole of fixing issues like this one (and I am
> >> using the bash shell), I’d like to know if anyone has build it on os x
> >> before.
> >
> >
> > If you install all of the GNU tools using brew or similar and put them first
> > on $PATH then you can get bitbake started.  Then you need to stub out the
> > linux-specific bits in bitbake.  I've previously started on this work
> > already
> > (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin <http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin>).
> > The next step is figuring out how to configure OE to build and link natively
> > on OSX using LLVM instead of GCC.
> >
> > However all of this is mostly academic because in Sierra (iirc) onwards
> > there is tighter security on processes, which means that pseudo won't work
> > even if you port it to macOS.
> >
> > So unless you fancy some non-trivial engineering the short version is just
> > use something like Docker to run a Linux system on your Mac.
> >
> > Ross
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> > https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>
> >
> >
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> > https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>
> >
> 
> marek
> 
> --
> as simple and primitive as possible
> -------------------------------------------------
> Marek Belisko - OPEN-NANDRA
> Freelance Developer
> 
> Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
> Tel: +421 915 052 184 <tel:%2B421%20915%20052%20184>
> skype: marekwhite
> twitter: #opennandra
> web: http://open-nandra.com <http://open-nandra.com/>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>
> 
> -- 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>

[-- Attachment #2: Type: text/html, Size: 16111 bytes --]

Re: Building on MacOS X

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

On 12 January 2017 at 17:41, Roger Smith <roger@sentientblue.com> wrote:

> As I mentioned, I tried to simply source the oe-init-build-env, and got an
> error that the readlink command that yocto is using is incompatible with
> the bsd version of readlink built into os x.
>

Yes, we assume GNU tools.  That was the first step I posted: use brew to
install gnu awk/coreutils/etc etc.  But thats also the easy bit...

Ross

[-- Attachment #2: Type: text/html, Size: 800 bytes --]

Re: Building on MacOS X

[Maciej Borzęcki]

On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
<timothy.t.orling@linux.intel.com> wrote:
> You can also build using Docker containers:
> https://github.com/crops/docker-win-mac-docs/wiki

IIRC docker on mac relies on docker-machine, which in turn spins up a
virtualbox VM.


>
> On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com> wrote:
>
>
> On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com> wrote:
>>
>> Is there any documentation for running the Yocto build system on Mac OS X
>> or macOS as Apple now calls it? I am working with the Intel Aero board.
>> Before I go down the rabbit hole of fixing issues like this one (and I am
>> using the bash shell), I’d like to know if anyone has build it on os x
>> before.
>
>
> If you install all of the GNU tools using brew or similar and put them first
> on $PATH then you can get bitbake started.  Then you need to stub out the
> linux-specific bits in bitbake.  I've previously started on this work
> already
> (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin).
> The next step is figuring out how to configure OE to build and link natively
> on OSX using LLVM instead of GCC.
>
> However all of this is mostly academic because in Sierra (iirc) onwards
> there is tighter security on processes, which means that pseudo won't work
> even if you port it to macOS.
>
> So unless you fancy some non-trivial engineering the short version is just
> use something like Docker to run a Linux system on your Mac.
>
> Ross
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>



-- 
Maciej Borzecki
RnDity

Re: Building on MacOS X

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 343 bytes --]

On 12 January 2017 at 17:55, Maciej Borzęcki <maciej.borzecki@rndity.com>
wrote:

> IIRC docker on mac relies on docker-machine, which in turn spins up a
> virtualbox VM.
>

That's the old Docker (Docker Toolbox), the new Docker (Docker for Mac)
uses the built-in hypervisor so basically works exactly like Docker on
Linux.

Ross

[-- Attachment #2: Type: text/html, Size: 755 bytes --]

Re: Building on MacOS X

[Andrea Galbusera]

[-- Attachment #1: Type: text/plain, Size: 2522 bytes --]

On Thu, Jan 12, 2017 at 6:55 PM, Maciej Borzęcki <maciej.borzecki@rndity.com
> wrote:

> On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
> <timothy.t.orling@linux.intel.com> wrote:
> > You can also build using Docker containers:
> > https://github.com/crops/docker-win-mac-docs/wiki
>
> IIRC docker on mac relies on docker-machine, which in turn spins up a
> virtualbox VM.
>

Not anymore! There's a native implementation [1] but still a linux kernel
around anyway! ;-)

[1]
https://www.docker.com/docker-news-and-press/docker-released-native-mac-and-windows-apps-optimize-developer-experience


>
>
> >
> > On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com> wrote:
> >
> >
> > On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com> wrote:
> >>
> >> Is there any documentation for running the Yocto build system on Mac OS
> X
> >> or macOS as Apple now calls it? I am working with the Intel Aero board.
> >> Before I go down the rabbit hole of fixing issues like this one (and I
> am
> >> using the bash shell), I’d like to know if anyone has build it on os x
> >> before.
> >
> >
> > If you install all of the GNU tools using brew or similar and put them
> first
> > on $PATH then you can get bitbake started.  Then you need to stub out the
> > linux-specific bits in bitbake.  I've previously started on this work
> > already
> > (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/
> log/?h=ross/darwin).
> > The next step is figuring out how to configure OE to build and link
> natively
> > on OSX using LLVM instead of GCC.
> >
> > However all of this is mostly academic because in Sierra (iirc) onwards
> > there is tighter security on processes, which means that pseudo won't
> work
> > even if you port it to macOS.
> >
> > So unless you fancy some non-trivial engineering the short version is
> just
> > use something like Docker to run a Linux system on your Mac.
> >
> > Ross
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
> >
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
>
>
>
> --
> Maciej Borzecki
> RnDity
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 4503 bytes --]

Re: Building on MacOS X

[Maciej Borzęcki]

On Thu, Jan 12, 2017 at 7:12 PM, Andrea Galbusera <gizero@gmail.com> wrote:
> On Thu, Jan 12, 2017 at 6:55 PM, Maciej Borzęcki
> <maciej.borzecki@rndity.com> wrote:
>>
>> On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
>> <timothy.t.orling@linux.intel.com> wrote:
>> > You can also build using Docker containers:
>> > https://github.com/crops/docker-win-mac-docs/wiki
>>
>> IIRC docker on mac relies on docker-machine, which in turn spins up a
>> virtualbox VM.
>
>
> Not anymore! There's a native implementation [1] but still a linux kernel
> around anyway! ;-)
>
> [1]
> https://www.docker.com/docker-news-and-press/docker-released-native-mac-and-windows-apps-optimize-developer-experience

Good to know. There is still hope for mac users after all :)

>>
>>
>>
>> >
>> > On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com> wrote:
>> >
>> >
>> > On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com> wrote:
>> >>
>> >> Is there any documentation for running the Yocto build system on Mac OS
>> >> X
>> >> or macOS as Apple now calls it? I am working with the Intel Aero board.
>> >> Before I go down the rabbit hole of fixing issues like this one (and I
>> >> am
>> >> using the bash shell), I’d like to know if anyone has build it on os x
>> >> before.
>> >
>> >
>> > If you install all of the GNU tools using brew or similar and put them
>> > first
>> > on $PATH then you can get bitbake started.  Then you need to stub out
>> > the
>> > linux-specific bits in bitbake.  I've previously started on this work
>> > already
>> >
>> > (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin).
>> > The next step is figuring out how to configure OE to build and link
>> > natively
>> > on OSX using LLVM instead of GCC.
>> >
>> > However all of this is mostly academic because in Sierra (iirc) onwards
>> > there is tighter security on processes, which means that pseudo won't
>> > work
>> > even if you port it to macOS.
>> >
>> > So unless you fancy some non-trivial engineering the short version is
>> > just
>> > use something like Docker to run a Linux system on your Mac.
>> >
>> > Ross
>> > --
>> > _______________________________________________
>> > yocto mailing list
>> > yocto@yoctoproject.org
>> > https://lists.yoctoproject.org/listinfo/yocto
>> >
>> >
>> >
>> > --
>> > _______________________________________________
>> > yocto mailing list
>> > yocto@yoctoproject.org
>> > https://lists.yoctoproject.org/listinfo/yocto
>> >
>>
>>
>>
>> --
>> Maciej Borzecki
>> RnDity
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>
>



-- 
Maciej Borzecki
RnDity

Re: Building on MacOS X

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 3161 bytes --]


> On Jan 12, 2017, at 10:43 AM, Maciej Borzęcki <maciej.borzecki@rndity.com> wrote:
> 
> On Thu, Jan 12, 2017 at 7:12 PM, Andrea Galbusera <gizero@gmail.com <mailto:gizero@gmail.com>> wrote:
>> On Thu, Jan 12, 2017 at 6:55 PM, Maciej Borzęcki
>> <maciej.borzecki@rndity.com> wrote:
>>> 
>>> On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
>>> <timothy.t.orling@linux.intel.com> wrote:
>>>> You can also build using Docker containers:
>>>> https://github.com/crops/docker-win-mac-docs/wiki
>>> 
>>> IIRC docker on mac relies on docker-machine, which in turn spins up a
>>> virtualbox VM.
>> 
>> 
>> Not anymore! There's a native implementation [1] but still a linux kernel
>> around anyway! ;-)
>> 
>> [1]
>> https://www.docker.com/docker-news-an.d-press/docker-released-native-mac-and-windows-apps-optimize-developer-experience
> 
> Good to know. There is still hope for mac users after all :)

My main development machine is Mac OS X 10.12.2 (Sierra). I stay up to date with the latest Docker for Mac beta. It is a very pleasant experience.

> 
>>> 
>>> 
>>> 
>>>> 
>>>> On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com> wrote:
>>>> 
>>>> 
>>>> On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com> wrote:
>>>>> 
>>>>> Is there any documentation for running the Yocto build system on Mac OS
>>>>> X
>>>>> or macOS as Apple now calls it? I am working with the Intel Aero board.
>>>>> Before I go down the rabbit hole of fixing issues like this one (and I
>>>>> am
>>>>> using the bash shell), I’d like to know if anyone has build it on os x
>>>>> before.
>>>> 
>>>> 
>>>> If you install all of the GNU tools using brew or similar and put them
>>>> first
>>>> on $PATH then you can get bitbake started.  Then you need to stub out
>>>> the
>>>> linux-specific bits in bitbake.  I've previously started on this work
>>>> already
>>>> 
>>>> (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin).
>>>> The next step is figuring out how to configure OE to build and link
>>>> natively
>>>> on OSX using LLVM instead of GCC.
>>>> 
>>>> However all of this is mostly academic because in Sierra (iirc) onwards
>>>> there is tighter security on processes, which means that pseudo won't
>>>> work
>>>> even if you port it to macOS.
>>>> 
>>>> So unless you fancy some non-trivial engineering the short version is
>>>> just
>>>> use something like Docker to run a Linux system on your Mac.
>>>> 
>>>> Ross
>>>> --
>>>> _______________________________________________
>>>> yocto mailing list
>>>> yocto@yoctoproject.org
>>>> https://lists.yoctoproject.org/listinfo/yocto
>>>> 
>>>> 
>>>> 
>>>> --
>>>> _______________________________________________
>>>> yocto mailing list
>>>> yocto@yoctoproject.org
>>>> https://lists.yoctoproject.org/listinfo/yocto
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Maciej Borzecki
>>> RnDity
>>> --
>>> _______________________________________________
>>> yocto mailing list
>>> yocto@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/yocto
>> 
>> 
> 
> 
> 
> -- 
> Maciej Borzecki
> RnDity


[-- Attachment #2: Type: text/html, Size: 11273 bytes --]

Re: Building on MacOS X

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 6199 bytes --]


> On Jan 12, 2017, at 8:50 AM, Andrea Galbusera <gizero@gmail.com> wrote:
> 
> On Thu, Jan 12, 2017 at 5:21 PM, Belisko Marek <marek.belisko@gmail.com <mailto:marek.belisko@gmail.com>> wrote:
> On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
> <timothy.t.orling@linux.intel.com <mailto:timothy.t.orling@linux.intel.com>> wrote:
> > You can also build using Docker containers:
> > https://github.com/crops/docker-win-mac-docs/wiki <https://github.com/crops/docker-win-mac-docs/wiki>
> Well the re is other limitation about slow filesystem access from
> docker on osx. There is workaround to use nfs but it's not possible to
> use nfs for building yocto - so it's kind of chicken-egg problem ;)
> 
> I shortly tested the CROPS docker-based setup after watching some presentation at ELCE 2016 in Berlin. It basically worked but I experienced the filesystem slowness your are talking about. I ended up waiting hours to see a simple core-image-minimal build complete (even after giving more cores to docker). One more point is that slightly more complex build scenarios, i.e. building resin.os, also required tweaking docker run parameters for the build container in order to give bitbake access to features like loop devices it needed (not always easily debuggable issues indeed). Turned out I decided to stick with more canonical linux based environments for the moment.
> 

Depending on your hardware, it will take hours to build a simple image for the first time. This is no different on native Linux, if the hardware is only 1-4 cores, 4-8 GB RAM and especially if you are using a spinning harddrive. Server class systems with 72+ cores and 128+ GB of RAM will have a significantly faster build time. Almost all my YP/OE builds are now run in containers.

We will run some tests and get back to you about the real speed differences. Assuming you are running on the same hardware, we have seen builds on Windows be slightly *faster* than on native Linux. We need to collect data for our latest Mac OS X builds to address your concern.

> Anyway, the technology behind CROPS is *very* interesting to me, and I'd like to hear from people closely involved (Tim?) what the state of the art is and what we can expect to see in the near future. IIRC, the roadmap for Yocto 2.3 release was supposed to resurrect the Eclipse plugin and adopt CROPS as an alternative for running eSDK in a seamless way on different development host OSs. Beside from the images on docker hub and the github projects that didn't have high activity in the latest months, I hardly find discussions and documentation on the whole approach. Isn't this hot enough anymore or are there big issues that will prevent this technology from taking off. I often manage SDKs for Windows-minded developers and I strongly yearn to find a better approach to help them feel at home while building stuff for OE/Yocto based systems... 
> 

This work is being done by a very small team. It is no less hot than since it was first announced. However, the state of the Eclipse plugin at the initial announcement was experimental and the perception was that it was production ready. That is a risk of developing a project in the open. Both the Docker technology and the Eclipse technologies that we are using are rapidly changing.

Our current work is hosted on GitHub (https://github.com/crops/eclipse-crops <https://github.com/crops/eclipse-crops>). We had to rethink some of the approach that we demoed at ELC San Diego. We are still targeting 2.3 for the “standard SDK” approach (e.g. no “devtool” integration).

>  
> >
> > On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com <mailto:ross.burton@intel.com>> wrote:
> >
> >
> > On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com <mailto:roger@sentientblue.com>> wrote:
> >>
> >> Is there any documentation for running the Yocto build system on Mac OS X
> >> or macOS as Apple now calls it? I am working with the Intel Aero board.
> >> Before I go down the rabbit hole of fixing issues like this one (and I am
> >> using the bash shell), I’d like to know if anyone has build it on os x
> >> before.
> >
> >
> > If you install all of the GNU tools using brew or similar and put them first
> > on $PATH then you can get bitbake started.  Then you need to stub out the
> > linux-specific bits in bitbake.  I've previously started on this work
> > already
> > (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin <http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin>).
> > The next step is figuring out how to configure OE to build and link natively
> > on OSX using LLVM instead of GCC.
> >
> > However all of this is mostly academic because in Sierra (iirc) onwards
> > there is tighter security on processes, which means that pseudo won't work
> > even if you port it to macOS.
> >
> > So unless you fancy some non-trivial engineering the short version is just
> > use something like Docker to run a Linux system on your Mac.
> >
> > Ross
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> > https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>
> >
> >
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> > https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>
> >
> 
> marek
> 
> --
> as simple and primitive as possible
> -------------------------------------------------
> Marek Belisko - OPEN-NANDRA
> Freelance Developer
> 
> Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
> Tel: +421 915 052 184 <tel:%2B421%20915%20052%20184>
> skype: marekwhite
> twitter: #opennandra
> web: http://open-nandra.com <http://open-nandra.com/>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>

[-- Attachment #2: Type: text/html, Size: 10759 bytes --]

Re: Building on MacOS X

[Mark Hatle]

On 1/12/17 11:41 AM, Roger Smith wrote:
>  I have Parallels (running on El Capitan the one before Sierra)  and ubuntu 14
>  running my current build environment on a MacBook Pro, but boy is the build
> slow… I also worked at Apple for 19 years on drivers inside MacOS X/iOS, so I am
> more than motivated to have this working natively rather than inside any
> container or disk space hogging environment.  As I mentioned I am working with
> the Intel Aero compute board, so slogging though all this fat to build an image
> is a productivity killer.

So take from this.. there is desire for oe/bitbake to work natively.  Most
people don't have the skill or free time to do the work.  A number of us started
it at one time or another and made enough progress to say "ya I think it's
possible" and then ran out of time.

As far as I know pseudo and the security introduced in 10.11 that affect
preloading is likely the biggest technical problem...  everything else is just
"it's not Linux".

--Mark

> I think most of the  incompatibilities between Linux and os x (which btw is
> coming from the ios side of the fence unfortunately), can be mitigated with boot
> args or via the command line . Apple’s compiler team had to make llvm compatible
> with gcc, so I am surprised if in 2017, there are compiler issues to building
> for an x86_64 platform with llvm on the Mac . That’s the kind of bug Apple likes
> to fix promptly.. 
> 
> As I mentioned, I tried to simply source the oe-init-build-env, and got an error
> that the readlink command that yocto is using is incompatible with the bsd
> version of readlink built into os x. 
> 
> i.e when I run .
> 
> source oe-init-build-env
> 
> I get the error
> 
> readlink: illegal option -- f
> 
> Which is because on OS X readlink doesn’t specify -f
> 
> YNOPSIS
>      stat [-FLnq] [-f format| -l | -r | -s | -x] [-t timefmt] [file...]
>      readlink [-n] [file...]
> 
> I didn’t want to go through this level of change in the Yocto sources if (1)
> people don’t care to take changes or (2) it had already been done before.. I was
> curious how far down this rabbit hole people had gone before..
> 
> Roger
> 
> 
>> On Jan 12, 2017, at 8:50 AM, Andrea Galbusera <gizero@gmail.com
>> <mailto:gizero@gmail.com>> wrote:
>>
>> On Thu, Jan 12, 2017 at 5:21 PM, Belisko Marek <marek.belisko@gmail.com
>> <mailto:marek.belisko@gmail.com>> wrote:
>>
>>     On Thu, Jan 12, 2017 at 4:39 PM, Tim Orling
>>     <timothy.t.orling@linux.intel.com
>>     <mailto:timothy.t.orling@linux.intel.com>> wrote:
>>     > You can also build using Docker containers:
>>     > https://github.com/crops/docker-win-mac-docs/wiki
>>     <https://github.com/crops/docker-win-mac-docs/wiki>
>>     Well the re is other limitation about slow filesystem access from
>>     docker on osx. There is workaround to use nfs but it's not possible to
>>     use nfs for building yocto - so it's kind of chicken-egg problem ;)
>>
>>
>> I shortly tested the CROPS docker-based setup after watching some presentation
>> at ELCE 2016 in Berlin. It basically worked but I experienced the filesystem
>> slowness your are talking about. I ended up waiting hours to see a simple
>> core-image-minimal build complete (even after giving more cores to docker).
>> One more point is that slightly more complex build scenarios, i.e. building
>> resin.os, also required tweaking docker run parameters for the build container
>> in order to give bitbake access to features like loop devices it needed (not
>> always easily debuggable issues indeed). Turned out I decided to stick with
>> more canonical linux based environments for the moment.
>>
>> Anyway, the technology behind CROPS is *very* interesting to me, and I'd like
>> to hear from people closely involved (Tim?) what the state of the art is and
>> what we can expect to see in the near future. IIRC, the roadmap for Yocto 2.3
>> release was supposed to resurrect the Eclipse plugin and adopt CROPS as an
>> alternative for running eSDK in a seamless way on different development host
>> OSs. Beside from the images on docker hub and the github projects that didn't
>> have high activity in the latest months, I hardly find discussions and
>> documentation on the whole approach. Isn't this hot enough anymore or are
>> there big issues that will prevent this technology from taking off. I often
>> manage SDKs for Windows-minded developers and I strongly yearn to find a
>> better approach to help them feel at home while building stuff for OE/Yocto
>> based systems... 
>>
>>  
>>
>>     >
>>     > On Jan 12, 2017, at 7:34 AM, Burton, Ross <ross.burton@intel.com
>>     <mailto:ross.burton@intel.com>> wrote:
>>     >
>>     >
>>     > On 12 January 2017 at 15:14, Roger Smith <roger@sentientblue.com
>>     <mailto:roger@sentientblue.com>> wrote:
>>     >>
>>     >> Is there any documentation for running the Yocto build system on Mac OS X
>>     >> or macOS as Apple now calls it? I am working with the Intel Aero board.
>>     >> Before I go down the rabbit hole of fixing issues like this one (and I am
>>     >> using the bash shell), I’d like to know if anyone has build it on os x
>>     >> before.
>>     >
>>     >
>>     > If you install all of the GNU tools using brew or similar and put them first
>>     > on $PATH then you can get bitbake started.  Then you need to stub out the
>>     > linux-specific bits in bitbake.  I've previously started on this work
>>     > already
>>     >
>>     (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin
>>     <http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/darwin>).
>>     > The next step is figuring out how to configure OE to build and link natively
>>     > on OSX using LLVM instead of GCC.
>>     >
>>     > However all of this is mostly academic because in Sierra (iirc) onwards
>>     > there is tighter security on processes, which means that pseudo won't work
>>     > even if you port it to macOS.
>>     >
>>     > So unless you fancy some non-trivial engineering the short version is just
>>     > use something like Docker to run a Linux system on your Mac.
>>     >
>>     > Ross
>>     > --
>>     > _______________________________________________
>>     > yocto mailing list
>>     > yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
>>     > https://lists.yoctoproject.org/listinfo/yocto
>>     <https://lists.yoctoproject.org/listinfo/yocto>
>>     >
>>     >
>>     >
>>     > --
>>     > _______________________________________________
>>     > yocto mailing list
>>     > yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
>>     > https://lists.yoctoproject.org/listinfo/yocto
>>     <https://lists.yoctoproject.org/listinfo/yocto>
>>     >
>>
>>     marek
>>
>>     --
>>     as simple and primitive as possible
>>     -------------------------------------------------
>>     Marek Belisko - OPEN-NANDRA
>>     Freelance Developer
>>
>>     Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
>>     Tel: +421 915 052 184 <tel:%2B421%20915%20052%20184>
>>     skype: marekwhite
>>     twitter: #opennandra
>>     web: http://open-nandra.com <http://open-nandra.com/>
>>     --
>>     _______________________________________________
>>     yocto mailing list
>>     yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
>>     https://lists.yoctoproject.org/listinfo/yocto
>>     <https://lists.yoctoproject.org/listinfo/yocto>
>>
>>
>> -- 
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
>> https://lists.yoctoproject.org/listinfo/yocto
> 
> 
> 

Re: Building on MacOS X

[Clemens Lang]

Hi,


> On 12 Jan 2017, at 23:59, Mark Hatle <mark.hatle@windriver.com> wrote:
> 
> As far as I know pseudo and the security introduced in 10.11 that affect
> preloading is likely the biggest technical problem...  everything else is just
> "it's not Linux”.

With System Integrity Protection disabled, pseudo should still work as it did before, if that’s an acceptable step for you.

If it isn’t, Apple’s new limitations can also be worked around in pseudo by hooking the exec(2) and posix_spawn(2) syscalls, checking if the binary to be executed is under system integrity protection, making a copy without the SIP-bit if it is and transparently running that copy instead. That code would need to be written, though (Let me know if you want to do that, I have the code for a different project.). It’s probably only a matter of time until Apple prevents that from working, too, though, e.g. by making some standard system tools signed binaries that no longer load preloaded libraries.


HTH,
Clemens
-- 
Clemens Lang • Development Specialist
BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com
-------------------------------------------------------------------------
BMW Car IT GmbH
Geschäftsführer: Michael Würtenberger und Alexis Trolin
Sitz und Registergericht: München HRB 134810
-------------------------------------------------------------------------

Re: Building on MacOS X

[Roger Smith]

> On Jan 13, 2017, at 12:50 AM, Clemens Lang <Clemens.Lang@bmw-carit.de> wrote:
> 
> Hi,
> 
> 
>> On 12 Jan 2017, at 23:59, Mark Hatle <mark.hatle@windriver.com> wrote:
>> 
>> As far as I know pseudo and the security introduced in 10.11 that affect
>> preloading is likely the biggest technical problem...  everything else is just
>> "it's not Linux”.
> 
> With System Integrity Protection disabled, pseudo should still work as it did before, if that’s an acceptable step for you.
> 
> If it isn’t, Apple’s new limitations can also be worked around in pseudo by hooking the exec(2) and posix_spawn(2) syscalls, checking if the binary to be executed is under system integrity protection, making a copy without the SIP-bit if it is and transparently running that copy instead. That code would need to be written, though (Let me know if you want to do that, I have the code for a different project.). It’s probably only a matter of time until Apple prevents that from working, too, though, e.g. by making some standard system tools signed binaries that no longer load preloaded libraries.
> 
> 


This topic could have been clarified better, since there are multiple ways to “build” Yocto Project in  MacOS X,

Is Building Yocto project on a POSIX system, a desire for the Yocto project? It would allow support on all bsd UNIX’s including macOS


There are many teams inside Apple that use Linux systems for one reason or another. They would appreciate  building open source tools out the door without some other additional overhead.. 

Replacing a platforms POSIX version of tools with gnu versions  because your build environment didn’t like it, seems like a step backwards for these types of initiatives. 

Curious to know about long term support for POSIX systems. 

Apple spent alot of time and effor years ago to vertify OS X as POSIX compliant.

Roger

Re: Building on MacOS X

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]


> On Jan 14, 2017, at 11:45 AM, Roger Smith <roger@sentientblue.com> wrote:
> 
>> 
>> On Jan 13, 2017, at 12:50 AM, Clemens Lang <Clemens.Lang@bmw-carit.de> wrote:
>> 
>> Hi,
>> 
>> 
>>> On 12 Jan 2017, at 23:59, Mark Hatle <mark.hatle@windriver.com> wrote:
>>> 
>>> As far as I know pseudo and the security introduced in 10.11 that affect
>>> preloading is likely the biggest technical problem... everything else is just
>>> "it's not Linux”.
>> 
>> With System Integrity Protection disabled, pseudo should still work as it did before, if that’s an acceptable step for you.
>> 
>> If it isn’t, Apple’s new limitations can also be worked around in pseudo by hooking the exec(2) and posix_spawn(2) syscalls, checking if the binary to be executed is under system integrity protection, making a copy without the SIP-bit if it is and transparently running that copy instead. That code would need to be written, though (Let me know if you want to do that, I have the code for a different project.). It’s probably only a matter of time until Apple prevents that from working, too, though, e.g. by making some standard system tools signed binaries that no longer load preloaded libraries.
>> 
>> 
> 
> 
> This topic could have been clarified better, since there are multiple ways to “build” Yocto Project in  MacOS X,
> 
> Is Building Yocto project on a POSIX system, a desire for the Yocto project? It would allow support on all bsd UNIX’s including macOS
> 

Many of us use Mac laptops for development, so I think the answer is yes. As Mark mentioned, it has been an issue of time and resources. We would welcome patches :)

> 
> There are many teams inside Apple that use Linux systems for one reason or another. They would appreciate  building open source tools out the door without some other additional overhead.. 
> 
> Replacing a platforms POSIX version of tools with gnu versions  because your build environment didn’t like it, seems like a step backwards for these types of initiatives. 
> 
> Curious to know about long term support for POSIX systems. 
> 
> Apple spent alot of time and effor years ago to vertify OS X as POSIX compliant.
> 
> Roger
> 
> -- 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>

[-- Attachment #2: Type: text/html, Size: 14743 bytes --]

Re: Building on MacOS X

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On 14 January 2017 at 19:45, Roger Smith <roger@sentientblue.com> wrote:

> Is Building Yocto project on a POSIX system, a desire for the Yocto
> project? It would allow support on all bsd UNIX’s including macOS
>

Making OE itself work isn't rocket science - fix a few Linuxisms in
bitbake, port pseudo to macOS.

The hard bit is then convincing the hundred-odd recipes that are often
Linux-centric if not Linux specific to build under something that isn't
Linux.  My ross/darwin branch (from before the security changes) has a
patch to gmp as 'echo' has different semantics. unlink() has different
error codes between macOS and Linux.  There's a very long tail of
differences that will need patching and testing.

But if this is something you care about, patches welcome!

Ross

[-- Attachment #2: Type: text/html, Size: 1417 bytes --]

Re: Building on MacOS X

[Brian Avery]

[-- Attachment #1: Type: text/plain, Size: 3664 bytes --]

Hi,

A couple of comments even though I'm coming late to the discussion.

So, from what I've understood from the above, the main issue with the
docker approach to building a yocto/oe image on the mac is that it was much
slower...  Here's  some numbers that don’t quite agree with that assertion:

I ran a couple of tests, some on my linux box and some on my mac laptop.
The tests involved building core-image-minimal for a qemux86 target. All
the downloads were in place. There was no sstate, parse cache, nor was
there an existing tmp directory before the tests were run.

My linux box has a Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz , 36 cores.
My mac laptop has a Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.

The test is "$ time bitbake core-image-minimal"

Here are the results of the tests:

Linux box Ubuntu 14.04

clock : 30m24.024

user  : 229m12.324

sys    : 20m10.188

—-

Linux box Ubuntu 14.04 running inside docker 1.12.3 using crops/poky:latest

clock : 30m37.66

user  : 231m35.984

sys    : 31m20.204


——

Linux box Ubuntu 14.04 running inside docker 1.12.3 using crops/poky:latest

I set the following in my local.conf file:

BB_NUMBER_THREADS=“2”

BB_NUMBER_PARSE_THREADS=“2”

PARALLEL_MAKE=“-j 2”

AND I constrained docker to 2 cpu cores and 8 gb of ram (This is what I
have my mac laptop set to).

clock : 127m8.523

user  : 201m32.468

sys    : 19m54.052


—

Mac OSX laptop - running docker 1.12.5 2 cpus, 8gb ram

I set the following in my local.conf file:

BB_NUMBER_THREADS=“2”

BB_NUMBER_PARSE_THREADS=“2”

PARALLEL_MAKE=“-j 2”


clock : 99m31.190

user  : 137m40.400

sys    : 18m38.650


——


So, from the above, it looks like my mac is actually faster when it has the
same number of cores and memory.  I haven’t seen any particular slowdown
with running docker linux programs on the mac other than those caused by
the difference in horsepower between my build server and my laptop.


Thanks,

Brian Avery

an Intel employee


p.s. Andrea, would you mind replying with how you changed the docker run to
make loopback work? I’d like to add it to the docs. Also, if you could
point me at what layers you used to make resin, I’d like to give that a try
as well.  Right now, we are providing a bare bones environment but I’d be
happy to write up a howto for inheriting from our images to customize your
own for special purposes (like doing builds that require loopback mounts,
for instance).







On Mon, Jan 16, 2017 at 3:19 AM, Burton, Ross <ross.burton@intel.com> wrote:

>
> On 14 January 2017 at 19:45, Roger Smith <roger@sentientblue.com> wrote:
>
>> Is Building Yocto project on a POSIX system, a desire for the Yocto
>> project? It would allow support on all bsd UNIX’s including macOS
>>
>
> Making OE itself work isn't rocket science - fix a few Linuxisms in
> bitbake, port pseudo to macOS.
>
> The hard bit is then convincing the hundred-odd recipes that are often
> Linux-centric if not Linux specific to build under something that isn't
> Linux.  My ross/darwin branch (from before the security changes) has a
> patch to gmp as 'echo' has different semantics. unlink() has different
> error codes between macOS and Linux.  There's a very long tail of
> differences that will need patching and testing.
>
> But if this is something you care about, patches welcome!
>
> Ross
>
>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
>

[-- Attachment #2: Type: text/html, Size: 7796 bytes --]