[Buildroot] manipulating raw disk images as non-root user

[Buildroot] manipulating raw disk images as non-root user

[Dallas Clement]

Hi All,

I am trying to figure out a way to produce a uSD card image file during my
build process and do it as a non-root user.  I am hoping to do this from a
post image script.

I can create the image file with dd and partition it with parted as a
non-root user.  However, I have not figured out how to format, and populate
the image file.  Tools such as kpartx and losetup seem ideal for this
purpose except that they must be run as root.

Does anyone know of a way to do this using qemu or fuse or related tools?

Thanks,

Dallas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140802/21044f09/attachment.html>

[Buildroot] manipulating raw disk images as non-root user

[Thomas Petazzoni]

Dear Dallas Clement,

On Sat, 2 Aug 2014 11:45:58 -0500, Dallas Clement wrote:

> I am trying to figure out a way to produce a uSD card image file during my
> build process and do it as a non-root user.  I am hoping to do this from a
> post image script.
> 
> I can create the image file with dd and partition it with parted as a
> non-root user.  However, I have not figured out how to format, and populate
> the image file.  Tools such as kpartx and losetup seem ideal for this
> purpose except that they must be run as root.
> 
> Does anyone know of a way to do this using qemu or fuse or related tools?

You could have a look at genimage that is available as a package in
Buildroot. Yann E. Morin also had developed some patches for Buildroot
to make that possible, but they were not merged due, I believe, to a
too high complexity. I'm sure he will be available to give you pointers
to these patches.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] manipulating raw disk images as non-root user

[Waldemar Brodkorb]

Hi Dallas,
Dallas Clement wrote,

> Hi All,
> 
> I am trying to figure out a way to produce a uSD card image file during my
> build process and do it as a non-root user. ?I am hoping to do this from a post
> image script.
> 
> I can create the image file with dd and partition it with parted as a non-root
> user. ?However, I have not figured out how to format, and populate the image
> file. ?Tools such as kpartx and losetup seem ideal for this purpose except that
> they must be run as root.
> 
> Does anyone know of a way to do this using qemu or fuse or related tools?

You could use genext2fs to generate a filesystem as non-root.

Installing a bootloader as non-root would be tricky, but is 
possible, too.
 
best regards
 Waldemar

[Buildroot] manipulating raw disk images as non-root user

[Dallas Clement]

On Mon, Aug 4, 2014 at 2:05 AM, Waldemar Brodkorb <wbx@openadk.org> wrote:

> Hi Dallas,
> Dallas Clement wrote,
>
> > Hi All,
> >
> > I am trying to figure out a way to produce a uSD card image file during
> my
> > build process and do it as a non-root user.  I am hoping to do this from
> a post
> > image script.
> >
> > I can create the image file with dd and partition it with parted as a
> non-root
> > user.  However, I have not figured out how to format, and populate the
> image
> > file.  Tools such as kpartx and losetup seem ideal for this purpose
> except that
> > they must be run as root.
> >
> > Does anyone know of a way to do this using qemu or fuse or related tools?
>
> You could use genext2fs to generate a filesystem as non-root.
>
> Installing a bootloader as non-root would be tricky, but is
> possible, too.
>
> best regards
>  Waldemar
>

Wow!  This is perfect.  I searched all over for a tool like this.  Couldn't
find anything.  Thanks Waldemar!  Do you know of a similar tool to make /
populate fat partitions.  I need to work with both fat and ext2.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140804/42d6a3fe/attachment.html>

[Buildroot] manipulating raw disk images as non-root user

[Thomas Petazzoni]

Dear Dallas Clement,

On Mon, 4 Aug 2014 12:56:40 -0500, Dallas Clement wrote:

> Wow!  This is perfect.  I searched all over for a tool like this.  Couldn't
> find anything.  Thanks Waldemar!  Do you know of a similar tool to make /
> populate fat partitions.  I need to work with both fat and ext2.

Huh? genext2fs has been part of Buildroot since years, and is directly
usable by asking Buildroot to generate an ext2/3/4 image in "Filesystem
images".

Regarding FAT partition, I guess most people use the mtools?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] manipulating raw disk images as non-root user

[Austin S Hemmelgarn]

On 2014-08-04 13:56, Dallas Clement wrote:
> 
> On Mon, Aug 4, 2014 at 2:05 AM, Waldemar Brodkorb <wbx@openadk.org
> <mailto:wbx@openadk.org>> wrote:
> 
>     Hi Dallas,
>     Dallas Clement wrote,
> 
>     > Hi All,
>     >
>     > I am trying to figure out a way to produce a uSD card image file
>     during my
>     > build process and do it as a non-root user.  I am hoping to do
>     this from a post
>     > image script.
>     >
>     > I can create the image file with dd and partition it with parted
>     as a non-root
>     > user.  However, I have not figured out how to format, and populate
>     the image
>     > file.  Tools such as kpartx and losetup seem ideal for this
>     purpose except that
>     > they must be run as root.
>     >
>     > Does anyone know of a way to do this using qemu or fuse or related
>     tools?
> 
>     You could use genext2fs to generate a filesystem as non-root.
> 
>     Installing a bootloader as non-root would be tricky, but is
>     possible, too.
> 
>     best regards
>      Waldemar
> 
> 
> Wow!  This is perfect.  I searched all over for a tool like this. 
> Couldn't find anything.  Thanks Waldemar!  Do you know of a similar tool
> to make / populate fat partitions.  I need to work with both fat and ext2.

you can use any mkfs command as a non-root user, and for FAT you should
be able to find a fuse driver for manipulating it (or there is always
mtools).


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2967 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140804/63ab5a1e/attachment.p7s>

[Buildroot] manipulating raw disk images as non-root user

[Dallas Clement]

On Mon, Aug 4, 2014 at 1:10 PM, Austin S Hemmelgarn <ahferroin7@gmail.com>
wrote:

> On 2014-08-04 13:56, Dallas Clement wrote:
> >
> > On Mon, Aug 4, 2014 at 2:05 AM, Waldemar Brodkorb <wbx@openadk.org
> > <mailto:wbx@openadk.org>> wrote:
> >
> >     Hi Dallas,
> >     Dallas Clement wrote,
> >
> >     > Hi All,
> >     >
> >     > I am trying to figure out a way to produce a uSD card image file
> >     during my
> >     > build process and do it as a non-root user.  I am hoping to do
> >     this from a post
> >     > image script.
> >     >
> >     > I can create the image file with dd and partition it with parted
> >     as a non-root
> >     > user.  However, I have not figured out how to format, and populate
> >     the image
> >     > file.  Tools such as kpartx and losetup seem ideal for this
> >     purpose except that
> >     > they must be run as root.
> >     >
> >     > Does anyone know of a way to do this using qemu or fuse or related
> >     tools?
> >
> >     You could use genext2fs to generate a filesystem as non-root.
> >
> >     Installing a bootloader as non-root would be tricky, but is
> >     possible, too.
> >
> >     best regards
> >      Waldemar
> >
> >
> > Wow!  This is perfect.  I searched all over for a tool like this.
> > Couldn't find anything.  Thanks Waldemar!  Do you know of a similar tool
> > to make / populate fat partitions.  I need to work with both fat and
> ext2.
>
> you can use any mkfs command as a non-root user, and for FAT you should
> be able to find a fuse driver for manipulating it (or there is always
> mtools).
>
>
>
> Huh? genext2fs has been part of Buildroot since years, and is directly
> usable by asking Buildroot to generate an ext2/3/4 image in "Filesystem
> images".

Ay yi yi.  I guess it never caught my eye.  How shameful.  :/

Regarding mtools, will it allow me to partition an image file, format a
partition as FAT, and copy files to it as non-root?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140804/517fbef5/attachment.html>

[Buildroot] manipulating raw disk images as non-root user

[Yann E. MORIN]

Dallas, All,

On 2014-08-02 11:45 -0500, Dallas Clement spake thusly:
> I am trying to figure out a way to produce a uSD card image file during my
> build process and do it as a non-root user.  I am hoping to do this from a
> post image script.
> 
> I can create the image file with dd and partition it with parted as a
> non-root user.  However, I have not figured out how to format, and populate
> the image file.  Tools such as kpartx and losetup seem ideal for this
> purpose except that they must be run as root.
> 
> Does anyone know of a way to do this using qemu or fuse or related tools?

Well, as already pointed out, we do have a few host tools just for that
kind of usage, in the "Host utilities" sub-menu.

Also, if you want to have a look, I make use of those tools in my
Buildroot.config project:
    http://ymorin.is-a-geek.org/git/buildroot.config/

You may want to have a look at:
    http://ymorin.is-a-geek.org/git/buildroot.config/tree/scripts/genimages.d
and the subfolders for how to use those tools.

Note that all the scripts there in are part of a bigger infra. Feel free
to poke around the project tree.

(Warning: no doc, bleeding edge, unstable, no support... ;-) )

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] manipulating raw disk images as non-root user

[Dallas Clement]

> Also, if you want to have a look, I make use of those tools in my
> Buildroot.config project:
>    http://ymorin.is-a-geek.org/git/buildroot.config/

> You may want to have a look at:
>
http://ymorin.is-a-geek.org/git/buildroot.config/tree/scripts/genimages.d
> and the subfolders for how to use those tools.

Thanks Yann.  This is good stuff!  The particular challenge I have is that
I need to create and populate one vfat partition as well as one ext[2,3,4]
partition, both within the same disk image file.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140805/90e1bca9/attachment.html>

[Buildroot] manipulating raw disk images as non-root user

[Yann E. MORIN]

Dallas, All,

On 2014-08-05 16:26 -0500, Dallas Clement spake thusly:
> > Also, if you want to have a look, I make use of those tools in my
> > Buildroot.config project:
> >    http://ymorin.is-a-geek.org/git/buildroot.config/
> 
> > You may want to have a look at:
> >
> http://ymorin.is-a-geek.org/git/buildroot.config/tree/scripts/genimages.d
> > and the subfolders for how to use those tools.
> 
> Thanks Yann.  This is good stuff!  The particular challenge I have is that
> I need to create and populate one vfat partition as well as one ext[2,3,4]
> partition, both within the same disk image file.

The easiest is to generate two different images, and then aggregate the
two to form the final image.

If you first create the final image, then create the two filesystems in
it, you have to provide the offsets to each filesystem when you want to
adress one or the other.

For ext2/3/4, I suggest you use genext2fs, which takes a directory and
its content (recursively), to create am ext2 filesystem, which you can
then convert to ext3 or ext4. We already have a script for that in
Buildroot, in fs/ext2/genext2fs.sh, which you can use as a source of
inspiration.

For vfat, I'm not aware of a tool similar to genext2fs, so you'd have to
create an empty file, then format it with nkfs.vfat, then use the mtools
to copy files into that filesystem image.

Finaly, create a partition table, and append the vfat and ext2
filesystem, to get the final image.

That's not trivial, mind you. And that's the reason I've been working on
this in my little buildroot.config project.

I tried to include this into Buildroot itself, but it is far from being
obvious (to say the least), for a lot of reasons I won't enumerate here.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'