* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
@ 2017-08-29 11:27 Jan Schmidt
2017-08-29 21:24 ` Yann E. MORIN
0 siblings, 1 reply; 9+ messages in thread
From: Jan Schmidt @ 2017-08-29 11:27 UTC (permalink / raw)
To: buildroot
Hi,
I was looking for a way to customize the root file system in the
way, that all except /var is read only. And /var is at a separate
partition which will be enlarged at the first start to the
maximum size of the storage media. (I'm using RPi 3)
I'm using BR 2017.02.1. The above patch seems not to be included
yet. And I didn't fount any discussion about accepting the patch.
Is there a other way to solve my issue?
Thanks and regards,
Jan Schmidt
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
2017-08-29 11:27 [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images Jan Schmidt
@ 2017-08-29 21:24 ` Yann E. MORIN
0 siblings, 0 replies; 9+ messages in thread
From: Yann E. MORIN @ 2017-08-29 21:24 UTC (permalink / raw)
To: buildroot
Jan, All,
On 2017-08-29 13:27 +0200, Jan Schmidt spake thusly:
> I was looking for a way to customize the root file system in the
> way, that all except /var is read only. And /var is at a separate
> partition which will be enlarged at the first start to the
> maximum size of the storage media. (I'm using RPi 3)
>
> I'm using BR 2017.02.1. The above patch seems not to be included
> yet. And I didn't fount any discussion about accepting the patch.
> Is there a other way to solve my issue?
At first, I was very puzzled about what patch this was refering to. ANd
it truns out it was something I wrote almost 4 years ago... Damn, time
flies...
Anyway, back on topic: no, that was not accepted at all, see the
discussion in the thread:
https://patchwork.ozlabs.org/patch/306631/
So, the best option for you is that you do something like:
- only generate a tarball out of Buildroot,
- provide a post-image script that:
- extract the tarball into ${TMP_ROOT_DIR}
- generate an ext2 (or other writable FS) out of ${TMP_ROOT_DIR}/var
- remove ${TMP_ROOT_DIR}/var/*
- tweak ${TMP_ROOT_DIR}/etc/fstab to add the mountpoint for /var
- create the readonly root fs with out of ${TMP_ROOT_DIR}/
Note that the post-image script should run under fakeroot, and Buildroot
does not guarantee that, so your script would probalby need to start
with something like:
if [ $(id -u) -ne 0 ]; then
fakeroot "${0}" "${@}"
exit ${?}
fi
which would ensure that the script runs under fakeroot.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
2014-04-09 17:09 ` Yann E. MORIN
@ 2014-04-10 13:53 ` Andreas Naumann
0 siblings, 0 replies; 9+ messages in thread
From: Andreas Naumann @ 2014-04-10 13:53 UTC (permalink / raw)
To: buildroot
Hi Yann,
thanks a lot for the explanations!
Am 09.04.2014 19:09, schrieb Yann E. MORIN:
>> into master yet. What is the current state?
>
> This has been somewhat refused: we do not want to add new tools in
> Buildroot, and this 'genimages' stuff is seen as a new tool, rather than
> as infrastructure.
>
> However, it has been suggested that we:
>
> - transform this genimages into a separate project which provides a
> new filesystem generator, and which Buildroot could use like it is
> currently using the other filesystem generators;
>
> - look at Pengutronix' genimage (without a trailing 's') to see if it
> would fit our needs.
I had a very quick look at your buildroot.config as well as Pengutronix
genimage. You seem to do much more than what genimage does, but actually
creating a bunch of ubi images is sufficient for me. Well I have to try
it out, needs usually appear when doing things. Right now I dont find
the time but hopefully in a couple of weeks I should be able to post my
experience.
regards
Andi
>
> I haven't had time to properly look at Pengutronix' genimage, but last I
> looked, it seemd a bit different than my genimages. I am not too fond of
> genimage, but that's mostly a personal feeling, and is not really
> founded on technical merits. I prefer my genimages implementation, which
> I know how to deal with.
>
> As for my genimages: I am still working on it, and it so far fits all my
> personal needs. I still have some feature that are missing for now, that
> are really needed before it can be widely deployed:
>
> - mixed filesystem hierarchies split between an initramfs and partitions
> on a real device;
>
> - partitions to be mounted that are not to be written (eg. when you
> upgrade, you do not want to overwrite a config partition, but you
> still want to have an fstab entry for it).
>
> That's being hamerred out, but it takes a bit of time...
>
>>> The basis behind this is a .ini-like description of the layout of the
>>> final target storage:
>>> - the list of device(s)
>>> - per-device, the list of partition(s)
>>> - per-partition, the content
>>>
>>> It is possible to create MBR- or GPT-based partitoining schemes. Adding
>>> new ones should be relatively easy (but would need adequate host tools).
>>>
>>> For now, the only content possible for partitions is a filesystem. It
>>> should be pretty easy to add new types (eg. un-formated, or raw blob).
>>>
>>> Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding
>>> more will be relatively easy, provided we have the necessary host
>>> packages to deal with those filesystems.
>>
>> However, I'not sure if it actually is what I am looking for. Which is
>> dividing the target content into multiple partitions, which are then mounted
>> into the rootfs (like /data, /etc/somespecialconfig, /var/log).?
>
> Yes, that's exactly what genimages is supposed to cover: a filesystem
> hierarchy made of more than the / mountpoint.
>
>> Are there other ways to do that?
>
> Depends what you mean. I am using a 'wrapper' project arounf Buildroot,
> which genimages originates from. In this project, I'm using genimages as
> a post-image script that just extract the rootfs.tar and works from
> there to generate the multiple filesystem images for the different
> partitions.
>
> But it is not really usable outside of my project, since it is tightly
> coupled with the post-build script.
>
> Basically, if all you have are filesystem on reald decives (flash,
> HDD...), then genimages could world stand-alone.
>
> But if your setup involves an initramfs for the rootfs, plus one or more
> partitions on a real device, you have to fill entries for those
> partitions in etc/fstab, but doing so in post-image is too late, because
> the etc/fstab is in the initramfs, which is in the kernel image, which
> is assembled during the image phase. So we can't fill-in etc/fstab from
> a post-image script in this case. [*]
>
> However, it is always possible to fill-in etc/fstab from a post-build
> (aka pre-image) script, whether your rootfs is on a real device or an
> initramfs. And that's what my post-build script does (among many other
> things).
>
> So my genimages is tightly coupled to my rootfs-fixup post-build script,
> and I currently do not see a clean way to provide genimages as just
> another filesystem generator. If we wanted to truly have genimages in
> Buildroot, we'd need more than hooking after the rootfs.tar generator.
>
>
> [*] Well, it could be possible to call back to Buildroot to regenerate
> the kernel image with the initramfs once we tweaked etc/fstab, but
> that's pretty ugly, and prone to errors.)
>
>
> If you want to have a look, here's my "buildroot.config' project:
> http://ymorin.is-a-geek.org/git/buildroot.config/
>
> There's currently pretty much no documentation, besides the one I wrote
> for inclusion in Buildroot (available in this thread). That's something
> I should really write... The README is a bit out-dated, so not completely
> accurate, but the idea is there.
>
> I have a tutorial on how to use it to build an XBMC for the RPi:
> http://ymorin.is-a-geek.org/projects/rpi-xbmc
>
> Any suggestion/idea on how to progress on Buildroot is welcome, of
> course! ;-)
>
> Regards,
> Yann E. MORIN.
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
2014-04-09 13:28 ` Andreas Naumann
@ 2014-04-09 17:09 ` Yann E. MORIN
2014-04-10 13:53 ` Andreas Naumann
0 siblings, 1 reply; 9+ messages in thread
From: Yann E. MORIN @ 2014-04-09 17:09 UTC (permalink / raw)
To: buildroot
Andreas, All,
On 2014-04-09 15:28 +0200, Andreas Naumann spake thusly:
> Hi Yann,
>
> >From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> >Contrary to the existing fs/ schemes, which each generate only a single
> >filesystem image for the root filesystem, this new scheme allows the
> >user to generate more complex images.
>
> if found this an interesting idea, but apparently it has not found its way
> into master yet. What is the current state?
This has been somewhat refused: we do not want to add new tools in
Buildroot, and this 'genimages' stuff is seen as a new tool, rather than
as infrastructure.
However, it has been suggested that we:
- transform this genimages into a separate project which provides a
new filesystem generator, and which Buildroot could use like it is
currently using the other filesystem generators;
- look at Pengutronix' genimage (without a trailing 's') to see if it
would fit our needs.
I haven't had time to properly look at Pengutronix' genimage, but last I
looked, it seemd a bit different than my genimages. I am not too fond of
genimage, but that's mostly a personal feeling, and is not really
founded on technical merits. I prefer my genimages implementation, which
I know how to deal with.
As for my genimages: I am still working on it, and it so far fits all my
personal needs. I still have some feature that are missing for now, that
are really needed before it can be widely deployed:
- mixed filesystem hierarchies split between an initramfs and partitions
on a real device;
- partitions to be mounted that are not to be written (eg. when you
upgrade, you do not want to overwrite a config partition, but you
still want to have an fstab entry for it).
That's being hamerred out, but it takes a bit of time...
> >The basis behind this is a .ini-like description of the layout of the
> >final target storage:
> > - the list of device(s)
> > - per-device, the list of partition(s)
> > - per-partition, the content
> >
> >It is possible to create MBR- or GPT-based partitoining schemes. Adding
> >new ones should be relatively easy (but would need adequate host tools).
> >
> >For now, the only content possible for partitions is a filesystem. It
> >should be pretty easy to add new types (eg. un-formated, or raw blob).
> >
> >Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding
> >more will be relatively easy, provided we have the necessary host
> >packages to deal with those filesystems.
>
> However, I'not sure if it actually is what I am looking for. Which is
> dividing the target content into multiple partitions, which are then mounted
> into the rootfs (like /data, /etc/somespecialconfig, /var/log).?
Yes, that's exactly what genimages is supposed to cover: a filesystem
hierarchy made of more than the / mountpoint.
> Are there other ways to do that?
Depends what you mean. I am using a 'wrapper' project arounf Buildroot,
which genimages originates from. In this project, I'm using genimages as
a post-image script that just extract the rootfs.tar and works from
there to generate the multiple filesystem images for the different
partitions.
But it is not really usable outside of my project, since it is tightly
coupled with the post-build script.
Basically, if all you have are filesystem on reald decives (flash,
HDD...), then genimages could world stand-alone.
But if your setup involves an initramfs for the rootfs, plus one or more
partitions on a real device, you have to fill entries for those
partitions in etc/fstab, but doing so in post-image is too late, because
the etc/fstab is in the initramfs, which is in the kernel image, which
is assembled during the image phase. So we can't fill-in etc/fstab from
a post-image script in this case. [*]
However, it is always possible to fill-in etc/fstab from a post-build
(aka pre-image) script, whether your rootfs is on a real device or an
initramfs. And that's what my post-build script does (among many other
things).
So my genimages is tightly coupled to my rootfs-fixup post-build script,
and I currently do not see a clean way to provide genimages as just
another filesystem generator. If we wanted to truly have genimages in
Buildroot, we'd need more than hooking after the rootfs.tar generator.
[*] Well, it could be possible to call back to Buildroot to regenerate
the kernel image with the initramfs once we tweaked etc/fstab, but
that's pretty ugly, and prone to errors.)
If you want to have a look, here's my "buildroot.config' project:
http://ymorin.is-a-geek.org/git/buildroot.config/
There's currently pretty much no documentation, besides the one I wrote
for inclusion in Buildroot (available in this thread). That's something
I should really write... The README is a bit out-dated, so not completely
accurate, but the idea is there.
I have a tutorial on how to use it to build an XBMC for the RPi:
http://ymorin.is-a-geek.org/projects/rpi-xbmc
Any suggestion/idea on how to progress on Buildroot is welcome, of
course! ;-)
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
2014-01-03 17:19 ` [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images Yann E. MORIN
2014-01-04 16:38 ` Maxime Hadjinlian
@ 2014-04-09 13:28 ` Andreas Naumann
2014-04-09 17:09 ` Yann E. MORIN
1 sibling, 1 reply; 9+ messages in thread
From: Andreas Naumann @ 2014-04-09 13:28 UTC (permalink / raw)
To: buildroot
Hi Yann,
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Contrary to the existing fs/ schemes, which each generate only a single
> filesystem image for the root filesystem, this new scheme allows the
> user to generate more complex images.
if found this an interesting idea, but apparently it has not found its
way into master yet. What is the current state?
>
> The basis behind this is a .ini-like description of the layout of the
> final target storage:
> - the list of device(s)
> - per-device, the list of partition(s)
> - per-partition, the content
>
> It is possible to create MBR- or GPT-based partitoining schemes. Adding
> new ones should be relatively easy (but would need adequate host tools).
>
> For now, the only content possible for partitions is a filesystem. It
> should be pretty easy to add new types (eg. un-formated, or raw blob).
>
> Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding
> more will be relatively easy, provided we have the necessary host
> packages to deal with those filesystems.
However, I'not sure if it actually is what I am looking for. Which is
dividing the target content into multiple partitions, which are then
mounted into the rootfs (like /data, /etc/somespecialconfig, /var/log).?
Are there other ways to do that?
cheers,
Andi
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
2014-01-04 17:43 ` Yann E. MORIN
@ 2014-01-04 17:52 ` Maxime Hadjinlian
0 siblings, 0 replies; 9+ messages in thread
From: Maxime Hadjinlian @ 2014-01-04 17:52 UTC (permalink / raw)
To: buildroot
On Sat, Jan 4, 2014 at 6:43 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Maxime, All,
>
> Thanks for the review! :-)
>
> On 2014-01-04 17:38 +0100, Maxime Hadjinlian spake thusly:
>> When doing scripting in Buildroot, do we accept the uses of various
>> bash-ism or do we want to stick to POSIX shell ?
>
> bash is already a hard-dependency of Buildroot.
>
>> On Fri, Jan 3, 2014 at 6:19 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> [--SNIP--]
>> > diff --git a/docs/manual/customize-filesystems.txt b/docs/manual/customize-filesystems.txt
>> > new file mode 100644
>> > index 0000000..fd65c97
>> > --- /dev/null
>> > +++ b/docs/manual/customize-filesystems.txt
>> > @@ -0,0 +1,36 @@
> [--SNIP--]
>> How about comments ? Are they supported ?
>> In a next step, it would be really nice to have support for "human"
>> input for the different size, like 1G/M/K.
>
> Yes, comments are supported; I've now added that to the manual.
>
> I've also added that values can be expresed as shell arith,metic
> expansion, such as $((16*1048576)) (aka 16MiB).
>
> Not that G/M/k are SI-prefixes that respectively mean 10^9, 10^6 and
> 10^3. The binary prefixes are Gi/Mi/ki, respectively 2^30, 2^20 and
> 2^10.
>
> If/when we add support for this kind of values, we'll need to add both
> decimal and binary prefixes.
You are absolutely right. Also, it is sad, almost no one uses the
right units :(.
>
>> > diff --git a/fs/custom/boot/gpt b/fs/custom/boot/gpt
>> > new file mode 100644
>> > index 0000000..f978524
>> > --- /dev/null
>> > +++ b/fs/custom/boot/gpt
>> > @@ -0,0 +1,129 @@
> [--SNIP--]
>> > +# Simg = 512 + 2*(Sgpt) + ?( aligned(Spart,512) )
>> Don't we want only ASCII char there ?
>
> I don't know.
>
> I find it convenient not to limit ourselves to ASCII.
I remember someone on this mailing saying something about it, don't
remember exactly what was the context.
>
> [--SNIP--]
>> > +# Sicne 4k-large sectors are not really explained on Wikipedia, we can
>> Small typo here.
>
> Thanks, fixed.
>
> [--SNIP--]
>> > + dd if=/dev/zero of="${img}" \
>> > + bs=1 seek=${begin} count=0 \
>> > + conv=sparse 2>/dev/null
>> Why the redirect to /dev/null ? Don't we want to see if there was an
>> error ? I know it's unlikely to happen but you never know.
>> I have noticed you did several times so maybe there is a reason I am
>> not seeing here.
>
> dd will always print something to stderr, even if successful. We do not
> want it to pollute the output, since we're using nice traces.
>
> [--SNIP--]
>> > + ret=${?}
>> > + [ ${ret} -eq 0 ] || exit ${ret}
>> Why not doing: [ ${ret} -ne 0 ] && exit ${ret} ?
>
> Same answer as to your comment in squashfs, since we're using set -e.
>
> But here, it's a bit complex. POSIX does not specify whether a sub-shell
> that exits with a return code !0 should be caught by set -e. And bash-4.0
> does not catch it, while bash-4.2 does catch it. Both are POSIX complianti
> in this respect, since POSIX does not state what should be done in this
> case.
>
> So, we have to explicitly store the return code, and test it.
>
> But if ret==0, your test would fail, and the complete shell compound
> command " [ ... ] && ... " would also be false, and we would create an
> error that the shell would catch because of set -e, when there was no
> error to begin with.
>
> Thank you for the comments! :-)
Thanks for the explanations !
>
> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
> | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
> '------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
2014-01-04 16:38 ` Maxime Hadjinlian
@ 2014-01-04 17:43 ` Yann E. MORIN
2014-01-04 17:52 ` Maxime Hadjinlian
0 siblings, 1 reply; 9+ messages in thread
From: Yann E. MORIN @ 2014-01-04 17:43 UTC (permalink / raw)
To: buildroot
Maxime, All,
Thanks for the review! :-)
On 2014-01-04 17:38 +0100, Maxime Hadjinlian spake thusly:
> When doing scripting in Buildroot, do we accept the uses of various
> bash-ism or do we want to stick to POSIX shell ?
bash is already a hard-dependency of Buildroot.
> On Fri, Jan 3, 2014 at 6:19 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
[--SNIP--]
> > diff --git a/docs/manual/customize-filesystems.txt b/docs/manual/customize-filesystems.txt
> > new file mode 100644
> > index 0000000..fd65c97
> > --- /dev/null
> > +++ b/docs/manual/customize-filesystems.txt
> > @@ -0,0 +1,36 @@
[--SNIP--]
> How about comments ? Are they supported ?
> In a next step, it would be really nice to have support for "human"
> input for the different size, like 1G/M/K.
Yes, comments are supported; I've now added that to the manual.
I've also added that values can be expresed as shell arith,metic
expansion, such as $((16*1048576)) (aka 16MiB).
Not that G/M/k are SI-prefixes that respectively mean 10^9, 10^6 and
10^3. The binary prefixes are Gi/Mi/ki, respectively 2^30, 2^20 and
2^10.
If/when we add support for this kind of values, we'll need to add both
decimal and binary prefixes.
> > diff --git a/fs/custom/boot/gpt b/fs/custom/boot/gpt
> > new file mode 100644
> > index 0000000..f978524
> > --- /dev/null
> > +++ b/fs/custom/boot/gpt
> > @@ -0,0 +1,129 @@
[--SNIP--]
> > +# Simg = 512 + 2*(Sgpt) + ?( aligned(Spart,512) )
> Don't we want only ASCII char there ?
I don't know.
I find it convenient not to limit ourselves to ASCII.
[--SNIP--]
> > +# Sicne 4k-large sectors are not really explained on Wikipedia, we can
> Small typo here.
Thanks, fixed.
[--SNIP--]
> > + dd if=/dev/zero of="${img}" \
> > + bs=1 seek=${begin} count=0 \
> > + conv=sparse 2>/dev/null
> Why the redirect to /dev/null ? Don't we want to see if there was an
> error ? I know it's unlikely to happen but you never know.
> I have noticed you did several times so maybe there is a reason I am
> not seeing here.
dd will always print something to stderr, even if successful. We do not
want it to pollute the output, since we're using nice traces.
[--SNIP--]
> > + ret=${?}
> > + [ ${ret} -eq 0 ] || exit ${ret}
> Why not doing: [ ${ret} -ne 0 ] && exit ${ret} ?
Same answer as to your comment in squashfs, since we're using set -e.
But here, it's a bit complex. POSIX does not specify whether a sub-shell
that exits with a return code !0 should be caught by set -e. And bash-4.0
does not catch it, while bash-4.2 does catch it. Both are POSIX complianti
in this respect, since POSIX does not state what should be done in this
case.
So, we have to explicitly store the return code, and test it.
But if ret==0, your test would fail, and the complete shell compound
command " [ ... ] && ... " would also be false, and we would create an
error that the shell would catch because of set -e, when there was no
error to begin with.
Thank you for the comments! :-)
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
2014-01-03 17:19 ` [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images Yann E. MORIN
@ 2014-01-04 16:38 ` Maxime Hadjinlian
2014-01-04 17:43 ` Yann E. MORIN
2014-04-09 13:28 ` Andreas Naumann
1 sibling, 1 reply; 9+ messages in thread
From: Maxime Hadjinlian @ 2014-01-04 16:38 UTC (permalink / raw)
To: buildroot
Hi Yann,
When doing scripting in Buildroot, do we accept the uses of various
bash-ism or do we want to stick to POSIX shell ?
On Fri, Jan 3, 2014 at 6:19 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Contrary to the existing fs/ schemes, which each generate only a single
> filesystem image for the root filesystem, this new scheme allows the
> user to generate more complex images.
>
> The basis behind this is a .ini-like description of the layout of the
> final target storage:
> - the list of device(s)
> - per-device, the list of partition(s)
> - per-partition, the content
>
> It is possible to create MBR- or GPT-based partitoining schemes. Adding
> new ones should be relatively easy (but would need adequate host tools).
>
> For now, the only content possible for partitions is a filesystem. It
> should be pretty easy to add new types (eg. un-formated, or raw blob).
>
> Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding
> more will be relatively easy, provided we have the necessary host
> packages to deal with those filesystems.
>
> The existing Buildroot filesystem generators are re-used as much as
> possible when it makes sense; when it does not (eg. for vfat), a specific
> generator is used.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Ryan Barnett <rjbarnet@rockwellcollins.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: "J?r?me Oufella" <jerome.oufella@savoirfairelinux.com>
> ---
> docs/manual/appendix.txt | 1 +
> docs/manual/customize-filesystems.txt | 36 ++++
> docs/manual/customize.txt | 2 +
> docs/manual/partition-layout.txt | 298 +++++++++++++++++++++++++++++
> fs/Config.in | 1 +
> fs/custom/Config.in | 16 ++
> fs/custom/boot/gpt | 129 +++++++++++++
> fs/custom/boot/mbr | 64 +++++++
> fs/custom/boot/pre-post | 14 ++
> fs/custom/custom.mk | 25 +++
> fs/custom/fs/ext | 26 +++
> fs/custom/fs/pre-post | 72 +++++++
> fs/custom/fs/vfat | 21 +++
> fs/custom/genimages | 342 ++++++++++++++++++++++++++++++++++
> 14 files changed, 1047 insertions(+)
> create mode 100644 docs/manual/customize-filesystems.txt
> create mode 100644 docs/manual/partition-layout.txt
> create mode 100644 fs/custom/Config.in
> create mode 100644 fs/custom/boot/gpt
> create mode 100644 fs/custom/boot/mbr
> create mode 100644 fs/custom/boot/pre-post
> create mode 100644 fs/custom/custom.mk
> create mode 100644 fs/custom/fs/ext
> create mode 100644 fs/custom/fs/pre-post
> create mode 100644 fs/custom/fs/vfat
> create mode 100755 fs/custom/genimages
>
> diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
> index 74ee8fd..53f4205 100644
> --- a/docs/manual/appendix.txt
> +++ b/docs/manual/appendix.txt
> @@ -6,6 +6,7 @@ Appendix
>
> include::makedev-syntax.txt[]
> include::makeusers-syntax.txt[]
> +include::partition-layout.txt[]
>
>
> // Automatically generated lists:
> diff --git a/docs/manual/customize-filesystems.txt b/docs/manual/customize-filesystems.txt
> new file mode 100644
> index 0000000..fd65c97
> --- /dev/null
> +++ b/docs/manual/customize-filesystems.txt
> @@ -0,0 +1,36 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +[[filesystem-custom]]
> +Customizing the generated filesystem images
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> ++Buildroot+ knows by default how to generate a few different kind of
> +filesystems, such as +squashfs+, +ext2/3/4+, +cramfs+... But those
> +filesystems are all generated to contain the complete target directory
> +hierarchy in a single filesystem, mounted as the root filesystem +/+.
> +That is, even if you select both an +ext2+ and a +squashfs+ filesystems,
> +the content of the two generated images will be the exact same, only the
> +types of the filesystems will be different.
> +
> +Most devices require a more complex setup, with different parts of the
> +directory structure split across different filesystems, each stored on
> +different partitions of one or more storage devices.
> +
> ++Buildroot+ can generate such complex setups, using a +partition table layout
> +description+. This is a simple text file, not unlike the +.ini+ style of
> +configuration files, that describes how the target directory hierarchy has
> +to be split across the target storage devices. It is a bit like a flattened
> +tree of the storage layout.
> +
> +Set the variable +BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE+ to the path of
> +the file containing your +partition table layout description+.
> +
> +See xref:part-layout-desc-syntax[] for the complete documentation of the
> ++partition table layout description+ syntax.
> +
> +[underline]*Note:* Although more versatile than the single filesystem image
> +mechanism, the +partition table layout description+ might be unable to
> +describe very complex setups. For example, it is not capable of handling
> +NFS-mounted filesystems, and +initramfs+ setups is not trivial (although
> +possible in most cases).
> diff --git a/docs/manual/customize.txt b/docs/manual/customize.txt
> index 7e46fd8..6d062ea 100644
> --- a/docs/manual/customize.txt
> +++ b/docs/manual/customize.txt
> @@ -14,6 +14,8 @@ include::customize-kernel-config.txt[]
>
> include::customize-toolchain.txt[]
>
> +include::customize-filesystems.txt[]
> +
> include::customize-store.txt[]
>
> include::customize-packages.txt[]
> diff --git a/docs/manual/partition-layout.txt b/docs/manual/partition-layout.txt
> new file mode 100644
> index 0000000..092ae1b
> --- /dev/null
> +++ b/docs/manual/partition-layout.txt
> @@ -0,0 +1,298 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +[[part-layout-desc-syntax]]
> +
> +Partition table layout description syntax
> +-----------------------------------------
> +
> +The +partition table layout description+ syntax is not unlike the standard
> +https://en.wikipedia.org/wiki/.ini[+.ini+] syntax. There are two types of
> +entries: +sections+, that may each contain zero or more +properties+.
> +
> ++Sections+ are specified between square brackets +[]+, _eg._: +[name]+.
> +
> ++Properties+ are specified as key-value pairs, _eg._: +key=value+, and
> +are documented as:
> +
> +* +key-name+ (optional or mandatory): description
> +** +value1+: description
> +** +value2+: description
> +** ...
> +
> +[underline]*Note:* Unlike the standard +.ini+ syntax, the +partition table
> +layout description+ _is_ case-sensitive.
> +
> +The order of +sections+ is irrelevant. However, for readability, we recomend
> +the +partition table layout description+ starts with the +global+ section.
> +
> +The global section
> +~~~~~~~~~~~~~~~~~~
> +
> +The +[global]+ section defines some global settings, and the list of devices.
> +
> +Properties for the global section
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +extract+ (mandatory): the type of base image to extract
> +** +tar+: extract the +rootfs.tar+ base image generated by +Buildroot+
> +
> +* +devices+ (mandatory): the comma-separated list of storage devices to
> + use on the device. Each device is the filename of the device node
> + present in +/dev+
> +
> +* +keep_partitions+ (optional): also copy the individual partition images
> + of all devices to +$(BINARIES_DIR)+. Settings from the device sections
> + or the partition sections take precedence over this one.
> +** +yes+: copy the individual partition images
> +** +no+ (the default): do not copy individual partition images
> +
> +The devices and partitions sections
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The devices and partitions sections define, for each device or partition,
> +the content of that device or partition.
> +
> +For each device listed in the +[global]+ section, there must be a
> +corresponding section named after that device.
> +
> +For each partition listed in a device section, there must be a corresponding
> +section named after that partition.
> +
> +Properties for the device section
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +type+ (mandatory): the type of content for that device or partition
> +** +boot+: the device contains one or more partitions, and
> + _may_ serve as a boot device
> +
> +* +keep_partitions+ (optional): also copy the individual partition images
> + for this device to +$(BINARIES_DIR)+. Settings from the partition
> + sections take precedence over this one.
> +** +yes+: copy the individual partition images
> +** +no+ (the default): do not copy individual partition images
> +
> +Properties for the partition section
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +type+ (mandatory): the type of content for that device or partition
> +** +fs+: the partition contains a filesystem
> +
> +* +size+: the size of that partition, in bytes
> +
> +* +keep+ (optional): copy this partition image to +$(BINARIES_DIR)+
> +** +yes+: copy this partition image
> +** +no+ (the default): do not copy this partition image
> +
> +Properties for +type=boot+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +boot_type+ (mandatory): the partitioning scheme to use on this device
> +** +mbr+: use an https://en.wikipedia.org/wiki/Master_boot_record[MBR]
> + partitioning scheme
> +** +gpt+: use a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
> + partitioning scheme
> +
> +* +partitions+ (mandatory): the comma-separated list of partition(s) on
> + this device; no two partitions may have the same name, even if they
> + reside on different devices; partitions names shall match this regexp:
> + `^[[:alpha:]][[:alnum:]-_]*$` (_ie._ starts with a letter, followed by
> + zero or more alpha-numeric character or a dash or an underscore)
> +
> +* +partalign+ (optional): the alignment of partitions, in bytes; defaults
> + to an alignment of one, which means no alignment; depending on the
> + +boot_type+, some restrictions may apply, and are documented for each
> + +boot_type+
> +
> +Properties for +boot_type=mbr+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +mbr_bootcode+ (optional): the bootcode to use, as a path to the file
> + containing the bootcode image, relative to the +$(BINARIES_DIR)+
> + directory; defaults to no bootcode (eg. filled with zeroes)
> +
> +* +partalign+: must be a multiple of 512
> +
> +Properties for +boot_type=gpt+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +partalign+: must be a multiple of 512
> +
> +********
> +Currently, only 512-byte sectors are supported. 4k sectors are not.
> +********
> +
> +Properties for partitions whose containing device is +boot_type=mbr+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +mbr_type+ (mandatory): the partition
> + https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs[type]
> +
> +Properties for +type=fs+
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +fs_type+ (mandatory): the type of filesystem to generate
> +** +ext+: generate an extended filesystem (ext2, ext3, ext4)
> +** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
> +
> +* +fs_label+ (optional): the label to assign to this filesystem, if that
> + filesystem supports a label
> +
> +* +fs_files_0+, +fs_files_1+, +fs_files_N+ (optional): the list of files,
> + relative to $(BINARIES_DIR), to store in the filesystem. These entries
> + must be indexed starting from 0, and must be sequential: the first
> + missing entry ends the list
> +
> +* +fs_root+ (optional): the mountpoint of the filesystem
> +
> +* +fs_vfstype+ (optional): the type of filesystem to use when calling
> + +mount+, if different from +fs_type+
> +
> +* +fs_mntopts+ (optional): the mount options; defaults to +defaults+
> +
> +_Note_: valid use-cases for +fs_root+ and +fs_files_N+:
> +
> +* if only +fs_root+ is specified (and no +fs_files_N+): the filesystem
> + content is made exclusively from +$(TARGET_DIR)/$(fs_root)+, and the
> + filesystem is mounted at runtime
> +
> +* if both +fs_root+ and at least one +fs_files_N+ are specified: the
> + filesystem content is made exclusively from the +fs_files_N+ entries,
> + and mounted at runtime. +$(TARGET_DIR)/$(fs_root)+ must be empty
> +
> +* if at least one +fs_files_N+ is specified, and +fs_root+ is not: the
> + filesystem content is made exclusively from the +fs_files_N+ entries,
> + and the filesystem is not mounted at runtime
> +
> +* if neither +fs_root+ nor +fs_files_N+ is specified: this is an error
> +
> +Properties for +fs_type=ext+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +ext_gen+ (mandatory): the generation of extended filesystem to generate
> +** +2+, +3+, +4+: for an ext2, ext3 or ext4 filesystem
> +
> +* +ext_rev+ (mandatory): the revision of the extended filesystem
> +** +0+ (ext2 only): generate a revision 0 extended filesystem filesystem
> +** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
> + filesystem
> +
> +Properties for +fs_type=vfat+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +vfat_size+ (optional): the VFAT-size of the filesystem
> +** +12+, +16+, +32+: generate a FAT12, FAT16, or FAT32
> +
> +Generation of the filesystems
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The filesystems are generated in an order such that a filesystem that is
> +mounted as a sub-directory of another filesystem is generated first.
> +
> +A filesystem is filled with the content of the directory corresponding to
> +its mountpoint, and then that directory is emptied before continuing to the
> +next filesystem. That way, the mountpoints are empty before the filesystem
> +that contains them are generated.
> +
> +Finally, an entry in added in +/etc/fstab+ for each generated filesystem, so
> +they are mounted at boot time.
> +
> +Examples
> +~~~~~~~~
> +
> +.Simplest partition table layout description
> +====
> +----
> +[global]
> +extract=tar
> +devices=sda
> +
> +[sda]
> +type=boot
> +boot_type=mbr
> +partitions=root
> +partalign=1048576
> +
> +[root]
> +type=fs
> +fs_type=ext
> +fs_vfstype=ext4
> +fs_root=/
> +ext_gen=4
> +ext_rev=1
> +----
> +
> +The +partition table layout description+ above defines a single device
> ++sda+. That device contains a single partition, +root+, with an ext4
> +filesystem, which is filled with the whole content of the +rootfs.tar+,
> +and is mounted on +/+.
> +====
> +
> +.More complex table layout description
> +====
> +----
> +[global]
> +extract=tar
> +devices=mmcblk0,sda
> +
> +[mmcblk0]
> +type=boot
> +boot_type=mbr
> +partitions=boot,root
> +partalign=$((1024*1024))
> +
> +[sda]
> +type=boot
> +boot_type=mbr
> +partitions=data
> +partalign=4096
> +
> +[boot]
> +type=fs
> +mbr_type=$((0xC))
> +size=$((16*1048576))
> +fs_type=vfat
> +fs_mntopts=ro
> +fs_label=BOOT
> +fs_root=/boot
> +vfat_size=32
> +
> +[root]
> +type=fs
> +mbr_type=$((0x83))
> +size=268435456
> +fs_type=ext
> +fs_vfstype=ext4
> +fs_mntopts=discard,delalloc
> +fs_root=/
> +fs_label=ROOT
> +ext_gen=4
> +ext_rev=1
> +
> +[data]
> +type=fs
> +mbr_type=$((0x83))
> +size=$((4*1024*1048576))
> +fs_type=ext
> +fs_vfstype=ext2
> +fs_root=/data
> +fs_label=DATA
> +ext_gen=2
> +ext_rev=1
> +----
> +====
> +
> +The example above defines two devices, +mmcblk0+ and +sda+.
> +
> +The +mmcblk0+ device contains two partitions, +boot+ and +root+; partitions
> +are aligned on a 1MiB boundary. The +sda+ device contains a single partition,
> ++data+, aligned on a 4KiB boundary.
> +
> +The +boot+ partition is a 16MiB FAT32 filesystem filled with the content
> +of, and mounted on, +/boot+, and with label +BOOT+.
> +
> +The +data+ partition is a 4GiB ext2r1 filesystem filled with the content
> +of, and mounted on, +/data+, and with label +DATA+.
> +
> +The +root+ partition is a 256MiB ext4 filesystem filled the the rest of,
> +and mounted on, +/+, and with label +ROOT+.
How about comments ? Are they supported ?
In a next step, it would be really nice to have support for "human"
input for the different size, like 1G/M/K.
> diff --git a/fs/Config.in b/fs/Config.in
> index da4c5ff..44e04f7 100644
> --- a/fs/Config.in
> +++ b/fs/Config.in
> @@ -3,6 +3,7 @@ menu "Filesystem images"
> source "fs/cloop/Config.in"
> source "fs/cpio/Config.in"
> source "fs/cramfs/Config.in"
> +source "fs/custom/Config.in"
> source "fs/ext2/Config.in"
> source "fs/initramfs/Config.in"
> source "fs/iso9660/Config.in"
> diff --git a/fs/custom/Config.in b/fs/custom/Config.in
> new file mode 100644
> index 0000000..e5a8ee7
> --- /dev/null
> +++ b/fs/custom/Config.in
> @@ -0,0 +1,16 @@
> +config BR2_TARGET_ROOTFS_CUSTOM
> + bool "Custom partition table layout"
> + select BR2_TARGET_ROOTFS_TAR
> +
> +config BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE
> + string "path to the custom partition table layout description"
> + depends on BR2_TARGET_ROOTFS_CUSTOM
> + help
> + Enter the path to a partition-table for your device.
> +
> + This will allow Buildroot to generate a more complex target
> + image, which may consist of more than one filesystem on more
> + than one partition.
> +
> + See docs/manual/bla-bla on how to construct such a partition
> + table.
> diff --git a/fs/custom/boot/gpt b/fs/custom/boot/gpt
> new file mode 100644
> index 0000000..f978524
> --- /dev/null
> +++ b/fs/custom/boot/gpt
> @@ -0,0 +1,129 @@
> +# Build a complete GPT-based image
> +
> +#-----------------------------------------------------------------------------
> +DEPENDS+=( parted )
> +
> +#-----------------------------------------------------------------------------
> +# For a GPT-based partitionning, we need to compute the complete
> +# image size before we can attempt to generate the partition table.
> +# Then, we need to add the size of the GPT itself, plus that of its
> +# backup copy, plus the protective MBR.
> +# The size of the GPT itself depends on the sector size, and the
> +# number of partitions in the GPT. Sectors can be either 512-byte
> +# or 4096-byte large; The numbers of partitions is unlimited, but
> +# it is suggested there is space to store at least 128 of them; a
> +# partition description is 128-byte large.
> +#
> +# https://en.wikipedia.org/wiki/GUID_Partition_Table
> +#
> +# So, here's what we do:
> +# - consider 512-byte sectors (since GPT on 4k sectors is not well
> +# documented)
> +# - consider at least 128 partitions; if the layout defines more than
> +# that, we need to round that number up to the smallest multiple of
> +# 4 (since there are 4 partition descriptions in a 512-byte sector)
> +# - generate an empty, sparse file that is big enough to store the MBR,
> +# the two GPT copies, and the aligned partitions.
> +# - dump each partition in turn in their final location in that file
> +# - generate a parted script that creates the partition table in that
> +# file
> +#
> +# Simg = 512 + 2*(Sgpt) + ?( aligned(Spart,512) )
Don't we want only ASCII char there ?
> +# Sgpt = 512 + Nent*128
> +#
> +# Where:
> +# Simg : size of the image
> +# Sgpt : size of one GPT
> +# Spart : size of each partition
> +# Nent : number of partition entries
> +# aligned() : the alignment function
> +#
> +# Sicne 4k-large sectors are not really explained on Wikipedia, we can
Small typo here.
> +# add this later on.
> +do_image() {
> + # ${1} is fs_root, irrelevant here
> + local img="${2}"
> +
> + # How many partitions do we have?
> + nb_parts=0
> + for part in ${partitions[${dev}]//,/ }; do
> + nb_parts=$((nb_parts+1))
> + done
> +
> + # How many partition entries do we need?
> + nb_entries=$((4*((nb_parts+3)/4)))
> + nb_entries=$((nb_entries<128?128:nb_entries))
> +
> + # The size of a single GPT
> + gpt_size=$((512+(128*nb_entries)))
> +
> + # Offset of the first partition
> + begin=$(align_val $((512+$(align_val ${gpt_size} 512))) ${partalign} )
> +
> + # Initialise our image file
> + dd if=/dev/zero of="${img}" \
> + bs=1 seek=${begin} count=0 \
> + conv=sparse 2>/dev/null
Why the redirect to /dev/null ? Don't we want to see if there was an
error ? I know it's unlikely to happen but you never know.
I have noticed you did several times so maybe there is a reason I am
not seeing here.
> +
> + # Compute the space required to store all partitions
> + # and store them in the image file
> + size_parts=0
> + _offset=${begin}
> + i=1
> + debug "adding partions descriptions\n"
> + for part in ${partitions[${dev}]//,/ }; do
> + debug " %s\n" "${part}"
> + _part_img="${tmp_dir}/${dev}.${part}.img"
> + _size=$( align_val $( stat -c '%s' "${_part_img}" ) 512 )
> + part_offset+=( ${_offset} )
> + _attr="${values["${part}:gpt_attr"]}"
> + _label="${values["${part}:gpt_label"]}"
> +
> + # If the partition has no label, use the filesystem label
> + if [ -z "${_label}" ]; then
> + _label="${values["${part}:fs_label"]}"
> + fi
> + if [ -z "${_label}" ]; then
> + _label="data"
> + fi
> +
> + debug " start=%s\n" "${_offset}"
> + debug " size =%s\n" "${_size}"
> + debug " end =%s\n" "$((_offset+_size-1))"
> +
> + dd if="${_part_img}" of="${img}" \
> + bs=512 seek=$((_offset/512)) \
> + conv=notrunc,sparse 2>/dev/null
> +
> + parted_script+=( mkpart "${_label}" \
> + ${_offset} \
> + $((_offset+_size-1)) \
> + )
> + if [ -n "${_attr}" ]; then
> + for attr in "${_attr//,/ }"; do
> + parted_script+=( set ${i} ${attr} on )
> + done
> + fi
> +
> + size_parts=$((size_parts+_size))
> + _offset=$((_offset+_size))
> + i=$((i+1))
> + done
> +
> + # Terminate our image file
> + img_size=$(align_val $(( begin + size_parts + gpt_size )) 512)
> + debug "begin =%s\n" ${begin}
> + debug "nb_entry=%s\n" ${nb_entries}
> + debug "gpt_size=%s\n" ${gpt_size}
> + debug "img_size=%s\n" ${img_size}
> + dd if=/dev/zero of="${img}" \
> + bs=1 seek=${img_size} \
> + count=0 conv=sparse 2>/dev/null
> +
> + for i in parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"; do
> + debug "--> '%s'\n" "${i}"
> + done
> + parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"
> +}
> +
> +# vim: ft=sh
> diff --git a/fs/custom/boot/mbr b/fs/custom/boot/mbr
> new file mode 100644
> index 0000000..667feed
> --- /dev/null
> +++ b/fs/custom/boot/mbr
> @@ -0,0 +1,64 @@
> +# Build a complete MBR-based image
> +
> +#-----------------------------------------------------------------------------
> +DEPENDS+=( genpart )
> +
> +#-----------------------------------------------------------------------------
> +do_image() {
> + # ${1} is fs_root, irrelevant here
> + local img="${2}"
> + local i begin part part_img size type _begin _size
> + local -a part_offset part_file bootcode
> +
> + # Fill-in the boot record
> + bootcode="${values["${dev}:mbr_bootcode"]}"
> + if [ -n "${bootcode}" ]; then
> + bootcode="${BINARIES_DIR}/${bootcode}"
> + else
> + bootcode="/dev/zero"
> + fi
> + debug "adding bootcode '%s'\n" "${bootcode}"
> + dd if="${bootcode}" of="${img}" bs=$((0x1be)) count=1 2>/dev/null
> +
> + # Generate partition entries
> + i=0
> + begin=${partalign}
> + debug "adding partitions descriptors\n"
> + for part in ${partitions[${dev}]//,/ }; do
> + debug " %s\n" "${part}"
> + part_offset+=( ${begin} )
> + part_img="${tmp_dir}/${dev}.${part}.img"
> + part_file+=( "${part_img}" )
> + size=$( align_val $( stat -c '%s' "${part_img}" ) 512 )
> + type="${values["${part}:mbr_type"]}"
> + # LBA is exressed in a number of 512-byte blocks
> + # and genparts only deals with LBA
> + _begin=$((begin/512)) # begin is already 512-byte aligned
> + _size=$((size/512)) # size is already 512-byte aligned
> + debug " start=%s (LBA %s)\n" "${begin}" "${_begin}"
> + debug " size =%s (LBA %s)\n" "${size}" "${_size}"
> + debug " type =%s\n" "${type}"
> + genpart -b ${_begin} -s ${_size} -t ${type} >>"${img}"
> + begin=$( align_val $((begin+size)) ${partalign} )
> + i=$((i+1))
> + done
> + nb_parts=${i}
> + # Generate entries for empty partitions
> + for(( ; i<4; i++ )); do
> + debug " (empty)\n"
> + genpart -t 0 >>"${img}"
> + done
> + # Dump the boot signature
> + printf "\x55\xaa" >>"${img}"
> +
> + for(( i=0; i<nb_parts; i++ )); do
> + part_img="${part_file[${i}]}"
> + offset=${part_offset[${i}]}
> + _offset=$(( offset/512 )) # offset is already 512-byte aligned
> + dd if="${part_img}" of="${img}" \
> + bs=512 seek=${_offset} \
> + conv=notrunc,sparse 2>/dev/null
> + done
> +}
> +
> +# vim: ft=sh
> diff --git a/fs/custom/boot/pre-post b/fs/custom/boot/pre-post
> new file mode 100644
> index 0000000..af4bcf5
> --- /dev/null
> +++ b/fs/custom/boot/pre-post
> @@ -0,0 +1,14 @@
> +#-----------------------------------------------------------------------------
> +# No dependencies
> +
> +#-----------------------------------------------------------------------------
> +do_image_pre() {
> + :
> +}
> +
> +#-----------------------------------------------------------------------------
> +do_image_post() {
> + :
> +}
> +
> +#vim: set ft=sh
> diff --git a/fs/custom/custom.mk b/fs/custom/custom.mk
> new file mode 100644
> index 0000000..ca53e0b
> --- /dev/null
> +++ b/fs/custom/custom.mk
> @@ -0,0 +1,25 @@
> +################################################################################
> +#
> +# custom partitioning
> +#
> +################################################################################
> +
> +# rootfs-custom uses rootfs.tar as the source to generate
> +# the resulting image(s), so we need to build it first.
> +ROOTFS_CUSTOM_DEPENDENCIES += rootfs-tar
> +
> +# If we are not selected, we won't have a partition table, so genimages
> +# will complain (both on stdout and return code), so it will faill, so
> +# we should not get our dependencies
> +ifeq ($(BR2_TARGET_ROOTFS_CUSTOM),y)
> +ROOTFS_CUSTOM_DEPENDENCIES += \
> + $(patsubst %,host-%,$(shell $(USER_HOOKS_EXTRA_ENV) fs/custom/genimages --show-depends \
> + '$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'))
> +endif
> +
> +define ROOTFS_CUSTOM_CMD
> + $(USER_HOOKS_EXTRA_ENV) fs/custom/genimages \
> + '$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
> +endef
> +
> +$(eval $(call ROOTFS_TARGET,custom))
> diff --git a/fs/custom/fs/ext b/fs/custom/fs/ext
> new file mode 100644
> index 0000000..a184969
> --- /dev/null
> +++ b/fs/custom/fs/ext
> @@ -0,0 +1,26 @@
> +# Create an extended file system
> +
> +#-----------------------------------------------------------------------------
> +DEPENDS+=( e2fsprogs genext2fs )
> +
> +#-----------------------------------------------------------------------------
> +do_image() {
> + local root_dir="${1}"
> + local img="${2}"
> + local -a fs_opts
> + local gen rev
> +
> + fs_opts+=( -z )
> + fs_opts+=( -d "${root_dir}" )
> + [ -z "${size}" ] || fs_opts+=( -b $((size/1024)) )
> + [ -n "${ext_gen}" ] || ext_gen=2
> + [ -n "${ext_rev}" ] || ext_rev=1
> +
> + # Remember, we're running from Buildroot's TOP_DIR
> + GEN=${ext_gen} REV=${ext_rev} \
> + ./fs/ext2/genext2fs.sh "${fs_opts[@]}" "${img}" >/dev/null
> +
> + [ -z "${fs_label}" ] || tune2fs -L "${fs_label}" "${img}" >/dev/null
> +}
> +
> +# vim: ft=sh
> diff --git a/fs/custom/fs/pre-post b/fs/custom/fs/pre-post
> new file mode 100644
> index 0000000..d931437
> --- /dev/null
> +++ b/fs/custom/fs/pre-post
> @@ -0,0 +1,72 @@
> +#-----------------------------------------------------------------------------
> +# No additional dependencies
> +
> +#-----------------------------------------------------------------------------
> +do_image_pre() {
> + local i file
> +
> + # if fs_root_dir is not specified, we have to create one
> + # It *does* override the caller's fs_root_dir value, but
> + # that's on purpose
> + # If fs_root_dir is specified, and we have at least fs_files_0,
> + # then fs_root_dir/ must be enpty
> + if [ -z "${fs_root_dir}" ]; then
> + if [ -n "${values["${part}:fs_files_0"]}" ]; then
> + error "%s: no fs_root specified, and no fs_files_0\n" "${part}"
> + fi
> + fs_root_dir="$( mktemp -d "${tmp_dir}/XXXXXX" )"
> + else
> + if [ -n "${values["${part}:fs_files_0"]}" \
> + -a $( ls -1A "${fs_root_dir}" 2>/dev/null \
> + |wc -l \
> + ) -ne 0 ]; then
> + error "%s: %s is not empty, but fs_files_0 is specified\n" \
> + "${part}" "${fs_root_dir#${fs_root}}"
> + fi
> + fi
> +
> + i=0
> + while true; do
> + file="${values["${part}:fs_files_${i}"]}"
> + [ -n "${file}" ] || break
> + debug "%s: adding fs_files_%d %s\n" "${part}" ${i} "${file}"
> + install -D "${BINARIES_DIR}/${file}" "${fs_root_dir}/${file##*/}"
> + i=$((i+1))
> + done
> +}
> +
> +#-----------------------------------------------------------------------------
> +do_image_post() {
> + local rootfs_dir="${1}"
> + local fs_root="${2}"
> + local img_file="${3}"
> + local part="${4}"
> + local dev mntops vfstype fs_root_esc
> +
> + subname+="[post-image]"
> +
> + # Empty the partition's mountpoint
> + find "${fs_root_dir}" -maxdepth 1 \! -path "${fs_root_dir}" -exec rm -rf {} +
> +
> + # Add entry in fstab, but not if this is '/'
> + # Don't add either if rootfs was not extracted
> + if [ "${fs_root}" = "/" -o -z "${fs_root}" \
> + -o -z "${values["global:extract"]}" ]; then
> + return 0
> + fi
> + fs_root_esc="$( sed -r -e 's:/:\\/:g;' <<<"${fs_root}" )"
> + sed -r -i -e "/[^[:space:]]+[[:space:]]+${fs_root_esc}[[:space:]]/d" \
> + "${rootfs_dir}/etc/fstab"
> + dev="$( get_part_dev_node "${part}" )"
> + vfstype="${fs_vfstype:-${fs_type}}"
> + mntops="${fs_mntops:-defaults}"
> + printf "/dev/%s %s %s %s 0 0\n" \
> + "${dev}" "${fs_root}" \
> + "${vfstype}" "${mntops}" \
> + >>"${rootfs_dir}/etc/fstab"
> +
> + subname="${subname%\[post-image\]}"
> +}
> +
> +#-----------------------------------------------------------------------------
> +# vim: ft=sh
> diff --git a/fs/custom/fs/vfat b/fs/custom/fs/vfat
> new file mode 100644
> index 0000000..5fc243d
> --- /dev/null
> +++ b/fs/custom/fs/vfat
> @@ -0,0 +1,21 @@
> +# Create a VFAT file system
> +
> +#-----------------------------------------------------------------------------
> +DEPENDS+=( dosfstools mtools )
> +
> +#-----------------------------------------------------------------------------
> +do_image() {
> + local root_dir="${1}"
> + local img="${2}"
> + local -a fs_opts
> +
> + dd if=/dev/zero of="${img}" bs=${size} count=0 seek=1 2>/dev/null
> +
> + [ -z "${vfat_size}" ] || fs_opts+=( -F ${vfat_size} )
> + [ -z "${fs_label}" ] || fs_opts+=( -n "${fs_label}" )
> + mkfs.vfat "${fs_opts[@]}" "${img}" >/dev/null
> +
> + mcopy -i "${img}" "${root_dir}/"* '::'
> +}
> +
> +# vim: ft=sh
> diff --git a/fs/custom/genimages b/fs/custom/genimages
> new file mode 100755
> index 0000000..204f13b
> --- /dev/null
> +++ b/fs/custom/genimages
> @@ -0,0 +1,342 @@
> +#!/bin/bash
> +
> +#-----------------------------------------------------------------------------
> +main() {
> + local part_table="${1}"
> + local tmp_dir
> + local rootfs_dir
> + local -a devices
> + local extract
> + local cur_section
> + local -a sections devices partitions
> + local -A variables values partdevs
> + local sec dev part var val
> + local secs devs parts vars vals
> + local has_global_section
> +
> + # We need bash 4 or above for associative arrays
> + if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
> + error "bash 4 or above is needed\n"
> + fi
> +
> + if [ "${part_table}" = "--show-depends" ]; then
> + SHOW_DEPENDS=1
> + part_table="${2}"
> + trace() { :; } # Be silent, we just want the dependencies...
> + else
> + SHOW_DEPENDS=0
> + fi
> +
> + if [ ! -f "${part_table}" ]; then
> + error "'%s': no such file\n" "${part_table}"
> + exit 1
> + fi
> +
> + export PATH="${HOST_DIR}/usr/bin:${HOST_DIR}/usr/sbin:${PATH}"
> +
> + # Parse all the sections in one go, we'll sort
> + # all the mess afterwards...
> + debug "parsing partitions descriptions file '%s'\n" \
> + "${part_table}"
> + parse_ini "${part_table}"
> +
> + # The 'global' section is mandatory
> + has_global_section=0
> + for s in "${sections[@]}"; do
> + if [ "${s}" = "global" ]; then
> + has_global_section=1
> + break
> + fi
> + done
> + if [ ${has_global_section} -eq 0 ]; then
> + error "no global section defined\n"
> + fi
> +
> + # Create lists of devices, partitions, and partition:device pairs.
> + debug "creating intermediate lists\n"
> + devices=( ${values["global:devices"]//,/ } )
> + for dev in "${devices[@]}"; do
> + # Sanity check first: all devices must have a corresponding
> + # section, which means they should have a type
> + if [ -z "${values["${dev}:type"]}" ]; then
> + error "device '%s' has no type (no section?)\n" "${dev}"
> + fi
> + partitions+=( ${values["${dev}:partitions"]//,/ } )
> + for part in ${values["${dev}:partitions"]//,/ }; do
> + # Sanity check first: all partitions must have a corresponding
> + # section, which means they should have a type
> + if [ -z "${values["${part}:type"]}" ]; then
> + error "partition '%s:%s' has no type (no section?)\n" \
> + "${dev}" "${part}"
> + fi
> + partdevs+=( ["${part}"]="${dev}" )
> + done
> + done
> +
> + # Now, we must order the partitions so that their mountpoint
> + # is empty by the time we build the upper-level partition.
> + # For example, given this layout of mountpoints:
> + # /
> + # /usr
> + # /usr/var
> + # We must ensure /usr/var is empty at the time we create the /usr
> + # filesystem image; and similarly, we must ensure /usr is empty by
> + # the time we create the / filesystem image
> + # So, a simple reverse alphabetical sort will do the trick
> + debug "sorting partitions\n"
> + sorted_parts=( $(
> + for part in "${partitions[@]}"; do
> + # Partitions that are not mounted can be generated
> + # in any order
> + if [ -n "${values["${part}:fs_root"]}" ]; then
> + printf "%s:%s\n" "${part}" "${values["${part}:fs_root"]}"
> + else
> + printf "%s\n" "${part}"
> + fi
> + done \
> + |sort -t: -k2 -r \
> + |sed -r -e 's/:[^:]+$//;'
> + ) )
> +
> + # We do not want to create anything if we only want the dependencies
> + if [ ${SHOW_DEPENDS} -eq 0 ]; then
> + tmp_dir="${BUILD_DIR}/genimages.tmp"
> + rootfs_dir="${tmp_dir}/rootfs"
> + # Since we don't remove it in case of error (to be able to inspect its
> + # content), we must remove it now (a previous run may have left it).
> + rm -rf "${tmp_dir}"
> + mkdir -p "${rootfs_dir}"
> +
> + case "${values["global:extract"]}" in
> + tar)
> + # We must be root for the extract to work correctly
> + # (since it may have /dev nodes, or some files may
> + # belong to different users...)
> + if [ $(id -u) -ne 0 ]; then
> + error "error: not root\n"
> + fi
> + trace "extracting rootfs.tar\n"
> + tar xf "${BINARIES_DIR}/rootfs.tar" -C "${rootfs_dir}"
> + ;;
> + *) error "unknown extract method '%s'\n" "${extract:-(none)}"
> + ;;
> + esac
> + fi # ! SHOW_DEPENDS
> +
> + # Render all partition images
> + for part in "${sorted_parts[@]}"; do
> + trace "preparing filesystem for partition '%s'\n" "${part}"
> + render_img "${rootfs_dir}" "${part}" \
> + "${tmp_dir}/${partdevs["${part}"]}.${part}.img"
> + done
> +
> + # Aggregate all devices images
> + for dev in "${devices[@]}"; do
> + trace "assembling partitions in device '%s'\n" "${dev}"
> + render_img "${rootfs_dir}" "${dev}" "${tmp_dir}/${dev}.img"
> + done
> +
> + # If we need the dependencies, we can stop right now
> + if [ ${SHOW_DEPENDS} -eq 1 ]; then
> + return 0
> + fi
> +
> + # Copy all partitions and devices images to the image dir
> + if [ "${values["global:keep_partitions"]}" = "yes" ]; then
> + for part in "${sorted_parts[@]}"; do
> + debug "copying partition '%s' to image dir\n" "${part}"
> + dd if="${tmp_dir}/${partdevs["${part}"]}.${part}.img" \
> + of="${BINARIES_DIR}/$( get_part_dev_node "${part}" ).img" \
> + bs=4096 conv=sparse 2>/dev/null
> + done
> + fi
> + for dev in "${devices[@]}"; do
> + debug "copying device '%s' to image dir\n" "${dev}"
> + dd if="${tmp_dir}/${dev}.img" \
> + of="${BINARIES_DIR}/${dev}.img" \
> + bs=4096 conv=sparse 2>/dev/null
> + done
> +
> + [ -n "${DEBUG}" ] || rm -rf "${tmp_dir}"
> +}
> +
> +#-----------------------------------------------------------------------------
> +render_img() {
> + local rootfs_dir="${1}"
> + local img="${2}"
> + local img_file="${3}"
> + local type sub_type fs_root_dir
> +
> + type="${values["${img}:type"]}"
> + sub_type="${values["${img}:${type}_type"]}"
> +
> + # Sanity checks
> + [ -n "${type}" ] || error "'%s': unspecified type\n" "${img}"
> + if [ ! -d "fs/custom/${type}" ]; then
> + error "'%s': unsupported type '%s'\n" "${img}" "${type}"
> + fi
> + [ -n "${sub_type}" ] || error "'%s': unspecified %s_type\n" "${img}" "${type}"
> + if [ ! -f "fs/custom/${type}/${sub_type}" ]; then
> + error "'%s': unknown %s_type '%s'\n" "${img}" "${type}" "${sub_type}"
> + fi
> +
> + # Need to call the renderer in a subshell so that its definitions
> + # do not pollute our environment
> + subname="${sub_type}"
> + (
> + trap 'exit $?' ERR
> +
> + declare -a DEPENDS
> +
> + for var in ${variables["${img}"]//,/ }; do
> + eval "${var}=\"${values["${img}:${var}"]}\""
> + done
> + fs_root_dir="${rootfs_dir}${fs_root}"
> + . "fs/custom/${type}/pre-post"
> + . "fs/custom/${type}/${sub_type}"
> + if [ ${SHOW_DEPENDS} -eq 1 ]; then
> + for dep in "${DEPENDS[@]}"; do
> + printf "%s\n" "${dep}"
> + done
> + else
> + do_image_pre "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
> + do_image "${fs_root_dir}" "${img_file}"
> + do_image_post "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
> + fi
> + )
> + ret=${?}
> + [ ${ret} -eq 0 ] || exit ${ret}
Why not doing: [ ${ret} -ne 0 ] && exit ${ret} ?
> + subname=""
> +}
> +
> +#------------------------------------------------------------------------------
> +# Parse a .ini file
> +# $1: .ini file to parse
> +# The caller should define the following variables:
> +# sections : array
> +# variables : associative array
> +# values : associative array
> +# parse_ini() will fill those variables with:
> +# sections : the list of sections, one section per array index
> +# variables : the comma-separated list of varibles for a section,
> +# indexed by the name of the section
> +# values : the value of a variable in a section, indexed by the
> +# 'section:variable' tuple
> +# Eg.:
> +# sections=( [0]='section-0' [1]='section-1' )
> +# variables=( ['section-0']='var-0,var-1' ['section-1']='var-10,var-11' [...] )
> +# values=( ['section-0:var-0']='value-0-0' ['section-1:var-10']='value-1-10' [...] )
> +parse_ini() {
> + local ini_file="${1}"
> + local line var val
> + local cur_section
> + local var_sep
> +
> + while read line; do
> + line="$( sed -r -e 's/[[:space:]]*#.*$//; //d;' <<<"${line}" )"
> +
> + # Detect start of global section, skip anything else
> + case "${line}" in
> + "") continue;;
> + '['*']')
> + cur_section="$( sed -r -e 's/[][]//g;' <<<"${line}" )"
> + debug " entering section '%s'\n" "${cur_section}"
> + sections+=( "${cur_section}" )
> + continue
> + ;;
> + ?*=*) ;;
> + *) error "malformed entry '%s'\n" "${line}";;
> + esac
> +
> + var="${line%%=*}"
> + eval val="${line#*=}"
> + debug " adding '%s'='%s'\n" "${var}" "${val}"
> + var_sep="${variables["${cur_section}"]+,}"
> + variables+=( ["${cur_section}"]="${var_sep}${var}" )
> + values+=( ["${cur_section}:${var}"]="${val}" )
> + done <"${ini_file}"
> +}
> +
> +#-----------------------------------------------------------------------------
> +get_part_dev_node() {
> + local part="${1}"
> + local dev
> + local i c p
> +
> + dev="${partdevs["${part}"]}"
> + i="${values["${dev}:partstart"]:-1}"
> +
> + # If device node ends with a number, partitions are denoted
> + # with a 'p' before the partition number, eg.:
> + # /dev/mmcblk0 --> /dev/mmcblk0p1
> + # /dev/sda --> /dev/sda1
> + case "${dev#${dev%?}}" in
> + [0-9]) c="p";;
> + *) c="";;
> + esac
> +
> + for p in ${values["${dev}:partitions"]//,/ }; do
> + if [ "${p}" = "${part}" ]; then
> + printf "%s%s%d" "${dev}" "${c}" ${i}
> + return 0
> + fi
> + i=$((i+1))
> + done
> +
> + error "'%s': partition not found. WTF?\n" "${part}"
> +}
> +
> +#------------------------------------------------------------------------------
> +align_val() {
> + local val="${1}"
> + local align="${2}"
> + local aligned
> +
> + aligned=$(( ( (val+align-1) / align ) * align ))
> +
> + printf "%d" ${aligned}
> +}
> +
> +#------------------------------------------------------------------------------
> +# Some trace functions
> +_trace() {
> + local fmt="${1}"
> + shift
> +
> + printf "%s" "${myname}"
> + if [ -n "${subname}" ]; then
> + printf "(%s)" "${subname}"
> + fi
> + printf ": ${fmt}" "${@}"
> +}
> +
> +trace() {
> + _trace "${@}"
> +}
> +
> +debug() { :; }
> +if [ -n "${DEBUG}" ]; then
> + debug() {
> + _trace "${@}" >&2
> + }
> +fi
> +
> +error() {
> + _trace "${@}" >&2
> + exit 1
> +}
> +
> +on_error() {
> + local ret=${?}
> +
> + error "unexpected error caught: %d\n" ${ret}
> +}
> +trap on_error ERR
> +set -E -e
> +
> +#-----------------------------------------------------------------------------
> +export myname="${0##*/}"
> +
> +main "${@}"
> +
> +# vim: ft=sh
> --
> 1.8.1.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
2014-01-03 17:19 [Buildroot] [PATCH 0/5 v4] SUBJECT Yann E. MORIN
@ 2014-01-03 17:19 ` Yann E. MORIN
2014-01-04 16:38 ` Maxime Hadjinlian
2014-04-09 13:28 ` Andreas Naumann
0 siblings, 2 replies; 9+ messages in thread
From: Yann E. MORIN @ 2014-01-03 17:19 UTC (permalink / raw)
To: buildroot
From: "Yann E. MORIN" <yann.morin.1998@free.fr>
Contrary to the existing fs/ schemes, which each generate only a single
filesystem image for the root filesystem, this new scheme allows the
user to generate more complex images.
The basis behind this is a .ini-like description of the layout of the
final target storage:
- the list of device(s)
- per-device, the list of partition(s)
- per-partition, the content
It is possible to create MBR- or GPT-based partitoining schemes. Adding
new ones should be relatively easy (but would need adequate host tools).
For now, the only content possible for partitions is a filesystem. It
should be pretty easy to add new types (eg. un-formated, or raw blob).
Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding
more will be relatively easy, provided we have the necessary host
packages to deal with those filesystems.
The existing Buildroot filesystem generators are re-used as much as
possible when it makes sense; when it does not (eg. for vfat), a specific
generator is used.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ryan Barnett <rjbarnet@rockwellcollins.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: "J?r?me Oufella" <jerome.oufella@savoirfairelinux.com>
---
docs/manual/appendix.txt | 1 +
docs/manual/customize-filesystems.txt | 36 ++++
docs/manual/customize.txt | 2 +
docs/manual/partition-layout.txt | 298 +++++++++++++++++++++++++++++
fs/Config.in | 1 +
fs/custom/Config.in | 16 ++
fs/custom/boot/gpt | 129 +++++++++++++
fs/custom/boot/mbr | 64 +++++++
fs/custom/boot/pre-post | 14 ++
fs/custom/custom.mk | 25 +++
fs/custom/fs/ext | 26 +++
fs/custom/fs/pre-post | 72 +++++++
fs/custom/fs/vfat | 21 +++
fs/custom/genimages | 342 ++++++++++++++++++++++++++++++++++
14 files changed, 1047 insertions(+)
create mode 100644 docs/manual/customize-filesystems.txt
create mode 100644 docs/manual/partition-layout.txt
create mode 100644 fs/custom/Config.in
create mode 100644 fs/custom/boot/gpt
create mode 100644 fs/custom/boot/mbr
create mode 100644 fs/custom/boot/pre-post
create mode 100644 fs/custom/custom.mk
create mode 100644 fs/custom/fs/ext
create mode 100644 fs/custom/fs/pre-post
create mode 100644 fs/custom/fs/vfat
create mode 100755 fs/custom/genimages
diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
index 74ee8fd..53f4205 100644
--- a/docs/manual/appendix.txt
+++ b/docs/manual/appendix.txt
@@ -6,6 +6,7 @@ Appendix
include::makedev-syntax.txt[]
include::makeusers-syntax.txt[]
+include::partition-layout.txt[]
// Automatically generated lists:
diff --git a/docs/manual/customize-filesystems.txt b/docs/manual/customize-filesystems.txt
new file mode 100644
index 0000000..fd65c97
--- /dev/null
+++ b/docs/manual/customize-filesystems.txt
@@ -0,0 +1,36 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[filesystem-custom]]
+Customizing the generated filesystem images
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++Buildroot+ knows by default how to generate a few different kind of
+filesystems, such as +squashfs+, +ext2/3/4+, +cramfs+... But those
+filesystems are all generated to contain the complete target directory
+hierarchy in a single filesystem, mounted as the root filesystem +/+.
+That is, even if you select both an +ext2+ and a +squashfs+ filesystems,
+the content of the two generated images will be the exact same, only the
+types of the filesystems will be different.
+
+Most devices require a more complex setup, with different parts of the
+directory structure split across different filesystems, each stored on
+different partitions of one or more storage devices.
+
++Buildroot+ can generate such complex setups, using a +partition table layout
+description+. This is a simple text file, not unlike the +.ini+ style of
+configuration files, that describes how the target directory hierarchy has
+to be split across the target storage devices. It is a bit like a flattened
+tree of the storage layout.
+
+Set the variable +BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE+ to the path of
+the file containing your +partition table layout description+.
+
+See xref:part-layout-desc-syntax[] for the complete documentation of the
++partition table layout description+ syntax.
+
+[underline]*Note:* Although more versatile than the single filesystem image
+mechanism, the +partition table layout description+ might be unable to
+describe very complex setups. For example, it is not capable of handling
+NFS-mounted filesystems, and +initramfs+ setups is not trivial (although
+possible in most cases).
diff --git a/docs/manual/customize.txt b/docs/manual/customize.txt
index 7e46fd8..6d062ea 100644
--- a/docs/manual/customize.txt
+++ b/docs/manual/customize.txt
@@ -14,6 +14,8 @@ include::customize-kernel-config.txt[]
include::customize-toolchain.txt[]
+include::customize-filesystems.txt[]
+
include::customize-store.txt[]
include::customize-packages.txt[]
diff --git a/docs/manual/partition-layout.txt b/docs/manual/partition-layout.txt
new file mode 100644
index 0000000..092ae1b
--- /dev/null
+++ b/docs/manual/partition-layout.txt
@@ -0,0 +1,298 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[part-layout-desc-syntax]]
+
+Partition table layout description syntax
+-----------------------------------------
+
+The +partition table layout description+ syntax is not unlike the standard
+https://en.wikipedia.org/wiki/.ini[+.ini+] syntax. There are two types of
+entries: +sections+, that may each contain zero or more +properties+.
+
++Sections+ are specified between square brackets +[]+, _eg._: +[name]+.
+
++Properties+ are specified as key-value pairs, _eg._: +key=value+, and
+are documented as:
+
+* +key-name+ (optional or mandatory): description
+** +value1+: description
+** +value2+: description
+** ...
+
+[underline]*Note:* Unlike the standard +.ini+ syntax, the +partition table
+layout description+ _is_ case-sensitive.
+
+The order of +sections+ is irrelevant. However, for readability, we recomend
+the +partition table layout description+ starts with the +global+ section.
+
+The global section
+~~~~~~~~~~~~~~~~~~
+
+The +[global]+ section defines some global settings, and the list of devices.
+
+Properties for the global section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +extract+ (mandatory): the type of base image to extract
+** +tar+: extract the +rootfs.tar+ base image generated by +Buildroot+
+
+* +devices+ (mandatory): the comma-separated list of storage devices to
+ use on the device. Each device is the filename of the device node
+ present in +/dev+
+
+* +keep_partitions+ (optional): also copy the individual partition images
+ of all devices to +$(BINARIES_DIR)+. Settings from the device sections
+ or the partition sections take precedence over this one.
+** +yes+: copy the individual partition images
+** +no+ (the default): do not copy individual partition images
+
+The devices and partitions sections
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The devices and partitions sections define, for each device or partition,
+the content of that device or partition.
+
+For each device listed in the +[global]+ section, there must be a
+corresponding section named after that device.
+
+For each partition listed in a device section, there must be a corresponding
+section named after that partition.
+
+Properties for the device section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +type+ (mandatory): the type of content for that device or partition
+** +boot+: the device contains one or more partitions, and
+ _may_ serve as a boot device
+
+* +keep_partitions+ (optional): also copy the individual partition images
+ for this device to +$(BINARIES_DIR)+. Settings from the partition
+ sections take precedence over this one.
+** +yes+: copy the individual partition images
+** +no+ (the default): do not copy individual partition images
+
+Properties for the partition section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +type+ (mandatory): the type of content for that device or partition
+** +fs+: the partition contains a filesystem
+
+* +size+: the size of that partition, in bytes
+
+* +keep+ (optional): copy this partition image to +$(BINARIES_DIR)+
+** +yes+: copy this partition image
+** +no+ (the default): do not copy this partition image
+
+Properties for +type=boot+
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +boot_type+ (mandatory): the partitioning scheme to use on this device
+** +mbr+: use an https://en.wikipedia.org/wiki/Master_boot_record[MBR]
+ partitioning scheme
+** +gpt+: use a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
+ partitioning scheme
+
+* +partitions+ (mandatory): the comma-separated list of partition(s) on
+ this device; no two partitions may have the same name, even if they
+ reside on different devices; partitions names shall match this regexp:
+ `^[[:alpha:]][[:alnum:]-_]*$` (_ie._ starts with a letter, followed by
+ zero or more alpha-numeric character or a dash or an underscore)
+
+* +partalign+ (optional): the alignment of partitions, in bytes; defaults
+ to an alignment of one, which means no alignment; depending on the
+ +boot_type+, some restrictions may apply, and are documented for each
+ +boot_type+
+
+Properties for +boot_type=mbr+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +mbr_bootcode+ (optional): the bootcode to use, as a path to the file
+ containing the bootcode image, relative to the +$(BINARIES_DIR)+
+ directory; defaults to no bootcode (eg. filled with zeroes)
+
+* +partalign+: must be a multiple of 512
+
+Properties for +boot_type=gpt+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +partalign+: must be a multiple of 512
+
+********
+Currently, only 512-byte sectors are supported. 4k sectors are not.
+********
+
+Properties for partitions whose containing device is +boot_type=mbr+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +mbr_type+ (mandatory): the partition
+ https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs[type]
+
+Properties for +type=fs+
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +fs_type+ (mandatory): the type of filesystem to generate
+** +ext+: generate an extended filesystem (ext2, ext3, ext4)
+** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
+
+* +fs_label+ (optional): the label to assign to this filesystem, if that
+ filesystem supports a label
+
+* +fs_files_0+, +fs_files_1+, +fs_files_N+ (optional): the list of files,
+ relative to $(BINARIES_DIR), to store in the filesystem. These entries
+ must be indexed starting from 0, and must be sequential: the first
+ missing entry ends the list
+
+* +fs_root+ (optional): the mountpoint of the filesystem
+
+* +fs_vfstype+ (optional): the type of filesystem to use when calling
+ +mount+, if different from +fs_type+
+
+* +fs_mntopts+ (optional): the mount options; defaults to +defaults+
+
+_Note_: valid use-cases for +fs_root+ and +fs_files_N+:
+
+* if only +fs_root+ is specified (and no +fs_files_N+): the filesystem
+ content is made exclusively from +$(TARGET_DIR)/$(fs_root)+, and the
+ filesystem is mounted at runtime
+
+* if both +fs_root+ and at least one +fs_files_N+ are specified: the
+ filesystem content is made exclusively from the +fs_files_N+ entries,
+ and mounted at runtime. +$(TARGET_DIR)/$(fs_root)+ must be empty
+
+* if at least one +fs_files_N+ is specified, and +fs_root+ is not: the
+ filesystem content is made exclusively from the +fs_files_N+ entries,
+ and the filesystem is not mounted at runtime
+
+* if neither +fs_root+ nor +fs_files_N+ is specified: this is an error
+
+Properties for +fs_type=ext+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +ext_gen+ (mandatory): the generation of extended filesystem to generate
+** +2+, +3+, +4+: for an ext2, ext3 or ext4 filesystem
+
+* +ext_rev+ (mandatory): the revision of the extended filesystem
+** +0+ (ext2 only): generate a revision 0 extended filesystem filesystem
+** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
+ filesystem
+
+Properties for +fs_type=vfat+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +vfat_size+ (optional): the VFAT-size of the filesystem
+** +12+, +16+, +32+: generate a FAT12, FAT16, or FAT32
+
+Generation of the filesystems
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The filesystems are generated in an order such that a filesystem that is
+mounted as a sub-directory of another filesystem is generated first.
+
+A filesystem is filled with the content of the directory corresponding to
+its mountpoint, and then that directory is emptied before continuing to the
+next filesystem. That way, the mountpoints are empty before the filesystem
+that contains them are generated.
+
+Finally, an entry in added in +/etc/fstab+ for each generated filesystem, so
+they are mounted at boot time.
+
+Examples
+~~~~~~~~
+
+.Simplest partition table layout description
+====
+----
+[global]
+extract=tar
+devices=sda
+
+[sda]
+type=boot
+boot_type=mbr
+partitions=root
+partalign=1048576
+
+[root]
+type=fs
+fs_type=ext
+fs_vfstype=ext4
+fs_root=/
+ext_gen=4
+ext_rev=1
+----
+
+The +partition table layout description+ above defines a single device
++sda+. That device contains a single partition, +root+, with an ext4
+filesystem, which is filled with the whole content of the +rootfs.tar+,
+and is mounted on +/+.
+====
+
+.More complex table layout description
+====
+----
+[global]
+extract=tar
+devices=mmcblk0,sda
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1024*1024))
+
+[sda]
+type=boot
+boot_type=mbr
+partitions=data
+partalign=4096
+
+[boot]
+type=fs
+mbr_type=$((0xC))
+size=$((16*1048576))
+fs_type=vfat
+fs_mntopts=ro
+fs_label=BOOT
+fs_root=/boot
+vfat_size=32
+
+[root]
+type=fs
+mbr_type=$((0x83))
+size=268435456
+fs_type=ext
+fs_vfstype=ext4
+fs_mntopts=discard,delalloc
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[data]
+type=fs
+mbr_type=$((0x83))
+size=$((4*1024*1048576))
+fs_type=ext
+fs_vfstype=ext2
+fs_root=/data
+fs_label=DATA
+ext_gen=2
+ext_rev=1
+----
+====
+
+The example above defines two devices, +mmcblk0+ and +sda+.
+
+The +mmcblk0+ device contains two partitions, +boot+ and +root+; partitions
+are aligned on a 1MiB boundary. The +sda+ device contains a single partition,
++data+, aligned on a 4KiB boundary.
+
+The +boot+ partition is a 16MiB FAT32 filesystem filled with the content
+of, and mounted on, +/boot+, and with label +BOOT+.
+
+The +data+ partition is a 4GiB ext2r1 filesystem filled with the content
+of, and mounted on, +/data+, and with label +DATA+.
+
+The +root+ partition is a 256MiB ext4 filesystem filled the the rest of,
+and mounted on, +/+, and with label +ROOT+.
diff --git a/fs/Config.in b/fs/Config.in
index da4c5ff..44e04f7 100644
--- a/fs/Config.in
+++ b/fs/Config.in
@@ -3,6 +3,7 @@ menu "Filesystem images"
source "fs/cloop/Config.in"
source "fs/cpio/Config.in"
source "fs/cramfs/Config.in"
+source "fs/custom/Config.in"
source "fs/ext2/Config.in"
source "fs/initramfs/Config.in"
source "fs/iso9660/Config.in"
diff --git a/fs/custom/Config.in b/fs/custom/Config.in
new file mode 100644
index 0000000..e5a8ee7
--- /dev/null
+++ b/fs/custom/Config.in
@@ -0,0 +1,16 @@
+config BR2_TARGET_ROOTFS_CUSTOM
+ bool "Custom partition table layout"
+ select BR2_TARGET_ROOTFS_TAR
+
+config BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE
+ string "path to the custom partition table layout description"
+ depends on BR2_TARGET_ROOTFS_CUSTOM
+ help
+ Enter the path to a partition-table for your device.
+
+ This will allow Buildroot to generate a more complex target
+ image, which may consist of more than one filesystem on more
+ than one partition.
+
+ See docs/manual/bla-bla on how to construct such a partition
+ table.
diff --git a/fs/custom/boot/gpt b/fs/custom/boot/gpt
new file mode 100644
index 0000000..f978524
--- /dev/null
+++ b/fs/custom/boot/gpt
@@ -0,0 +1,129 @@
+# Build a complete GPT-based image
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( parted )
+
+#-----------------------------------------------------------------------------
+# For a GPT-based partitionning, we need to compute the complete
+# image size before we can attempt to generate the partition table.
+# Then, we need to add the size of the GPT itself, plus that of its
+# backup copy, plus the protective MBR.
+# The size of the GPT itself depends on the sector size, and the
+# number of partitions in the GPT. Sectors can be either 512-byte
+# or 4096-byte large; The numbers of partitions is unlimited, but
+# it is suggested there is space to store at least 128 of them; a
+# partition description is 128-byte large.
+#
+# https://en.wikipedia.org/wiki/GUID_Partition_Table
+#
+# So, here's what we do:
+# - consider 512-byte sectors (since GPT on 4k sectors is not well
+# documented)
+# - consider at least 128 partitions; if the layout defines more than
+# that, we need to round that number up to the smallest multiple of
+# 4 (since there are 4 partition descriptions in a 512-byte sector)
+# - generate an empty, sparse file that is big enough to store the MBR,
+# the two GPT copies, and the aligned partitions.
+# - dump each partition in turn in their final location in that file
+# - generate a parted script that creates the partition table in that
+# file
+#
+# Simg = 512 + 2*(Sgpt) + ?( aligned(Spart,512) )
+# Sgpt = 512 + Nent*128
+#
+# Where:
+# Simg : size of the image
+# Sgpt : size of one GPT
+# Spart : size of each partition
+# Nent : number of partition entries
+# aligned() : the alignment function
+#
+# Sicne 4k-large sectors are not really explained on Wikipedia, we can
+# add this later on.
+do_image() {
+ # ${1} is fs_root, irrelevant here
+ local img="${2}"
+
+ # How many partitions do we have?
+ nb_parts=0
+ for part in ${partitions[${dev}]//,/ }; do
+ nb_parts=$((nb_parts+1))
+ done
+
+ # How many partition entries do we need?
+ nb_entries=$((4*((nb_parts+3)/4)))
+ nb_entries=$((nb_entries<128?128:nb_entries))
+
+ # The size of a single GPT
+ gpt_size=$((512+(128*nb_entries)))
+
+ # Offset of the first partition
+ begin=$(align_val $((512+$(align_val ${gpt_size} 512))) ${partalign} )
+
+ # Initialise our image file
+ dd if=/dev/zero of="${img}" \
+ bs=1 seek=${begin} count=0 \
+ conv=sparse 2>/dev/null
+
+ # Compute the space required to store all partitions
+ # and store them in the image file
+ size_parts=0
+ _offset=${begin}
+ i=1
+ debug "adding partions descriptions\n"
+ for part in ${partitions[${dev}]//,/ }; do
+ debug " %s\n" "${part}"
+ _part_img="${tmp_dir}/${dev}.${part}.img"
+ _size=$( align_val $( stat -c '%s' "${_part_img}" ) 512 )
+ part_offset+=( ${_offset} )
+ _attr="${values["${part}:gpt_attr"]}"
+ _label="${values["${part}:gpt_label"]}"
+
+ # If the partition has no label, use the filesystem label
+ if [ -z "${_label}" ]; then
+ _label="${values["${part}:fs_label"]}"
+ fi
+ if [ -z "${_label}" ]; then
+ _label="data"
+ fi
+
+ debug " start=%s\n" "${_offset}"
+ debug " size =%s\n" "${_size}"
+ debug " end =%s\n" "$((_offset+_size-1))"
+
+ dd if="${_part_img}" of="${img}" \
+ bs=512 seek=$((_offset/512)) \
+ conv=notrunc,sparse 2>/dev/null
+
+ parted_script+=( mkpart "${_label}" \
+ ${_offset} \
+ $((_offset+_size-1)) \
+ )
+ if [ -n "${_attr}" ]; then
+ for attr in "${_attr//,/ }"; do
+ parted_script+=( set ${i} ${attr} on )
+ done
+ fi
+
+ size_parts=$((size_parts+_size))
+ _offset=$((_offset+_size))
+ i=$((i+1))
+ done
+
+ # Terminate our image file
+ img_size=$(align_val $(( begin + size_parts + gpt_size )) 512)
+ debug "begin =%s\n" ${begin}
+ debug "nb_entry=%s\n" ${nb_entries}
+ debug "gpt_size=%s\n" ${gpt_size}
+ debug "img_size=%s\n" ${img_size}
+ dd if=/dev/zero of="${img}" \
+ bs=1 seek=${img_size} \
+ count=0 conv=sparse 2>/dev/null
+
+ for i in parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"; do
+ debug "--> '%s'\n" "${i}"
+ done
+ parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"
+}
+
+# vim: ft=sh
diff --git a/fs/custom/boot/mbr b/fs/custom/boot/mbr
new file mode 100644
index 0000000..667feed
--- /dev/null
+++ b/fs/custom/boot/mbr
@@ -0,0 +1,64 @@
+# Build a complete MBR-based image
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( genpart )
+
+#-----------------------------------------------------------------------------
+do_image() {
+ # ${1} is fs_root, irrelevant here
+ local img="${2}"
+ local i begin part part_img size type _begin _size
+ local -a part_offset part_file bootcode
+
+ # Fill-in the boot record
+ bootcode="${values["${dev}:mbr_bootcode"]}"
+ if [ -n "${bootcode}" ]; then
+ bootcode="${BINARIES_DIR}/${bootcode}"
+ else
+ bootcode="/dev/zero"
+ fi
+ debug "adding bootcode '%s'\n" "${bootcode}"
+ dd if="${bootcode}" of="${img}" bs=$((0x1be)) count=1 2>/dev/null
+
+ # Generate partition entries
+ i=0
+ begin=${partalign}
+ debug "adding partitions descriptors\n"
+ for part in ${partitions[${dev}]//,/ }; do
+ debug " %s\n" "${part}"
+ part_offset+=( ${begin} )
+ part_img="${tmp_dir}/${dev}.${part}.img"
+ part_file+=( "${part_img}" )
+ size=$( align_val $( stat -c '%s' "${part_img}" ) 512 )
+ type="${values["${part}:mbr_type"]}"
+ # LBA is exressed in a number of 512-byte blocks
+ # and genparts only deals with LBA
+ _begin=$((begin/512)) # begin is already 512-byte aligned
+ _size=$((size/512)) # size is already 512-byte aligned
+ debug " start=%s (LBA %s)\n" "${begin}" "${_begin}"
+ debug " size =%s (LBA %s)\n" "${size}" "${_size}"
+ debug " type =%s\n" "${type}"
+ genpart -b ${_begin} -s ${_size} -t ${type} >>"${img}"
+ begin=$( align_val $((begin+size)) ${partalign} )
+ i=$((i+1))
+ done
+ nb_parts=${i}
+ # Generate entries for empty partitions
+ for(( ; i<4; i++ )); do
+ debug " (empty)\n"
+ genpart -t 0 >>"${img}"
+ done
+ # Dump the boot signature
+ printf "\x55\xaa" >>"${img}"
+
+ for(( i=0; i<nb_parts; i++ )); do
+ part_img="${part_file[${i}]}"
+ offset=${part_offset[${i}]}
+ _offset=$(( offset/512 )) # offset is already 512-byte aligned
+ dd if="${part_img}" of="${img}" \
+ bs=512 seek=${_offset} \
+ conv=notrunc,sparse 2>/dev/null
+ done
+}
+
+# vim: ft=sh
diff --git a/fs/custom/boot/pre-post b/fs/custom/boot/pre-post
new file mode 100644
index 0000000..af4bcf5
--- /dev/null
+++ b/fs/custom/boot/pre-post
@@ -0,0 +1,14 @@
+#-----------------------------------------------------------------------------
+# No dependencies
+
+#-----------------------------------------------------------------------------
+do_image_pre() {
+ :
+}
+
+#-----------------------------------------------------------------------------
+do_image_post() {
+ :
+}
+
+#vim: set ft=sh
diff --git a/fs/custom/custom.mk b/fs/custom/custom.mk
new file mode 100644
index 0000000..ca53e0b
--- /dev/null
+++ b/fs/custom/custom.mk
@@ -0,0 +1,25 @@
+################################################################################
+#
+# custom partitioning
+#
+################################################################################
+
+# rootfs-custom uses rootfs.tar as the source to generate
+# the resulting image(s), so we need to build it first.
+ROOTFS_CUSTOM_DEPENDENCIES += rootfs-tar
+
+# If we are not selected, we won't have a partition table, so genimages
+# will complain (both on stdout and return code), so it will faill, so
+# we should not get our dependencies
+ifeq ($(BR2_TARGET_ROOTFS_CUSTOM),y)
+ROOTFS_CUSTOM_DEPENDENCIES += \
+ $(patsubst %,host-%,$(shell $(USER_HOOKS_EXTRA_ENV) fs/custom/genimages --show-depends \
+ '$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'))
+endif
+
+define ROOTFS_CUSTOM_CMD
+ $(USER_HOOKS_EXTRA_ENV) fs/custom/genimages \
+ '$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
+endef
+
+$(eval $(call ROOTFS_TARGET,custom))
diff --git a/fs/custom/fs/ext b/fs/custom/fs/ext
new file mode 100644
index 0000000..a184969
--- /dev/null
+++ b/fs/custom/fs/ext
@@ -0,0 +1,26 @@
+# Create an extended file system
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( e2fsprogs genext2fs )
+
+#-----------------------------------------------------------------------------
+do_image() {
+ local root_dir="${1}"
+ local img="${2}"
+ local -a fs_opts
+ local gen rev
+
+ fs_opts+=( -z )
+ fs_opts+=( -d "${root_dir}" )
+ [ -z "${size}" ] || fs_opts+=( -b $((size/1024)) )
+ [ -n "${ext_gen}" ] || ext_gen=2
+ [ -n "${ext_rev}" ] || ext_rev=1
+
+ # Remember, we're running from Buildroot's TOP_DIR
+ GEN=${ext_gen} REV=${ext_rev} \
+ ./fs/ext2/genext2fs.sh "${fs_opts[@]}" "${img}" >/dev/null
+
+ [ -z "${fs_label}" ] || tune2fs -L "${fs_label}" "${img}" >/dev/null
+}
+
+# vim: ft=sh
diff --git a/fs/custom/fs/pre-post b/fs/custom/fs/pre-post
new file mode 100644
index 0000000..d931437
--- /dev/null
+++ b/fs/custom/fs/pre-post
@@ -0,0 +1,72 @@
+#-----------------------------------------------------------------------------
+# No additional dependencies
+
+#-----------------------------------------------------------------------------
+do_image_pre() {
+ local i file
+
+ # if fs_root_dir is not specified, we have to create one
+ # It *does* override the caller's fs_root_dir value, but
+ # that's on purpose
+ # If fs_root_dir is specified, and we have at least fs_files_0,
+ # then fs_root_dir/ must be enpty
+ if [ -z "${fs_root_dir}" ]; then
+ if [ -n "${values["${part}:fs_files_0"]}" ]; then
+ error "%s: no fs_root specified, and no fs_files_0\n" "${part}"
+ fi
+ fs_root_dir="$( mktemp -d "${tmp_dir}/XXXXXX" )"
+ else
+ if [ -n "${values["${part}:fs_files_0"]}" \
+ -a $( ls -1A "${fs_root_dir}" 2>/dev/null \
+ |wc -l \
+ ) -ne 0 ]; then
+ error "%s: %s is not empty, but fs_files_0 is specified\n" \
+ "${part}" "${fs_root_dir#${fs_root}}"
+ fi
+ fi
+
+ i=0
+ while true; do
+ file="${values["${part}:fs_files_${i}"]}"
+ [ -n "${file}" ] || break
+ debug "%s: adding fs_files_%d %s\n" "${part}" ${i} "${file}"
+ install -D "${BINARIES_DIR}/${file}" "${fs_root_dir}/${file##*/}"
+ i=$((i+1))
+ done
+}
+
+#-----------------------------------------------------------------------------
+do_image_post() {
+ local rootfs_dir="${1}"
+ local fs_root="${2}"
+ local img_file="${3}"
+ local part="${4}"
+ local dev mntops vfstype fs_root_esc
+
+ subname+="[post-image]"
+
+ # Empty the partition's mountpoint
+ find "${fs_root_dir}" -maxdepth 1 \! -path "${fs_root_dir}" -exec rm -rf {} +
+
+ # Add entry in fstab, but not if this is '/'
+ # Don't add either if rootfs was not extracted
+ if [ "${fs_root}" = "/" -o -z "${fs_root}" \
+ -o -z "${values["global:extract"]}" ]; then
+ return 0
+ fi
+ fs_root_esc="$( sed -r -e 's:/:\\/:g;' <<<"${fs_root}" )"
+ sed -r -i -e "/[^[:space:]]+[[:space:]]+${fs_root_esc}[[:space:]]/d" \
+ "${rootfs_dir}/etc/fstab"
+ dev="$( get_part_dev_node "${part}" )"
+ vfstype="${fs_vfstype:-${fs_type}}"
+ mntops="${fs_mntops:-defaults}"
+ printf "/dev/%s %s %s %s 0 0\n" \
+ "${dev}" "${fs_root}" \
+ "${vfstype}" "${mntops}" \
+ >>"${rootfs_dir}/etc/fstab"
+
+ subname="${subname%\[post-image\]}"
+}
+
+#-----------------------------------------------------------------------------
+# vim: ft=sh
diff --git a/fs/custom/fs/vfat b/fs/custom/fs/vfat
new file mode 100644
index 0000000..5fc243d
--- /dev/null
+++ b/fs/custom/fs/vfat
@@ -0,0 +1,21 @@
+# Create a VFAT file system
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( dosfstools mtools )
+
+#-----------------------------------------------------------------------------
+do_image() {
+ local root_dir="${1}"
+ local img="${2}"
+ local -a fs_opts
+
+ dd if=/dev/zero of="${img}" bs=${size} count=0 seek=1 2>/dev/null
+
+ [ -z "${vfat_size}" ] || fs_opts+=( -F ${vfat_size} )
+ [ -z "${fs_label}" ] || fs_opts+=( -n "${fs_label}" )
+ mkfs.vfat "${fs_opts[@]}" "${img}" >/dev/null
+
+ mcopy -i "${img}" "${root_dir}/"* '::'
+}
+
+# vim: ft=sh
diff --git a/fs/custom/genimages b/fs/custom/genimages
new file mode 100755
index 0000000..204f13b
--- /dev/null
+++ b/fs/custom/genimages
@@ -0,0 +1,342 @@
+#!/bin/bash
+
+#-----------------------------------------------------------------------------
+main() {
+ local part_table="${1}"
+ local tmp_dir
+ local rootfs_dir
+ local -a devices
+ local extract
+ local cur_section
+ local -a sections devices partitions
+ local -A variables values partdevs
+ local sec dev part var val
+ local secs devs parts vars vals
+ local has_global_section
+
+ # We need bash 4 or above for associative arrays
+ if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
+ error "bash 4 or above is needed\n"
+ fi
+
+ if [ "${part_table}" = "--show-depends" ]; then
+ SHOW_DEPENDS=1
+ part_table="${2}"
+ trace() { :; } # Be silent, we just want the dependencies...
+ else
+ SHOW_DEPENDS=0
+ fi
+
+ if [ ! -f "${part_table}" ]; then
+ error "'%s': no such file\n" "${part_table}"
+ exit 1
+ fi
+
+ export PATH="${HOST_DIR}/usr/bin:${HOST_DIR}/usr/sbin:${PATH}"
+
+ # Parse all the sections in one go, we'll sort
+ # all the mess afterwards...
+ debug "parsing partitions descriptions file '%s'\n" \
+ "${part_table}"
+ parse_ini "${part_table}"
+
+ # The 'global' section is mandatory
+ has_global_section=0
+ for s in "${sections[@]}"; do
+ if [ "${s}" = "global" ]; then
+ has_global_section=1
+ break
+ fi
+ done
+ if [ ${has_global_section} -eq 0 ]; then
+ error "no global section defined\n"
+ fi
+
+ # Create lists of devices, partitions, and partition:device pairs.
+ debug "creating intermediate lists\n"
+ devices=( ${values["global:devices"]//,/ } )
+ for dev in "${devices[@]}"; do
+ # Sanity check first: all devices must have a corresponding
+ # section, which means they should have a type
+ if [ -z "${values["${dev}:type"]}" ]; then
+ error "device '%s' has no type (no section?)\n" "${dev}"
+ fi
+ partitions+=( ${values["${dev}:partitions"]//,/ } )
+ for part in ${values["${dev}:partitions"]//,/ }; do
+ # Sanity check first: all partitions must have a corresponding
+ # section, which means they should have a type
+ if [ -z "${values["${part}:type"]}" ]; then
+ error "partition '%s:%s' has no type (no section?)\n" \
+ "${dev}" "${part}"
+ fi
+ partdevs+=( ["${part}"]="${dev}" )
+ done
+ done
+
+ # Now, we must order the partitions so that their mountpoint
+ # is empty by the time we build the upper-level partition.
+ # For example, given this layout of mountpoints:
+ # /
+ # /usr
+ # /usr/var
+ # We must ensure /usr/var is empty at the time we create the /usr
+ # filesystem image; and similarly, we must ensure /usr is empty by
+ # the time we create the / filesystem image
+ # So, a simple reverse alphabetical sort will do the trick
+ debug "sorting partitions\n"
+ sorted_parts=( $(
+ for part in "${partitions[@]}"; do
+ # Partitions that are not mounted can be generated
+ # in any order
+ if [ -n "${values["${part}:fs_root"]}" ]; then
+ printf "%s:%s\n" "${part}" "${values["${part}:fs_root"]}"
+ else
+ printf "%s\n" "${part}"
+ fi
+ done \
+ |sort -t: -k2 -r \
+ |sed -r -e 's/:[^:]+$//;'
+ ) )
+
+ # We do not want to create anything if we only want the dependencies
+ if [ ${SHOW_DEPENDS} -eq 0 ]; then
+ tmp_dir="${BUILD_DIR}/genimages.tmp"
+ rootfs_dir="${tmp_dir}/rootfs"
+ # Since we don't remove it in case of error (to be able to inspect its
+ # content), we must remove it now (a previous run may have left it).
+ rm -rf "${tmp_dir}"
+ mkdir -p "${rootfs_dir}"
+
+ case "${values["global:extract"]}" in
+ tar)
+ # We must be root for the extract to work correctly
+ # (since it may have /dev nodes, or some files may
+ # belong to different users...)
+ if [ $(id -u) -ne 0 ]; then
+ error "error: not root\n"
+ fi
+ trace "extracting rootfs.tar\n"
+ tar xf "${BINARIES_DIR}/rootfs.tar" -C "${rootfs_dir}"
+ ;;
+ *) error "unknown extract method '%s'\n" "${extract:-(none)}"
+ ;;
+ esac
+ fi # ! SHOW_DEPENDS
+
+ # Render all partition images
+ for part in "${sorted_parts[@]}"; do
+ trace "preparing filesystem for partition '%s'\n" "${part}"
+ render_img "${rootfs_dir}" "${part}" \
+ "${tmp_dir}/${partdevs["${part}"]}.${part}.img"
+ done
+
+ # Aggregate all devices images
+ for dev in "${devices[@]}"; do
+ trace "assembling partitions in device '%s'\n" "${dev}"
+ render_img "${rootfs_dir}" "${dev}" "${tmp_dir}/${dev}.img"
+ done
+
+ # If we need the dependencies, we can stop right now
+ if [ ${SHOW_DEPENDS} -eq 1 ]; then
+ return 0
+ fi
+
+ # Copy all partitions and devices images to the image dir
+ if [ "${values["global:keep_partitions"]}" = "yes" ]; then
+ for part in "${sorted_parts[@]}"; do
+ debug "copying partition '%s' to image dir\n" "${part}"
+ dd if="${tmp_dir}/${partdevs["${part}"]}.${part}.img" \
+ of="${BINARIES_DIR}/$( get_part_dev_node "${part}" ).img" \
+ bs=4096 conv=sparse 2>/dev/null
+ done
+ fi
+ for dev in "${devices[@]}"; do
+ debug "copying device '%s' to image dir\n" "${dev}"
+ dd if="${tmp_dir}/${dev}.img" \
+ of="${BINARIES_DIR}/${dev}.img" \
+ bs=4096 conv=sparse 2>/dev/null
+ done
+
+ [ -n "${DEBUG}" ] || rm -rf "${tmp_dir}"
+}
+
+#-----------------------------------------------------------------------------
+render_img() {
+ local rootfs_dir="${1}"
+ local img="${2}"
+ local img_file="${3}"
+ local type sub_type fs_root_dir
+
+ type="${values["${img}:type"]}"
+ sub_type="${values["${img}:${type}_type"]}"
+
+ # Sanity checks
+ [ -n "${type}" ] || error "'%s': unspecified type\n" "${img}"
+ if [ ! -d "fs/custom/${type}" ]; then
+ error "'%s': unsupported type '%s'\n" "${img}" "${type}"
+ fi
+ [ -n "${sub_type}" ] || error "'%s': unspecified %s_type\n" "${img}" "${type}"
+ if [ ! -f "fs/custom/${type}/${sub_type}" ]; then
+ error "'%s': unknown %s_type '%s'\n" "${img}" "${type}" "${sub_type}"
+ fi
+
+ # Need to call the renderer in a subshell so that its definitions
+ # do not pollute our environment
+ subname="${sub_type}"
+ (
+ trap 'exit $?' ERR
+
+ declare -a DEPENDS
+
+ for var in ${variables["${img}"]//,/ }; do
+ eval "${var}=\"${values["${img}:${var}"]}\""
+ done
+ fs_root_dir="${rootfs_dir}${fs_root}"
+ . "fs/custom/${type}/pre-post"
+ . "fs/custom/${type}/${sub_type}"
+ if [ ${SHOW_DEPENDS} -eq 1 ]; then
+ for dep in "${DEPENDS[@]}"; do
+ printf "%s\n" "${dep}"
+ done
+ else
+ do_image_pre "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
+ do_image "${fs_root_dir}" "${img_file}"
+ do_image_post "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
+ fi
+ )
+ ret=${?}
+ [ ${ret} -eq 0 ] || exit ${ret}
+ subname=""
+}
+
+#------------------------------------------------------------------------------
+# Parse a .ini file
+# $1: .ini file to parse
+# The caller should define the following variables:
+# sections : array
+# variables : associative array
+# values : associative array
+# parse_ini() will fill those variables with:
+# sections : the list of sections, one section per array index
+# variables : the comma-separated list of varibles for a section,
+# indexed by the name of the section
+# values : the value of a variable in a section, indexed by the
+# 'section:variable' tuple
+# Eg.:
+# sections=( [0]='section-0' [1]='section-1' )
+# variables=( ['section-0']='var-0,var-1' ['section-1']='var-10,var-11' [...] )
+# values=( ['section-0:var-0']='value-0-0' ['section-1:var-10']='value-1-10' [...] )
+parse_ini() {
+ local ini_file="${1}"
+ local line var val
+ local cur_section
+ local var_sep
+
+ while read line; do
+ line="$( sed -r -e 's/[[:space:]]*#.*$//; //d;' <<<"${line}" )"
+
+ # Detect start of global section, skip anything else
+ case "${line}" in
+ "") continue;;
+ '['*']')
+ cur_section="$( sed -r -e 's/[][]//g;' <<<"${line}" )"
+ debug " entering section '%s'\n" "${cur_section}"
+ sections+=( "${cur_section}" )
+ continue
+ ;;
+ ?*=*) ;;
+ *) error "malformed entry '%s'\n" "${line}";;
+ esac
+
+ var="${line%%=*}"
+ eval val="${line#*=}"
+ debug " adding '%s'='%s'\n" "${var}" "${val}"
+ var_sep="${variables["${cur_section}"]+,}"
+ variables+=( ["${cur_section}"]="${var_sep}${var}" )
+ values+=( ["${cur_section}:${var}"]="${val}" )
+ done <"${ini_file}"
+}
+
+#-----------------------------------------------------------------------------
+get_part_dev_node() {
+ local part="${1}"
+ local dev
+ local i c p
+
+ dev="${partdevs["${part}"]}"
+ i="${values["${dev}:partstart"]:-1}"
+
+ # If device node ends with a number, partitions are denoted
+ # with a 'p' before the partition number, eg.:
+ # /dev/mmcblk0 --> /dev/mmcblk0p1
+ # /dev/sda --> /dev/sda1
+ case "${dev#${dev%?}}" in
+ [0-9]) c="p";;
+ *) c="";;
+ esac
+
+ for p in ${values["${dev}:partitions"]//,/ }; do
+ if [ "${p}" = "${part}" ]; then
+ printf "%s%s%d" "${dev}" "${c}" ${i}
+ return 0
+ fi
+ i=$((i+1))
+ done
+
+ error "'%s': partition not found. WTF?\n" "${part}"
+}
+
+#------------------------------------------------------------------------------
+align_val() {
+ local val="${1}"
+ local align="${2}"
+ local aligned
+
+ aligned=$(( ( (val+align-1) / align ) * align ))
+
+ printf "%d" ${aligned}
+}
+
+#------------------------------------------------------------------------------
+# Some trace functions
+_trace() {
+ local fmt="${1}"
+ shift
+
+ printf "%s" "${myname}"
+ if [ -n "${subname}" ]; then
+ printf "(%s)" "${subname}"
+ fi
+ printf ": ${fmt}" "${@}"
+}
+
+trace() {
+ _trace "${@}"
+}
+
+debug() { :; }
+if [ -n "${DEBUG}" ]; then
+ debug() {
+ _trace "${@}" >&2
+ }
+fi
+
+error() {
+ _trace "${@}" >&2
+ exit 1
+}
+
+on_error() {
+ local ret=${?}
+
+ error "unexpected error caught: %d\n" ${ret}
+}
+trap on_error ERR
+set -E -e
+
+#-----------------------------------------------------------------------------
+export myname="${0##*/}"
+
+main "${@}"
+
+# vim: ft=sh
--
1.8.1.2
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-08-29 21:24 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-29 11:27 [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images Jan Schmidt
2017-08-29 21:24 ` Yann E. MORIN
-- strict thread matches above, loose matches on Subject: below --
2014-01-03 17:19 [Buildroot] [PATCH 0/5 v4] SUBJECT Yann E. MORIN
2014-01-03 17:19 ` [Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images Yann E. MORIN
2014-01-04 16:38 ` Maxime Hadjinlian
2014-01-04 17:43 ` Yann E. MORIN
2014-01-04 17:52 ` Maxime Hadjinlian
2014-04-09 13:28 ` Andreas Naumann
2014-04-09 17:09 ` Yann E. MORIN
2014-04-10 13:53 ` Andreas Naumann
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.