[Buildroot] [PATCH] docs/manual: document filesystems

[Buildroot] [PATCH] docs/manual: document filesystems

[Yann E. MORIN]

We do expect that users be able to implement new filesystems and
submit them, or have local, custom and private filesystems in
their br2-external trees.

So, add documentation for filesystems, based on the format we have
for packages.

Reported-by: Carlos Santos <casantos@datacom.com.br>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Carlos Santos <casantos@datacom.com.br>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

---
Changes v1 -> v2:
  - credit Carlos for prompting this addition to the manual. :-)
  - fix the name of the _CMD macro (no trailing 'S')
---
 docs/manual/adding-filesystems.txt    | 119 ++++++++++++++++++++++++++++++++++
 docs/manual/adding-packages-hooks.txt |   7 +-
 docs/manual/adding-packages.txt       |   2 +
 3 files changed, 127 insertions(+), 1 deletion(-)
 create mode 100644 docs/manual/adding-filesystems.txt

diff --git a/docs/manual/adding-filesystems.txt b/docs/manual/adding-filesystems.txt
new file mode 100644
index 0000000000..4731526c0f
--- /dev/null
+++ b/docs/manual/adding-filesystems.txt
@@ -0,0 +1,119 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+=== Infrastructure for filesystems
+
+As for packages, Buildroot provides an infrastructure to build
+filesystems.
+
+The filesystem infrastructure follows a carefully-crafted sequence:
+
+1. Assemble an intermediate, internal tarball from the +target/+
+   directory:
+  * create users and groups,
+  * set the correct rights on files,
+  * create device nodes,
+  * create the intermediate tarball;
+2. Then for each filesystem:
+  * extract the intermediate tarball to a filesystem-specific
+    +$(TARGET_DIR)+,
+  * execute the filesystem pre-gen hooks (see below),
+  * optionally, touch everything to a specific date, for reproducibility,
+  * execute the filesystem commands,
+  * execute the filesystem post-gen hooks (see below),
+  * remove the filesystem-specific +$(TARGET_DIR)+;
+3. Remove the intermediate tarball.
+
+[[filesystem-tutorial]]
+==== +filesystem+ tutorial
+
+Let's look at an example filesytem. First and foremost, like for a
+package, create a directory named from your filesystem, e.g. +foo+.
+
+Then, create a +Config.in+ file describing the menuconfig entries
+for the filesystem. Be sure to have that +Config.in+ included from
++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree).
+For example:
+
+----
+01: config BR2_TARGET_ROOTFS_FOO
+02: 	bool "foo root filesystem"
+03: 	help
+04: 	  Build a foo root filesystem.
+05: 
+06: if BR2_TARGET_ROOTFS_FOO
+07: 
+08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR
+09: 	bool "bar format"
+10: 	help
+11: 	  Enable the bar format.
+12: 
+13: endif
+----
+
+Then, in the same directory, create a +foo.mk+ file describing how
+to actually assemble the filesystem image from the content of
++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the
+Makefile code (automatic for built-in filesystems, not for filesystems
+in a *br2-external* tree.) For example:
+
+----
+01: ################################################################################
+02: #
+03: # foo filesystem
+04: #
+05: ################################################################################
+06: 
+07: ROOTFS_FOO_DEPENDENCIES = host-foo-utils
+08: 
+09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y)
+10: ROOTFS_FOO_OPTS = --format=bar
+11: else
+12: ROOTFS_FOO_OPTS = --format=foo
+13: endif
+14: 
+15: define ROOTFS_FOO_CMD
+16: 	$(HOST_DIR)/bin/mkfs.foo \
+17: 		--root-dir=$(TARGET_DIR) \
+18: 		--option-foo --option-bar \
+19: 		$(ROOTFS_FOO_OPTS) \
+20: 		--output-file=$(@)
+21: endef
+22: 
+23: $(eval $(rootfs))
+----
+
+On line 7, we specify the dependencies on packages needed to build the
+filesystem. Usually, those dependencies are one or more host-packages
+that provide the filesystem generator.
+
+On lines 9 to 13, we define a conditional set of options, depending on
+the user's choice.
+
+On lines 15 to 21, we define the actual command that will generate the
+filesystem image.
+
+Finally, on line 23, we invoke the +rootfs+ macro that generates all the
+Makefile rules that actually allow the filesystem to be built.
+
+[[filesystem-reference]]
+==== +filesystem+ reference
+
+If one compares a package with a filesystem, one may notice that the
+variables for a filesystem start with the prefix +ROOTFS_+, when the
+variables for packages have no prefix.
+
+There is a single macro that a filesystem may set:
+
+* +ROOTFS_FOO_CMD+ contains the commands to execute to actually
+  generate the filesystem iamge from the content of the +$(TARGET_DIR)+
+  directory.
+
+Additionally, like for packages, there are a set of hooks; see
+xref:hooks[] for details.
+
+And here is a list of variables that may be used (but may *not* be set):
+
+* +$(TARGET_DIR)+ is the directory containing the +target/+ directory.
+
+* +$(@)+ is the filename where to store the filesystem image into.
diff --git a/docs/manual/adding-packages-hooks.txt b/docs/manual/adding-packages-hooks.txt
index 0ce79f8907..34d5801169 100644
--- a/docs/manual/adding-packages-hooks.txt
+++ b/docs/manual/adding-packages-hooks.txt
@@ -11,7 +11,7 @@ Most hooks aren't really useful for generic packages, since the +.mk+
 file already has full control over the actions performed in each step
 of the package construction.
 
-The following hook points are available:
+The following hook points are available for packages:
 
 * +LIBFOO_PRE_DOWNLOAD_HOOKS+
 * +LIBFOO_POST_DOWNLOAD_HOOKS+
@@ -46,6 +46,11 @@ The following hook points are available:
 * +LIBFOO_PRE_LEGAL_INFO_HOOKS+
 * +LIBFOO_POST_LEGAL_INFO_HOOKS+
 
+The following hook points are available for filesystems:
+
+* +ROOTFS_FOO_PRE_GEN_HOOKS+
+* +ROOTFS_FOO_POST_GEN_HOOKS+
+
 These variables are 'lists' of variable names containing actions to be
 performed at this hook point. This allows several hooks to be
 registered at a given hook point. Here is an example:
diff --git a/docs/manual/adding-packages.txt b/docs/manual/adding-packages.txt
index 4a4a17e879..83c93f3dde 100644
--- a/docs/manual/adding-packages.txt
+++ b/docs/manual/adding-packages.txt
@@ -47,6 +47,8 @@ include::adding-packages-asciidoc.txt[]
 
 include::adding-packages-linux-kernel-spec-infra.txt[]
 
+include::adding-filesystems.txt[]
+
 include::adding-packages-hooks.txt[]
 
 include::adding-packages-gettext.txt[]
-- 
2.14.1

[Buildroot] [PATCH] docs/manual: document filesystems

[Carlos Santos]

> From: "Yann Morin" <yann.morin.1998@free.fr>
> To: "buildroot" <buildroot@buildroot.org>
> Cc: "Yann Morin" <yann.morin.1998@free.fr>, "Thomas De Schampheleire" <patrickdepinguin@gmail.com>, "DATACOM"
> <casantos@datacom.com.br>, "Thomas Petazzoni" <thomas.petazzoni@bootlin.com>
> Sent: Saturday, June 9, 2018 6:46:47 AM
> Subject: [PATCH] docs/manual: document filesystems

[...]
> 
> diff --git a/docs/manual/adding-filesystems.txt
> b/docs/manual/adding-filesystems.txt
> new file mode 100644
> index 0000000000..4731526c0f
> --- /dev/null
> +++ b/docs/manual/adding-filesystems.txt
> @@ -0,0 +1,119 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +=== Infrastructure for filesystems

Shouldn't this be a separate chapter instead of a section of packages?
Filesystems are different from packages, regardless the syntactical
similarities in the configurations and recipe files. See also the
comment below about the hooks section.

> +
> +As for packages, Buildroot provides an infrastructure to build
> +filesystems.
> +
> +The filesystem infrastructure follows a carefully-crafted sequence:
> +
> +1. Assemble an intermediate, internal tarball from the +target/+
> +   directory:
> +  * create users and groups,
> +  * set the correct rights on files,
> +  * create device nodes,
> +  * create the intermediate tarball;
> +2. Then for each filesystem:
> +  * extract the intermediate tarball to a filesystem-specific
> +    +$(TARGET_DIR)+,
> +  * execute the filesystem pre-gen hooks (see below),
> +  * optionally, touch everything to a specific date, for reproducibility,
> +  * execute the filesystem commands,
> +  * execute the filesystem post-gen hooks (see below),
> +  * remove the filesystem-specific +$(TARGET_DIR)+;
> +3. Remove the intermediate tarball.
> +
> +[[filesystem-tutorial]]
> +==== +filesystem+ tutorial
> +
> +Let's look at an example filesytem. First and foremost, like for a
> +package, create a directory named from your filesystem, e.g. +foo+.
> +
> +Then, create a +Config.in+ file describing the menuconfig entries
> +for the filesystem. Be sure to have that +Config.in+ included from
> ++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree).
> +For example:
> +
> +----
> +01: config BR2_TARGET_ROOTFS_FOO
> +02: 	bool "foo root filesystem"
> +03: 	help
> +04: 	  Build a foo root filesystem.
> +05:
> +06: if BR2_TARGET_ROOTFS_FOO
> +07:
> +08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR
> +09: 	bool "bar format"
> +10: 	help
> +11: 	  Enable the bar format.
> +12:
> +13: endif
> +----
> +
> +Then, in the same directory, create a +foo.mk+ file describing how
> +to actually assemble the filesystem image from the content of
> ++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the
> +Makefile code (automatic for built-in filesystems, not for filesystems
> +in a *br2-external* tree.) For example:
> +
> +----
> +01:
> ################################################################################
> +02: #
> +03: # foo filesystem
> +04: #
> +05:
> ################################################################################
> +06:
> +07: ROOTFS_FOO_DEPENDENCIES = host-foo-utils
> +08:
> +09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y)
> +10: ROOTFS_FOO_OPTS = --format=bar
> +11: else
> +12: ROOTFS_FOO_OPTS = --format=foo
> +13: endif
> +14:
> +15: define ROOTFS_FOO_CMD
> +16: 	$(HOST_DIR)/bin/mkfs.foo \
> +17: 		--root-dir=$(TARGET_DIR) \
> +18: 		--option-foo --option-bar \
> +19: 		$(ROOTFS_FOO_OPTS) \
> +20: 		--output-file=$(@)
> +21: endef
> +22:
> +23: $(eval $(rootfs))
> +----
> +
> +On line 7, we specify the dependencies on packages needed to build the
> +filesystem. Usually, those dependencies are one or more host-packages
> +that provide the filesystem generator.
> +
> +On lines 9 to 13, we define a conditional set of options, depending on
> +the user's choice.
> +
> +On lines 15 to 21, we define the actual command that will generate the
> +filesystem image.
> +
> +Finally, on line 23, we invoke the +rootfs+ macro that generates all the
> +Makefile rules that actually allow the filesystem to be built.
> +
> +[[filesystem-reference]]
> +==== +filesystem+ reference
> +
> +If one compares a package with a filesystem, one may notice that the
> +variables for a filesystem start with the prefix +ROOTFS_+, when the
> +variables for packages have no prefix.
> +
> +There is a single macro that a filesystem may set:
> +
> +* +ROOTFS_FOO_CMD+ contains the commands to execute to actually
> +  generate the filesystem iamge from the content of the +$(TARGET_DIR)+
> +  directory.
> +
> +Additionally, like for packages, there are a set of hooks; see
> +xref:hooks[] for details.

There are two fundamental differences between packages and filesystem
images:

Filesystem images are always rebuilt and installed while packages are
built/installed once and rebuilt/re-installed only on explicit calls
to "make pkg-name-{reconfigure,rebuild,reinstall}".

Packages are supposed to install, remove or modify files in the global
$(TARGET_DIR) hierarchy. Genarating filesystem images do not modify the
contents of the global $(TARGET_DIR) and user-provided rootfs extensions
must not do this.

[Side note: we could add an optional "paranoid mode" config which would
trigger a checksum generation/checking right before/after the creation
of the rootfs images.] 

> +And here is a list of variables that may be used (but may *not* be set):

"must" not be set.

> +
> +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory.
> +
> +* +$(@)+ is the filename where to store the filesystem image into.
> diff --git a/docs/manual/adding-packages-hooks.txt
> b/docs/manual/adding-packages-hooks.txt
> index 0ce79f8907..34d5801169 100644
> --- a/docs/manual/adding-packages-hooks.txt
> +++ b/docs/manual/adding-packages-hooks.txt

Consider moving the hooks section to a separate chapter if you put the
filesystem text in a separate chapter too.

[...]

-- 
Carlos Santos (Casantos) - DATACOM, P&D
?Marched towards the enemy, spear upright, armed with the certainty
that only the ignorant can have.? ? Epitaph of a volunteer

[Buildroot] [PATCH] docs/manual: document filesystems

[Yann E. MORIN]

Carlos, All,

Thanks for the review! :-)

On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
> > From: "Yann Morin" <yann.morin.1998@free.fr>
[--SNIP--]
> > diff --git a/docs/manual/adding-filesystems.txt
> > b/docs/manual/adding-filesystems.txt
> > new file mode 100644
> > index 0000000000..4731526c0f
> > --- /dev/null
> > +++ b/docs/manual/adding-filesystems.txt
> > @@ -0,0 +1,119 @@
> > +// -*- mode:doc; -*-
> > +// vim: set syntax=asciidoc:
> > +
> > +=== Infrastructure for filesystems
> 
> Shouldn't this be a separate chapter instead of a section of packages?
> Filesystems are different from packages, regardless the syntactical
> similarities in the configurations and recipe files. See also the
> comment below about the hooks section.

I also wondered about that.

But I was a bit wary of creating another chapter just for filesystems.

And notice that in the package chapter, we also talk about the
'asciidoc-document' infra, which is already not a package infra.

I'd like input from others. Thomas, Peter, Arnout?

[--SNIP--]
> There are two fundamental differences between packages and filesystem
> images:
> 
> Filesystem images are always rebuilt and installed while packages are
> built/installed once and rebuilt/re-installed only on explicit calls
> to "make pkg-name-{reconfigure,rebuild,reinstall}".

ACK, good point. I'll add this to the first section, that explains the
infra.

> Packages are supposed to install, remove or modify files in the global
> $(TARGET_DIR) hierarchy.

Well, in the first part of this patch, I added a description of the
sequence, making it obvious (or so I hoped) that indeed filesystems were
acting on a transient copy:

    2.  Then for each filesystem:
      * extract the intermediate tarball to a filesystem-specific $(TARGET_DIR),
      [...]
      * remove the filesystem-specific $(TARGET_DIR);

> Genarating filesystem images do not modify the
> contents of the global $(TARGET_DIR) and user-provided rootfs extensions
> must not do this.

Why do you insist on modifying the global target/ directory from your
filesystem? If you do that, and build two or more filesystems [*], they
might each try to modify target/ is incompatible ways, and this is not
even safe, concurrency-wise (e.g. ony may try to archive the content of
/var while another is doing a rm -rf on it.)

The filesystem infra has been changed to provide each filesystem a copy
of the target/ directory, so that each filesystem can do their own
tweaks without impacting the others, without the need to be idempotent,
and allowing filesystems to be run in parallel.

I have still not seen a *compelling* reason to let filesystems muck in
the original target/ directory.

[*] Buildroot terminology, like BR2_TARGET_ROOTFS_SQUASHFS=y and
BR2_TARGET_ROOTFS_ISO9660=y at the same time.

> [Side note: we could add an optional "paranoid mode" config which would
> trigger a checksum generation/checking right before/after the creation
> of the rootfs images.] 

Not needed, since there is no reason to let them do.

> > +And here is a list of variables that may be used (but may *not* be set):
> "must" not be set.

https://www.merriam-webster.com/dictionary/may

4. : SHALL, MUST ? used in law where the sense, purpose, or policy
     requires this interpretation

(yes, I tend to use a pedant or/and archaic english. ;-])

> > +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory.
> > +
> > +* +$(@)+ is the filename where to store the filesystem image into.
> > diff --git a/docs/manual/adding-packages-hooks.txt
> > b/docs/manual/adding-packages-hooks.txt
> > index 0ce79f8907..34d5801169 100644
> > --- a/docs/manual/adding-packages-hooks.txt
> > +++ b/docs/manual/adding-packages-hooks.txt
> 
> Consider moving the hooks section to a separate chapter if you put the
> filesystem text in a separate chapter too.

If others agreee that the filesystem infra requires its own chapter,
then I will, yes.

Thanks again for the review! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] docs/manual: document filesystems

[Yann E. MORIN]

Carlos, Arnout, All,

On 2018-06-10 10:12 +0200, Yann E. MORIN spake thusly:
> On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
[--SNIP--]
> > Packages are supposed to install, remove or modify files in the global
> > $(TARGET_DIR) hierarchy.

I forgot to say that this will no longer be true when we (eventually)
introduce the top-level parallel build, because, as Arnout already said,
each package will install in its own, private copy of TARGET_DIR, not
only that, but also its private STAGING_DIR and HOST_DIR as well.

One thing that Arnout suggested, is to (mostly) get rid of TARGET_DIR,
in favour of a package-prefix FOO_TARGET_DIR (amd similarly for
filesystems).

However I still don't think that is a good idea, because that would
allow packages to easily find and write into another package's
TARGET_DIR. Using a single TARGET_DIR, instead, will make it harder
(but not impossible, agreed).

Ditto filesystems.

So I believe we still want to keep a context-specific TARGET_DIR. It is
easier to use, it is more stable, we can update it at will, and it makes
it harder for packages and filesystems to do weird stuff.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] docs/manual: document filesystems

[Thomas Petazzoni]

Hello,

On Sun, 10 Jun 2018 10:12:17 +0200, Yann E. MORIN wrote:

> I also wondered about that.
> 
> But I was a bit wary of creating another chapter just for filesystems.
> 
> And notice that in the package chapter, we also talk about the
> 'asciidoc-document' infra, which is already not a package infra.
> 
> I'd like input from others. Thomas, Peter, Arnout?

The section about packages is already very long, so I wouldn't mind
seeing other sections being created outside of it.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] docs/manual: document filesystems

[Carlos Santos]

> From: "Yann Morin" <yann.morin.1998@free.fr>
> To: "DATACOM" <casantos@datacom.com.br>
> Cc: "buildroot" <buildroot@buildroot.org>, "Thomas De Schampheleire" <patrickdepinguin@gmail.com>, "Thomas Petazzoni"
> <thomas.petazzoni@bootlin.com>, "Peter Korsgaard" <peter@korsgaard.com>, "Arnout Vandecappelle" <arnout@mind.be>
> Sent: Sunday, June 10, 2018 5:12:17 AM
> Subject: Re: [PATCH] docs/manual: document filesystems

> Carlos, All,
> 
> Thanks for the review! :-)
> 

[...]

>> Genarating filesystem images do not modify the
>> contents of the global $(TARGET_DIR) and user-provided rootfs extensions
>> must not do this.
> 
> Why do you insist on modifying the global target/ directory from your
> filesystem? If you do that, and build two or more filesystems [*], they
> might each try to modify target/ is incompatible ways, and this is not
> even safe, concurrency-wise (e.g. ony may try to archive the content of
> /var while another is doing a rm -rf on it.)

I never intended to modify the global $(TARGET_DIR).

-- 
Carlos Santos (Casantos) - DATACOM, P&D
?Marched towards the enemy, spear upright, armed with the certainty
that only the ignorant can have.? ? Epitaph of a volunteer

[Buildroot] [PATCH] docs/manual: document filesystems

[Yann E. MORIN]

Carlos, All,

On 2018-06-10 10:38 -0300, Carlos Santos spake thusly:
> > From: "Yann Morin" <yann.morin.1998@free.fr>
> >> Genarating filesystem images do not modify the
> >> contents of the global $(TARGET_DIR) and user-provided rootfs extensions
> >> must not do this.
[--SNIP--]
> > Why do you insist on modifying the global target/ directory from your
> > filesystem? If you do that, and build two or more filesystems [*], they
> > might each try to modify target/ is incompatible ways, and this is not
> > even safe, concurrency-wise (e.g. ony may try to archive the content of
> > /var while another is doing a rm -rf on it.)
> 
> I never intended to modify the global $(TARGET_DIR).

Sorry, I mis-interpreted your comment above, as "user-provided rootfs []
must not do this" to which I appended in my head "but that is exactly
what I want to do in my case."

I'm sorry, I should be more careful in my interpretations in the future.

So, I'll take that as a hint to further improve the section that
explains how filesystems are generated, to more carefully describe and
explain the sequence.

Thanks again for the feedback! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] docs/manual: document filesystems

[Arnout Vandecappelle]

On 10-06-18 10:35, Yann E. MORIN wrote:
> Carlos, Arnout, All,
> 
> On 2018-06-10 10:12 +0200, Yann E. MORIN spake thusly:
>> On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
> [--SNIP--]
>>> Packages are supposed to install, remove or modify files in the global
>>> $(TARGET_DIR) hierarchy.
> 
> I forgot to say that this will no longer be true when we (eventually)
> introduce the top-level parallel build, because, as Arnout already said,
> each package will install in its own, private copy of TARGET_DIR, not
> only that, but also its private STAGING_DIR and HOST_DIR as well.
> 
> One thing that Arnout suggested, is to (mostly) get rid of TARGET_DIR,
> in favour of a package-prefix FOO_TARGET_DIR (amd similarly for
> filesystems).
> 
> However I still don't think that is a good idea, because that would
> allow packages to easily find and write into another package's
> TARGET_DIR. Using a single TARGET_DIR, instead, will make it harder
> (but not impossible, agreed).

 How does the definition of TARGET_DIR have any influence on how fs/foo/foo.mk
uses ROOTFS_BAR_TARGET_DIR?


> Ditto filesystems.
> 
> So I believe we still want to keep a context-specific TARGET_DIR. It is
> easier to use,

 TARGET_DIR is only marginally easier to use than FOO_TARGET_DIR...


> it is more stable,

 ???

 You actually may have a point here, but I don't understand it.

> we can update it at will

 We can also update the definition of FOO_TARGET_DIR at will.


, and it makes it harder for packages and filesystems to do weird stuff.

 No it doesn't.


 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] docs/manual: document filesystems

[Yann E. MORIN]

We do expect that users be able to implement new filesystems and submit
them, or have local, custom and private filesystems in their
br2-external trees.

So, add docuemtnation for filesystems, based on the format we have for
packages.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 docs/manual/adding-filesystems.txt    | 119 ++++++++++++++++++++++++++++++++++
 docs/manual/adding-packages-hooks.txt |   7 +-
 docs/manual/adding-packages.txt       |   2 +
 3 files changed, 127 insertions(+), 1 deletion(-)
 create mode 100644 docs/manual/adding-filesystems.txt

diff --git a/docs/manual/adding-filesystems.txt b/docs/manual/adding-filesystems.txt
new file mode 100644
index 0000000000..bc122a9a64
--- /dev/null
+++ b/docs/manual/adding-filesystems.txt
@@ -0,0 +1,119 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+=== Infrastructure for filesystems
+
+As for packages, Buildroot provides an infrastructure to build
+filesystems.
+
+The filesystem infrastructure follows a carefully-crafted sequence:
+
+1. Assemble an intermediate, internal tarball from the +target/+
+   directory:
+  * create users and groups,
+  * set the correct rights on files,
+  * create device nodes,
+  * create the intermediate tarball;
+2. Then for each filesystem:
+  * extract the intermediate tarball to a filesystem-specific
+    +$(TARGET_DIR)+,
+  * execute the filesystem pre-gen hooks (see below),
+  * optionally, touch everything to a specific date, for reproducibility,
+  * execute the filesystem commands,
+  * execute the filesystem post-gen hooks (see below),
+  * remove the filesystem-specific +$(TARGET_DIR)+;
+3. Remove the intermediate tarball.
+
+[[filesystem-tutorial]]
+==== +filesystem+ tutorial
+
+Let's look at an example filesytem. First and foremost, like for a
+package, create a directory named from your filesystem, e.g. +foo+.
+
+Then, create a +Config.in+ file describing the menuconfig entries
+for the filesystem. Be sure to have that +Config.in+ included from
++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree).
+For example:
+
+----
+01: config BR2_TARGET_ROOTFS_FOO
+02: 	bool "foo root filesystem"
+03: 	help
+04: 	  Build a foo root filesystem.
+05: 
+06: if BR2_TARGET_ROOTFS_FOO
+07: 
+08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR
+09: 	bool "bar format"
+10: 	help
+11: 	  Enable the bar format.
+12: 
+13: endif
+----
+
+Then, in the same directory, create a +foo.mk+ file describing how
+to actually assemble the filesystem image from the content of
++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the
+Makefile code (automatic for built-in filesystems, not for filesystems
+in a *br2-external* tree.) For example:
+
+----
+01: ################################################################################
+02: #
+03: # foo filesystem
+04: #
+05: ################################################################################
+06: 
+07: ROOTFS_FOO_DEPENDENCIES = host-foo-utils
+08: 
+09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y)
+10: ROOTFS_FOO_OPTS = --format=bar
+11: else
+12: ROOTFS_FOO_OPTS = --format=foo
+13: endif
+14: 
+15: define ROOTFS_FOO_CMDS
+16: 	$(HOST_DIR)/bin/mkfs.foo \
+17: 		--root-dir=$(TARGET_DIR) \
+18: 		--option-foo --option-bar \
+19: 		$(ROOTFS_FOO_OPTS) \
+20: 		--output-file=$(@)
+21: endef
+22: 
+23: $(eval $(rootfs))
+----
+
+On line 7, we specify the dependencies on packages needed to build the
+filesystem. Usually, those dependencies are one or more host-packages
+that provide the filesystem generator.
+
+On lines 9 to 13, we define a conditional set of options, depending on
+the user's choice.
+
+On lines 15 to 21, we define the actual command that will generate the
+filesystem image.
+
+Finally, on line 23, we invoke the +rootfs+ macro that generates all the
+Makefile rules that actually allow the filesystem to be built.
+
+[[filesystem-reference]]
+==== +filesystem+ reference
+
+If one compares a package with a filesystem, one may notice that the
+variables for a filesystem start with the prefix +ROOTFS_+, when the
+variables for packages have no prefix.
+
+There is a single macro that a filesystem may set:
+
+* +ROOTFS_FOO_CMDS+ contains the commands to execute to actually
+  generate the filesystem iamge from the content of the +$(TARGET_DIR)+
+  directory.
+
+Additionally, like for packages, there are a set of hooks; see
+xref:hooks[] for details.
+
+And here is a list of variables that may be used (but may *not* be set):
+
+* +$(TARGET_DIR)+ is the directory containing the +target/+ directory.
+
+* +$(@)+ is the filename where to store the filesystem image into.
diff --git a/docs/manual/adding-packages-hooks.txt b/docs/manual/adding-packages-hooks.txt
index 0ce79f8907..34d5801169 100644
--- a/docs/manual/adding-packages-hooks.txt
+++ b/docs/manual/adding-packages-hooks.txt
@@ -11,7 +11,7 @@ Most hooks aren't really useful for generic packages, since the +.mk+
 file already has full control over the actions performed in each step
 of the package construction.
 
-The following hook points are available:
+The following hook points are available for packages:
 
 * +LIBFOO_PRE_DOWNLOAD_HOOKS+
 * +LIBFOO_POST_DOWNLOAD_HOOKS+
@@ -46,6 +46,11 @@ The following hook points are available:
 * +LIBFOO_PRE_LEGAL_INFO_HOOKS+
 * +LIBFOO_POST_LEGAL_INFO_HOOKS+
 
+The following hook points are available for filesystems:
+
+* +ROOTFS_FOO_PRE_GEN_HOOKS+
+* +ROOTFS_FOO_POST_GEN_HOOKS+
+
 These variables are 'lists' of variable names containing actions to be
 performed at this hook point. This allows several hooks to be
 registered at a given hook point. Here is an example:
diff --git a/docs/manual/adding-packages.txt b/docs/manual/adding-packages.txt
index 4a4a17e879..83c93f3dde 100644
--- a/docs/manual/adding-packages.txt
+++ b/docs/manual/adding-packages.txt
@@ -47,6 +47,8 @@ include::adding-packages-asciidoc.txt[]
 
 include::adding-packages-linux-kernel-spec-infra.txt[]
 
+include::adding-filesystems.txt[]
+
 include::adding-packages-hooks.txt[]
 
 include::adding-packages-gettext.txt[]
-- 
2.14.1