[U-Boot] [RFC 0/3] uboot-doc User's Manual Generation Tool

[jschmoller <jschmoller@xes-inc.com>]

On Tue, 2009-07-28 at 23:27 +0200, Wolfgang Denk wrote:
> Dear John,
> 
> in message <1248813631.3915.102.camel@johns> you wrote:
> >
> > It seems to me that DUTS is designed to test U-Boot and also automates
> > the running of commands whose output can be put online in the DULG. I
> 
> Correct. And the DULG itself is a wiki (TWiki at the moment, to be
> converted to Foswiki ASAP) based framework which holds the "static"
> parts of the documentation.
> 
> > didn't notice any documented procedure on how to turn the DULG into XML,
> > extensible PDFs, etc. It also seems as if the DULG is a combination of
> > hand-edited wiki pages as well as the DUTS output? I was hoping to
> 
> You probably can export the DULG into XML, too - but what would that
> be good for? At the moment we export it into PDF, PostScript, HTML and
> plain text.

Exporting to XML (and specifically DocBook XML, which is widely
supported and accepted as a standard (at least that's what the tech
writer next to me says :) ) is, I feel, a key to this whole process.
Since XML doesn't specify a style, but rather what the content *means*,
style sheets can be designed to turn the DocBook XML into anything, in
any format, rearranged in any way, and with any style (fonts, colors,
graphics).  Again, I'm not familiar with the process you use to export
HTML to PFD, PS, etc. but I'm guessing that the formatting and style end
up being very similar.

> > develop a system that's completely automated. I also noticed in a quick
> 
> Hm... "completely automated" is a nice buzzword, but you still have to
> write the documentation in the first place.

You're in luck, I've already volunteered to write it in the first
place :) and much of it is already done.  Take a look at the manual
posted on the X-ES website and see for yourself.
http://www.xes-inc.com/sources/u-boot/xpedite5370.pdf This manual was
completely generated with my tool.

> The nice thing of the wiki based approach is that you can have a
> (even random) collection of pages which can be "linearized" and
> brought into arbitrary sets of linear doucumentation - this is what
> the WebOrder pages are good for - see
> http://www.denx.de/wiki/DULG/WebOrder
> 
> So you can use the same set of wiki pages an genreate for example the
> full fledged manual, a short version including only the most
> significant topics and an extended version including other stuff that
> is normally not part of the manual - all from the same envrionment.

This is definitely possible with DocBook XML.  For those who care, I'm
thinking the @role attribute available in standard DocBook, though there
may be a better way.  I'll talk this through with our tech writers.

> > probe around that a few items in the DULG seem to be out of sync (imd
> > vs. i2c md, source vs. autoscr, etc.) and DHCP seems to be out of date
> > as well. Is the process for updating the DULG automatic? If so, how is
> > it done?
> 
> Soemone still has to write the documentation - and this is not always
> done  in  sync  with  the  code.  Your  approach  of  including   the
> documentation  with  the  source  code  makes it more likely that one
> remebers to update the docs as  well,  but  just  remember  how  many
> places  there  are  where  code  and  comments  don't agree - it's no
> guarantee either.

I would guess someone is several dozen times more likely to update a
documentation snippet in the source over modifying a Wiki page.

> If you find such areas in the DULG that are out of sync or missing
> please feel free to add them - everybody can contribute and modify the
> pages or add new content. And everyboidy can modify and extend the
> DUTS test cases, too.
> 
> > As I mentioned, I borrowed this idea from the kernel-doc script in
> > Linux, which does do API documentation.  But my hope for the uboot-doc
> > tool would be to create user documentation, or a manual we'd provide to
> > a customer when they purchased a product that would describe available
> > commands, environment variables, common operations, etc.
> 
> That's what the DULG does, so you are truely duplicating efforts.

As I mentioned earlier, it seems to me that most vendors are creating
their own manuals.  That either means knowledge of the existence of DULG
is limited, or the tool isn't meeting people's needs.  In order to use
it, you need to download a completely separate package (DUTS), learn to
use it, then hand edit the output into a Wiki page.  The advantage with
a documentation engine distributed with the sources is that there is no
learning curve.  "make pdf" creates a perfectly suitable PDF manual, and
with a little work (first time only) they can make it look like anything
and chug out manuals for all their boards in minutes.

> > > Your approach may be suitable for standard  documents,  but  in  many
> > > years  of  working  with U-Boot (and Linux) we found that it does not
> > > work so well with the specific needs we have for a User's Manual. One
> > > issue is that you have to support many different boards (well,  maybe
> > > not  you  as  a user, but we as a community). And you have to include
> > > examples. And examples must really fit the board. If you for  example
> > > provide  a  manual entry for the "erase" command you better make sure
> > > that your example does not erase a range of flash that on some  board
> > > happens to hold the U-Boot image. etc.
> > 
> > It's my hope that the documentation system I proposed can be made
> > flexible enough to support things such as this without too much
> > headache.  That's the hope at least :)
> 
> Heh. Look at the DULG implementation... it's simple for the first 3 or
> 4 boards, especially when these are similar. But try supporting ARM
> and PowerPC and MIPS boards from N different vendors, with different
> kernel versions etc.

That, I believe, is one of the strengths of the tool I created.  First,
it uses the C pre-compiler to conditionally include comments from
source.  Secondly, the common XML templates support pre-compiler
directives, so you can #ifdef in a section only when such-and-such is
defined.  And finally, if your manual requirements are dramatically
different than others, a whole template can be overridden (or a whole
new template added or deleted) in the board/${BOARDDIR}/manual
directory.  In my opinion, that's pretty flexible.

> > This would make DUTS/DULG more appealing.  What is the process of
> > generating DocBook XML from DUTS/DULG?
> 
> What would you need DocBook XML for?

As I mentioned above, I think the presence of DocBook XML is crucial to
gain wide acceptance.  When people see that this tool can generate
manuals that look just like the manuals they offer now, I think that
will make the sale.

> > It's true, I may have. :)  On the other hand, it seems that there are
> > still a lot of people who are in the same boat.  Most manuals I have
> > seen from other embedded companies (Freescale, Analog Devices, etc.)
> > seem to provide their own format/content. Additionally, most companies
> > will prefer to have their content formatted to match the rest of their
> > manuals, which is easily done from DocBook XML. Again, if you can do
> > that with DUTS/DULG, then that's great and probably eliminates the need
> > for this tool.
> 
> Ah, I see.
> 
> Well, we use htmldoc to generate .pdf and .ps files from HTML; you
> could for example use Tidy to convert HTML to DocBook (and I'm sure
> there are lots of other tools that can do that).

XML specifies what the data means.  HTML specifies what the data should
look like.  In my head, it's intuitive to say titles should be big and
bold, but it's not intuitive to say big and bold should be a title. The
conversion is imperfect.  DocBook has a page on doing just that, and
it's 5 steps which would probably several hours to get right for each
manual.
http://wiki.docbook.org/topic/Html2DocBook

> > > Hm... we cannot look at your patches, you just posted the patch
> > > statistics, but no content.
> > 
> > This is just the patch series introduction, the actual content appears
> > in 1-3.
> 
> Yes, I know. Sorry, patches arrived shortly after I started replying,
> and I didn;t notice.
> 
> > As I mentioned, I'm not all that familiar with the abilities of
> > DUTS/DULG, but I got the impression that there was still a fair amount
> > of manual labor involved for each manual.  Thus most people (that I'm
> 
> Indeed there is a lot of manual labor involved in preparing and
> updating the documentation (the framework, that is; once a board has
> been configured and added to DUTS, generating the needed include
> files is mostly an automatic procedure - just the upload of the files
> to the web server has to be triggered manually, but even this is
> scripted).
> 
> But you have to spend at least the same amount of writing and updating
> the documentation, don't you?

At least the way I see it, with your tool there's a fair amount of
manual labor for every manual.  With my tool, it's a similar amount of
manual labor once for everyone in the whole community.  Say we add a new
command, it's extra work to comment it properly, but then everyone in
the community reaps the benefits.  Or more complexly, we add an entirely
new interface of some sort, and a whole new manual section needs to be
written.  Someone writes it once and everyone in the community can reuse
that section.  And sure, you could do that with the DULG, but that's
copy-and-paste-and-edit manual labor.  For this tool, it would either be
just run "make pdf" again, or possibly add a #define to your config file
and then run "make pdf".

> > I was hoping that creating documentation from comments in the source
> > would be easier to maintain and providing DocBook output would allow
> > others to more easily reuse the U-Boot manual contents.
> 
> A very, very old version of the DULG was set up like this, using a set
> of Perl scripts and generating DokBoot output. We switched to the wiki
> based approach some 6 years ago, and I never had the slightest wish to
> switch back.

I do see the benefit of using a wiki, but I feel that it doesn't fit the
needs of many U-Boot developers who need to generate documentation in a
variety of styles and formats.  I have to assume the vast majority of
current U-Boot developers are rolling their own documentation as most
boards are not supported by DULG.  My hope is that if we generated
U-Boot User's manuals in a different, easier manner with a more flexible
output many more people could use it, thereby saving time which they can
put to improving U-Boot itself :)

Thanks,
John