[U-Boot] RFC: Testing U-Boot Part 1

[U-Boot] RFC: Testing U-Boot Part 1

[Simon Glass]

Hi,

Summary: I am quite keen on improving the test infrastructure in
U-Boot. I would like to have a test suite that can run in a minute or
two on a Linux PC and test all non-platform code.

Detail
======
We can break the U-Boot code base into two parts:

1. Platform code, which is SOC-specifc. At present this is the CPU
init (arch/xxx/cpu/...) and SOC-specific drivers (mostly in the
drivers directory). There is also a small amount of generic CPU code
in arch/xxx/lib and some board-specific code/drivers (e.g. drivers not
within the SOC).

2. Portable/Generic U-Boot code, which is cross-platform. This
includes many drivers, the various commands, file system support,
things like the MMC, USB and network stacks, etc.

My focus in this email the the second part of the code base - all the
code which is not platform-specific, but can still have bugs.

Proposal
========
To a large extent, testing of this part of the code base could simply
be built on one or more of the available platforms. We could then run
the tests on that platform, and announce that the code base works fine
on that platform. Obviously the platform needs to support all possible
features of U-Boot.

However, this approach fails to take advantage of two useful
properties of this code:

- It is cross-platform, and even runs on x86
- It is standalone, requiring no OS libraries to run

For speed, debugging and convenience, it would be nice to run U-Boot
under a generic Linux environment on a workstation, and test all the
generic non-platform code. The basic problem with this is that the
non-platform code requires the platform code to operate. Even the x86
platform code is designed for standalone operation on a particular x86
board, and is not suitable for running under x86 Linux.

To get around this I propose that we create a new ?native?
architecture. We write code in ?arch/native? which can run under
Linux. Since all the non-platform code will happily run under this new
?architecture?, we can then write tests which run quickly under x86
Linux (or another Linux for that matter). This U-Boot 'architecture'
should build natively on any 32/64-bit Linux machine since it just
uses standard Linux system calls. Calls to Linux would be entirely
within this arch/native subdirectory.

Benefit
=======
What will this buy us? Well we can do things like:

- Create a test SPI flash device, which is file-backed. Use this to
write a test of the cmd_sf layer by issuing a series of commands and
making sure that the correct SPI flash contents is obtained

- Create a test MMC or IDE device, backed by a file. Use this to issue
ext2 and fat commands to manipulate the filesystem. Then loopback
mount the file and check from Linux that U-Boot did the right thing

- Create a test Ethernet device with a mocked remote host attached.
Use this to issue network commands like bootp and tftp and check that
the correct results are obtained.

Ultimately (one day) we might have a set of unit tests for U-Boot
which can run to very quickly check that a patch does not break the
core U-Boot code.

Comments
========
At this early stage I am looking for comments on the concept - how
useful it is and how best to implement it.

Regards,
Simon

[U-Boot] RFC: Testing U-Boot Part 1

[Andreas Bießmann]

Dear Simon,

Am 25.08.2011 14:58, schrieb Simon Glass:
> Hi,
> 
> Summary: I am quite keen on improving the test infrastructure in
> U-Boot. I would like to have a test suite that can run in a minute or
> two on a Linux PC and test all non-platform code.

<snip>

> To get around this I propose that we create a new ?native?
> architecture. We write code in ?arch/native? which can run under
> Linux. Since all the non-platform code will happily run under this new
> ?architecture?, we can then write tests which run quickly under x86
> Linux (or another Linux for that matter). This U-Boot 'architecture'
> should build natively on any 32/64-bit Linux machine since it just
> uses standard Linux system calls. Calls to Linux would be entirely
> within this arch/native subdirectory.

why don't use some unit testing framework like cunit, or ceedling (which
can do HW mocks easily)?

regards

Andreas Bie?mann

[U-Boot] RFC: Testing U-Boot Part 1

[Marek Vasut]

On Thursday, August 25, 2011 02:58:00 PM Simon Glass wrote:
> Hi,
> 
> Summary: I am quite keen on improving the test infrastructure in
> U-Boot. I would like to have a test suite that can run in a minute or
> two on a Linux PC and test all non-platform code.
> 
> Detail
> ======
> We can break the U-Boot code base into two parts:
> 
> 1. Platform code, which is SOC-specifc. At present this is the CPU
> init (arch/xxx/cpu/...) and SOC-specific drivers (mostly in the
> drivers directory). There is also a small amount of generic CPU code
> in arch/xxx/lib and some board-specific code/drivers (e.g. drivers not
> within the SOC).
> 
> 2. Portable/Generic U-Boot code, which is cross-platform. This
> includes many drivers, the various commands, file system support,
> things like the MMC, USB and network stacks, etc.
> 
> My focus in this email the the second part of the code base - all the
> code which is not platform-specific, but can still have bugs.
> 
> Proposal
> ========
> To a large extent, testing of this part of the code base could simply
> be built on one or more of the available platforms. We could then run
> the tests on that platform, and announce that the code base works fine
> on that platform. Obviously the platform needs to support all possible
> features of U-Boot.
> 
> However, this approach fails to take advantage of two useful
> properties of this code:
> 
> - It is cross-platform, and even runs on x86
> - It is standalone, requiring no OS libraries to run
> 
> For speed, debugging and convenience, it would be nice to run U-Boot
> under a generic Linux environment on a workstation, and test all the
> generic non-platform code. The basic problem with this is that the
> non-platform code requires the platform code to operate. Even the x86
> platform code is designed for standalone operation on a particular x86
> board, and is not suitable for running under x86 Linux.
> 
> To get around this I propose that we create a new ?native?
> architecture. We write code in ?arch/native? which can run under
> Linux. Since all the non-platform code will happily run under this new
> ?architecture?, we can then write tests which run quickly under x86
> Linux (or another Linux for that matter). This U-Boot 'architecture'
> should build natively on any 32/64-bit Linux machine since it just
> uses standard Linux system calls. Calls to Linux would be entirely
> within this arch/native subdirectory.
> 
> Benefit
> =======
> What will this buy us? Well we can do things like:
> 
> - Create a test SPI flash device, which is file-backed. Use this to
> write a test of the cmd_sf layer by issuing a series of commands and
> making sure that the correct SPI flash contents is obtained
> 
> - Create a test MMC or IDE device, backed by a file. Use this to issue
> ext2 and fat commands to manipulate the filesystem. Then loopback
> mount the file and check from Linux that U-Boot did the right thing
> 
> - Create a test Ethernet device with a mocked remote host attached.
> Use this to issue network commands like bootp and tftp and check that
> the correct results are obtained.
> 
> Ultimately (one day) we might have a set of unit tests for U-Boot
> which can run to very quickly check that a patch does not break the
> core U-Boot code.
> 
> Comments
> ========
> At this early stage I am looking for comments on the concept - how
> useful it is and how best to implement it.

Hi Simon, 

sounds just awesome. I was planning on doing a similar thing, but "from the 
other side". I'm sure you're aware of the POST test framework. I was planning to 
either extend it or write new self-standing implementation that'd allow debuging 
platform-dependent drivers.

I can CC you on that if you have some interest.

> 
> Regards,
> Simon
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] RFC: Testing U-Boot Part 1

[Mike Frysinger]

On Thursday, August 25, 2011 09:56:02 Andreas Bie?mann wrote:
> Am 25.08.2011 14:58, schrieb Simon Glass:
> > Summary: I am quite keen on improving the test infrastructure in
> > U-Boot. I would like to have a test suite that can run in a minute or
> > two on a Linux PC and test all non-platform code.
> 
> <snip>
> 
> > To get around this I propose that we create a new ?native?
> > architecture. We write code in ?arch/native? which can run under
> > Linux. Since all the non-platform code will happily run under this new
> > ?architecture?, we can then write tests which run quickly under x86
> > Linux (or another Linux for that matter). This U-Boot 'architecture'
> > should build natively on any 32/64-bit Linux machine since it just
> > uses standard Linux system calls. Calls to Linux would be entirely
> > within this arch/native subdirectory.
> 
> why don't use some unit testing framework like cunit, or ceedling (which
> can do HW mocks easily)?

these testing frameworks wont make any difference to what Simon is proposing.  
he is focusing on getting u-boot to build & run on your desktop machine.  
after that is done, we can talk about the actual tests and harnesses (although 
anything that requires ruby should immediately be disqualified imo :P).
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20110825/bb4fcf7b/attachment.pgp 

[U-Boot] RFC: Testing U-Boot Part 1

[Mike Frysinger]

On Thursday, August 25, 2011 08:58:00 Simon Glass wrote:
> Proposal
> ========

for people who might be familiar with the barebox boot loader, Simon is 
basically proposing the same thing as barebox's "sandbox" target.

to avoid confusion/fragmentation in this area, and since the name is a large 
part bike shedding, i propose we adopt the same name that barebox has.  that 
way we can sanely use the same term when comparing/contrasting.

from the barebox website:
	Sandbox

	If you develop features for Barebox, you can use the 'sandbox' target
	which compiles Barebox as a POSIX application in the Linux userspace: it
	can be started like a normal command and even has network access
	(tun/tap). Files from the local filesytem can be used to simulate
	devices.

> Comments
> ========
> At this early stage I am looking for comments on the concept - how
> useful it is and how best to implement it.

i certainly think it's a worthwhile exercise, especially if it gets us to the 
point where we can do `make check` and have confidence in our common code 
changes not being completely f-ed up.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20110825/4f8ac0f4/attachment.pgp 

[U-Boot] RFC: Testing U-Boot Part 1

[Anton Staaf]

On Thu, Aug 25, 2011 at 8:01 AM, Mike Frysinger <vapier@gentoo.org> wrote:
> On Thursday, August 25, 2011 08:58:00 Simon Glass wrote:
>> Proposal
>> ========
>
> for people who might be familiar with the barebox boot loader, Simon is
> basically proposing the same thing as barebox's "sandbox" target.
>
> to avoid confusion/fragmentation in this area, and since the name is a large
> part bike shedding, i propose we adopt the same name that barebox has. ?that
> way we can sanely use the same term when comparing/contrasting.
>
> from the barebox website:
> ? ? ? ?Sandbox
>
> ? ? ? ?If you develop features for Barebox, you can use the 'sandbox' target
> ? ? ? ?which compiles Barebox as a POSIX application in the Linux userspace: it
> ? ? ? ?can be started like a normal command and even has network access
> ? ? ? ?(tun/tap). Files from the local filesytem can be used to simulate
> ? ? ? ?devices.

Makes good sense.

>> Comments
>> ========
>> At this early stage I am looking for comments on the concept - how
>> useful it is and how best to implement it.
>
> i certainly think it's a worthwhile exercise, especially if it gets us to the
> point where we can do `make check` and have confidence in our common code
> changes not being completely f-ed up.

Indeed, I am very excited about this as well.  :)

Thanks,
    Anton

> -mike
>
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot
>
>

[U-Boot] RFC: Testing U-Boot Part 1

[Wolfgang Denk]

Dear Simon Glass,

In message <CAPnjgZ3ZqFhZgtCAUSRngU3qB5ajxO9FthkmyuWurAWYkEvAHg@mail.gmail.com> you wrote:
> 
> Summary: I am quite keen on improving the test infrastructure in
> U-Boot. I would like to have a test suite that can run in a minute or
> two on a Linux PC and test all non-platform code.

I highly appreaciate such efforts!

> For speed, debugging and convenience, it would be nice to run U-Boot
> under a generic Linux environment on a workstation, and test all the
> generic non-platform code. The basic problem with this is that the
> non-platform code requires the platform code to operate. Even the x86
> platform code is designed for standalone operation on a particular x86
> board, and is not suitable for running under x86 Linux.

Some parts of what you are looking for can be done by running U-Boot
images in a QEMU based emulated envrionment.  So far we have only a
few targets in the tree that actually support this.  I wish we had
more of them.

I am aware that this is a different thing from what you propose here,
but I would like to state that we should not forget this option -
anybody who wants to work in that area will be more than welcome, too.


> To get around this I propose that we create a new ?native?
> architecture. We write code in ?arch/native? which can run under
> Linux. Since all the non-platform code will happily run under this new
> ?architecture?, we can then write tests which run quickly under x86
> Linux (or another Linux for that matter). This U-Boot 'architecture'
> should build natively on any 32/64-bit Linux machine since it just
> uses standard Linux system calls. Calls to Linux would be entirely
> within this arch/native subdirectory.

As Mike already pointed out, this is what BB has as "sandbox"
configuration.

> - Create a test SPI flash device, which is file-backed. Use this to
> write a test of the cmd_sf layer by issuing a series of commands and
> making sure that the correct SPI flash contents is obtained
>
> - Create a test MMC or IDE device, backed by a file. Use this to issue
> ext2 and fat commands to manipulate the filesystem. Then loopback
> mount the file and check from Linux that U-Boot did the right thing

In a later step it might even be possible to implement some "pass
through" mode to acces actual devices / drivers in some way.  Think of
using libusb to talk to USB devices.

> - Create a test Ethernet device with a mocked remote host attached.

I'm not sure what you mean by "a mocked remote host".  We should be
able to send and receive packets from a real network interface as
well.

> At this early stage I am looking for comments on the concept - how
> useful it is and how best to implement it.

Any step in this direction, how inclomplete it may be in the
beginning, is highly appreciated.  It may make sense to look over the
fence and study how BBis doing this.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Hokey religions and ancient weapons are  no  substitute  for  a  good
blaster at your side.                                      - Han Solo

[U-Boot] RFC: Testing U-Boot Part 1

[Mike Frysinger]

On Thursday, August 25, 2011 16:21:51 Wolfgang Denk wrote:
> Simon Glass wrote:
> > For speed, debugging and convenience, it would be nice to run U-Boot
> > under a generic Linux environment on a workstation, and test all the
> > generic non-platform code. The basic problem with this is that the
> > non-platform code requires the platform code to operate. Even the x86
> > platform code is designed for standalone operation on a particular x86
> > board, and is not suitable for running under x86 Linux.
> 
> Some parts of what you are looking for can be done by running U-Boot
> images in a QEMU based emulated envrionment.  So far we have only a
> few targets in the tree that actually support this.  I wish we had
> more of them.
> 
> I am aware that this is a different thing from what you propose here,
> but I would like to state that we should not forget this option -
> anybody who wants to work in that area will be more than welcome, too.

i do a lot of testing using the GNU sim and Blackfin boards.  the goal there 
is that the GNU sim is able to execute all of the Blackfin builds 
*unmodified*, so i dont need sep configs.  as you say, both methods are very 
useful to have.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20110825/93d1cf1a/attachment.pgp 

[U-Boot] RFC: Testing U-Boot Part 1

[Marek Vasut]

On Thursday, August 25, 2011 04:56:39 PM Mike Frysinger wrote:
> On Thursday, August 25, 2011 09:56:02 Andreas Bie?mann wrote:
> > Am 25.08.2011 14:58, schrieb Simon Glass:
> > > Summary: I am quite keen on improving the test infrastructure in
> > > U-Boot. I would like to have a test suite that can run in a minute or
> > > two on a Linux PC and test all non-platform code.
> > 
> > <snip>
> > 
> > > To get around this I propose that we create a new ?native?
> > > architecture. We write code in ?arch/native? which can run under
> > > Linux. Since all the non-platform code will happily run under this new
> > > ?architecture?, we can then write tests which run quickly under x86
> > > Linux (or another Linux for that matter). This U-Boot 'architecture'
> > > should build natively on any 32/64-bit Linux machine since it just
> > > uses standard Linux system calls. Calls to Linux would be entirely
> > > within this arch/native subdirectory.
> > 
> > why don't use some unit testing framework like cunit, or ceedling (which
> > can do HW mocks easily)?
> 
> these testing frameworks wont make any difference to what Simon is
> proposing. he is focusing on getting u-boot to build & run on your desktop
> machine. after that is done, we can talk about the actual tests and
> harnesses (although anything that requires ruby should immediately be
> disqualified imo :P). -mike

Definitelly agree with the RUBY part.

[U-Boot] RFC: Testing U-Boot Part 1

[Graeme Russ]

Hi Simon, Mike

On Fri, Aug 26, 2011 at 1:01 AM, Mike Frysinger <vapier@gentoo.org> wrote:
> On Thursday, August 25, 2011 08:58:00 Simon Glass wrote:
>> Proposal
>> ========
>
> for people who might be familiar with the barebox boot loader, Simon is
> basically proposing the same thing as barebox's "sandbox" target.
>
> to avoid confusion/fragmentation in this area, and since the name is a large
> part bike shedding, i propose we adopt the same name that barebox has.  that
> way we can sanely use the same term when comparing/contrasting.
>

Yes, I agree that naming the target 'sandbox' is a good idea

>
>> Comments
>> ========
>> At this early stage I am looking for comments on the concept - how
>> useful it is and how best to implement it.
>
> i certainly think it's a worthwhile exercise, especially if it gets us to the
> point where we can do `make check` and have confidence in our common code
> changes not being completely f-ed up.

Indead - And I think it will serve as further impetus to arch-neutralise
code outside /arch/

Regards,

Graeme

[U-Boot] RFC: Testing U-Boot Part 1

[Simon Glass]

Hi,

Thanks all for comments. It sounds like people are keen on the idea so
far as it goes. I will work on a patch set complete enough to bring up
a U-Boot prompt, allowing typing of 'help' and with a special segfault
feature for anything else.

Before I do this, and to avoid me redoing work later:

1. What should I call the architecture? I have so far called it 'native'.

2. What should I call the vendor (board/xxx)? 'test' or 'sandbox'?

3. What should I call the board? Is that 'sandbox'?

4. When I create a driver, like the serial test driver, should that be
serial_test.c, test_serial.c, sandbox_serial or something else?

5. I need an arch/xxx/lib/board.c. Rather than try to create a new
minimal one, I think I should copy an existing one and change it as
little as possible (in the light of Graeme's comments and to avoid yet
another version of this file). Thoughts?

Regarding the actual test framework, I think this can come later.
Clearly we can do unit tests at the file level, or more
integration-like tests such as I listed in my email. But I know very
little about Ruby (little girl up the road?)

Summary of suggestions so that I don't flood this thread with replies:

Andreas Bie?mann: why don't use some unit testing framework like
cunit, or ceedling (which
can do HW mocks easily)?

- the test framework is TBD - as Mike commented this is something else
- another level that can be piled on later.

Marek Vasut: sounds just awesome. I was planning on doing a similar
thing, but "from the
other side". I'm sure you're aware of the POST test framework. I was planning to
either extend it or write new self-standing implementation that'd allow debuging
platform-dependent drivers.

- sounds great to me - perhaps we could have a standard test for SPI
flash, another for NAND, another for USB, etc.

Mike Frysinger: i certainly think it's a worthwhile exercise,
especially if it gets us to the
point where we can do `make check` and have confidence in our common code
changes not being completely f-ed up.

- that is my hope, particularly if 'make check' is fast.

Wolfgang Denk: In a later step it might even be possible to implement some "pass
through" mode to acces actual devices / drivers in some way.  Think of
using libusb to talk to USB devices.

- Yes - that would allow checking of more layers of the USB stack. But
I even see some value in just mocking out the whole USB stack and
providing a pretend USB memory stick as a block device..

Wolfgang Denk: I'm not sure what you mean by "a mocked remote host".
We should be
able to send and receive packets from a real network interface as
well.

- I mean that the tftp command will 'obtain' a file when it asks for
one, although the actual Ethernet layer is mocked and doesn't actually
go out on the wire. Imagine an Ethernet driver which has a half-baked
tftp server in it. Yes I also see value in actually using machine
interfaces since the testing can be more thorough.

Wolfgang Denk: Some parts of what you are looking for can be done by
running U-Boot
images in a QEMU based emulated envrionment.  So far we have only a
few targets in the tree that actually support this.  I wish we had
more of them.

...I am aware that this is a different thing from what you propose here,
but I would like to state that we should not forget this option -
anybody who wants to work in that area will be more than welcome, too.

-Yes, both are valuable. Maybe one day we will get a s/w model of an
SOC when it is released. For now, this sort of thing is quite a bit of
work. There are loads of automated ways to test things- ones I can
think of are:

1.  real hardware, with some sort of control board which can push
buttons, produce/access Ethernet/USB traffic, camera to watch the LCD,
LCD to driver the camera, etc.

2. test code which runs on real hardware either at the board level or
at the U-Boot command level. Requires a user to watch / driver things

3. test code which runs on faked hardware (this is what I am proposing
here). Basically the opposite of 2.

4. test code could run on real hardware, but is actually run on QEMU
etc., with special hooks in the simulator for injecting failure,
handling the other end of ethernet//USB, etc..A lot of work but really
cool.

5. unit tests for each file, where anything that it calls is mocked
and replaced with something which can inject failure, etc. We should
have more of this anyway. People often don't like writing it though
:-(

Graeme Russ: Indead - And I think it will serve as further impetus to
arch-neutralise
code outside /arch/

- Yes - let's hope so.

Regards,
Simon

[U-Boot] RFC: Testing U-Boot Part 1

[Graeme Russ]

Hi Simon,

On Fri, Aug 26, 2011 at 1:32 PM, Simon Glass <sjg@chromium.org> wrote:
> Hi,
>
> Thanks all for comments. It sounds like people are keen on the idea so
> far as it goes. I will work on a patch set complete enough to bring up
> a U-Boot prompt, allowing typing of 'help' and with a special segfault
> feature for anything else.
>
> Before I do this, and to avoid me redoing work later:
>
> 1. What should I call the architecture? I have so far called it 'native'.

sandbox

>
> 2. What should I call the vendor (board/xxx)? 'test' or 'sandbox'?

sandbox

>
> 3. What should I call the board? Is that 'sandbox'?

sandbox

>
> 4. When I create a driver, like the serial test driver, should that be
> serial_test.c, test_serial.c, sandbox_serial or something else?

I guess you'll have /drivers/serial/sandbox.c, /drivers/net/sandbox.c
etc.

/include/configs/sandbox.h will need to include defaults for how
these devices are configured. For example, you may want to have
the sandbox serial go to /dev/ttyS0 or /dev/ttyS1

Make sure that printf() goes through U-Boot printf() not the host's libc

And have you dealt with putc() and getc() hooking so that the U-Boot
stdio can go to either the hosts stdio or a serial port?


> 5. I need an arch/xxx/lib/board.c. Rather than try to create a new
> minimal one, I think I should copy an existing one and change it as
> little as possible (in the light of Graeme's comments and to avoid yet
> another version of this file). Thoughts?

Make this board an example of a New World Order ;) - Seriously, take
arch/x86/lib/board.c as it has arrays for both init_f and init_r.
If you look at board_init_f it has very little other than the loop. But
board_init_r is a mess - It starts out as a loop and finished being a
sequence of function calls. Implement board_init_r so it consists of
only a processing loop - Debatable as to whether or not to but the
while(1) { main_loop;} as a final 'init' function or whether to have
this after the init loop is processed

I imagine that sandbox will have no _f init functions as there will
be no relocation anyway. Still, put in board_init_f but have an
empty sequence.

[snip]

> Wolfgang Denk: I'm not sure what you mean by "a mocked remote host".
> We should be
> able to send and receive packets from a real network interface as
> well.
>
> - I mean that the tftp command will 'obtain' a file when it asks for
> one, although the actual Ethernet layer is mocked and doesn't actually
> go out on the wire. Imagine an Ethernet driver which has a half-baked
> tftp server in it. Yes I also see value in actually using machine
> interfaces since the testing can be more thorough.

Yes, using a real interface (loopback even) would be good as you can then
also test that your host's tftp server is working

[snip]

Regards,

Graeme

[U-Boot] RFC: Testing U-Boot Part 1

[Mike Frysinger]

On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:
> 1. What should I call the architecture? I have so far called it 'native'.
> 2. What should I call the vendor (board/xxx)? 'test' or 'sandbox'?
> 3. What should I call the board? Is that 'sandbox'?

as Graeme said, just call them all "sandbox"

> 4. When I create a driver, like the serial test driver, should that be
> serial_test.c, test_serial.c, sandbox_serial or something else?

i think it depends on its function.  if the serial driver actually goes to 
std{in,err,out}, then perhaps "serial/sandbox_stdio.c".  let's not assume 
we'll only ever have one pseudo driver that we can use under the sandbox :).

> Wolfgang Denk: I'm not sure what you mean by "a mocked remote host".
> We should be
> able to send and receive packets from a real network interface as
> well.
> 
> - I mean that the tftp command will 'obtain' a file when it asks for
> one, although the actual Ethernet layer is mocked and doesn't actually
> go out on the wire. Imagine an Ethernet driver which has a half-baked
> tftp server in it. Yes I also see value in actually using machine
> interfaces since the testing can be more thorough.

why not just build on top of tun/tap ?  then we do get "real" network traffic, 
and you dont have to write your own tftp server because you can simply use the 
same exact one on your development machine that the board would connect to.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20110826/6fb86d74/attachment.pgp 

[U-Boot] RFC: Testing U-Boot Part 1

[Mike Frysinger]

On Friday, August 26, 2011 00:36:15 Graeme Russ wrote:
> On Fri, Aug 26, 2011 at 1:32 PM, Simon Glass wrote:
> > 4. When I create a driver, like the serial test driver, should that be
> > serial_test.c, test_serial.c, sandbox_serial or something else?
> 
> I guess you'll have /drivers/serial/sandbox.c, /drivers/net/sandbox.c
> etc.
> 
> /include/configs/sandbox.h will need to include defaults for how
> these devices are configured. For example, you may want to have
> the sandbox serial go to /dev/ttyS0 or /dev/ttyS1

since we get a main() entry point, we can make these into runtime flags

> Make sure that printf() goes through U-Boot printf() not the host's libc

u-boot already takes care of this by running the linker directly.  it is the 
compiler driver (i.e. `gcc`) that adds the implicit -lc and friends.

> And have you dealt with putc() and getc() hooking so that the U-Boot
> stdio can go to either the hosts stdio or a serial port?

that would be the problem of the sandbox serial driver, and i dont think it'd 
be that hard.  simply use read/write syscalls directly :).
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20110826/dce19d12/attachment.pgp 

[U-Boot] RFC: Testing U-Boot Part 1

[Simon Glass]

Hi Mike,

On Fri, Aug 26, 2011 at 1:55 PM, Mike Frysinger <vapier@gentoo.org> wrote:
> On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:
>> 1. What should I call the architecture? I have so far called it 'native'.
>> 2. What should I call the vendor (board/xxx)? 'test' or 'sandbox'?
>> 3. What should I call the board? Is that 'sandbox'?
>
> as Graeme said, just call them all "sandbox"

OK, sandbox it is.

>
>> 4. When I create a driver, like the serial test driver, should that be
>> serial_test.c, test_serial.c, sandbox_serial or something else?
>
> i think it depends on its function. ?if the serial driver actually goes to
> std{in,err,out}, then perhaps "serial/sandbox_stdio.c". ?let's not assume
> we'll only ever have one pseudo driver that we can use under the sandbox :).

OK well I suppose I can start with sandbox and we can see where it takes us.

>
>> Wolfgang Denk: I'm not sure what you mean by "a mocked remote host".
>> We should be
>> able to send and receive packets from a real network interface as
>> well.
>>
>> - I mean that the tftp command will 'obtain' a file when it asks for
>> one, although the actual Ethernet layer is mocked and doesn't actually
>> go out on the wire. Imagine an Ethernet driver which has a half-baked
>> tftp server in it. Yes I also see value in actually using machine
>> interfaces since the testing can be more thorough.
>
> why not just build on top of tun/tap ? ?then we do get "real" network traffic,
> and you dont have to write your own tftp server because you can simply use the
> same exact one on your development machine that the board would connect to.
> -mike

Because then you need to set up a real tftp server. It's fine to do
what you suggest, but if possible it would be nice to have
self-contained tests also, so long as it isn't too much work.

Regards,
Simon

>

[U-Boot] RFC: Testing U-Boot Part 1

[Simon Glass]

Hi Mike,

On Fri, Aug 26, 2011 at 1:59 PM, Mike Frysinger <vapier@gentoo.org> wrote:
> On Friday, August 26, 2011 00:36:15 Graeme Russ wrote:
>> On Fri, Aug 26, 2011 at 1:32 PM, Simon Glass wrote:
>> > 4. When I create a driver, like the serial test driver, should that be
>> > serial_test.c, test_serial.c, sandbox_serial or something else?
>>
>> I guess you'll have /drivers/serial/sandbox.c, /drivers/net/sandbox.c
>> etc.
>>
>> /include/configs/sandbox.h will need to include defaults for how
>> these devices are configured. For example, you may want to have
>> the sandbox serial go to /dev/ttyS0 or /dev/ttyS1
>
> since we get a main() entry point, we can make these into runtime flags

Yes that's the plan.

>
>> Make sure that printf() goes through U-Boot printf() not the host's libc
>
> u-boot already takes care of this by running the linker directly. ?it is the
> compiler driver (i.e. `gcc`) that adds the implicit -lc and friends.

My plan is actually to have a minimal link script and mostly do things
as the native compiler intends.

>
>> And have you dealt with putc() and getc() hooking so that the U-Boot
>> stdio can go to either the hosts stdio or a serial port?
>
> that would be the problem of the sandbox serial driver, and i dont think it'd
> be that hard. ?simply use read/write syscalls directly :).

Yes - I am thinking of an os.c file which contains the OS interface -
it will include Linux headers and no common.h, so avoid conflicts.

Regards,
Simon

> -mike
>

[U-Boot] RFC: Testing U-Boot Part 1

[Graeme Russ]

Hi Simon,

On 27/08/11 10:25, Simon Glass wrote:
> Hi Mike,
> 
> On Fri, Aug 26, 2011 at 1:55 PM, Mike Frysinger <vapier@gentoo.org> wrote:
>> On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:

[snip]

>>> - I mean that the tftp command will 'obtain' a file when it asks for
>>> one, although the actual Ethernet layer is mocked and doesn't actually
>>> go out on the wire. Imagine an Ethernet driver which has a half-baked
>>> tftp server in it. Yes I also see value in actually using machine
>>> interfaces since the testing can be more thorough.
>>
>> why not just build on top of tun/tap ?  then we do get "real" network traffic,
>> and you dont have to write your own tftp server because you can simply use the
>> same exact one on your development machine that the board would connect to.
>> -mike
> 
> Because then you need to set up a real tftp server. It's fine to do
> what you suggest, but if possible it would be nice to have
> self-contained tests also, so long as it isn't too much work.
> 

I don't consider having to set up a tftp server as a bad thing - Quite the
opposite really. There is plenty of network code in U-Boot that will
benefit from testing under the sandbox target because it is much easier to
debug. And there will be minimal impact on U-Boot code (just a sandbox
'Ethernet' driver is all that will be needed)

I am reminded of when the R&D department that developed to eNET board were
doing the firmware development before the first prototypes were available -
The created a HAL which used pcap from memory. I wasn't part of that team,
so I really don't know the details...

Regards,

Graeme

[U-Boot] RFC: Testing U-Boot Part 1

[Simon Glass]

Hi Graeme,

On Fri, Aug 26, 2011 at 7:23 PM, Graeme Russ <graeme.russ@gmail.com> wrote:
> Hi Simon,
>
> On 27/08/11 10:25, Simon Glass wrote:
>> Hi Mike,
>>
>> On Fri, Aug 26, 2011 at 1:55 PM, Mike Frysinger <vapier@gentoo.org> wrote:
>>> On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:
>
> [snip]
>
>>>> - I mean that the tftp command will 'obtain' a file when it asks for
>>>> one, although the actual Ethernet layer is mocked and doesn't actually
>>>> go out on the wire. Imagine an Ethernet driver which has a half-baked
>>>> tftp server in it. Yes I also see value in actually using machine
>>>> interfaces since the testing can be more thorough.
>>>
>>> why not just build on top of tun/tap ? ?then we do get "real" network traffic,
>>> and you dont have to write your own tftp server because you can simply use the
>>> same exact one on your development machine that the board would connect to.
>>> -mike
>>
>> Because then you need to set up a real tftp server. It's fine to do
>> what you suggest, but if possible it would be nice to have
>> self-contained tests also, so long as it isn't too much work.
>>
>
> I don't consider having to set up a tftp server as a bad thing - Quite the
> opposite really. There is plenty of network code in U-Boot that will
> benefit from testing under the sandbox target because it is much easier to
> debug. And there will be minimal impact on U-Boot code (just a sandbox
> 'Ethernet' driver is all that will be needed)
>
> I am reminded of when the R&D department that developed to eNET board were
> doing the firmware development before the first prototypes were available -
> The created a HAL which used pcap from memory. I wasn't part of that team,
> so I really don't know the details...

Yes, sounds good, will put you down for the network testing part :-)

I will work on a basic patch set and try to get it to the list early next week.

Regards,
Simon

>
> Regards,
>
> Graeme
>

[U-Boot] RFC: Testing U-Boot Part 1

[Graeme Russ]

Hi Simon,

On Tue, Aug 30, 2011 at 8:04 AM, Simon Glass <sjg@chromium.org> wrote:
> Hi Graeme,
>
> On Fri, Aug 26, 2011 at 7:23 PM, Graeme Russ <graeme.russ@gmail.com> wrote:
>> Hi Simon,
>>
>> On 27/08/11 10:25, Simon Glass wrote:
>>> Hi Mike,
>>>
>>> On Fri, Aug 26, 2011 at 1:55 PM, Mike Frysinger <vapier@gentoo.org> wrote:
>>>> On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:

[snip]

>>
>> I don't consider having to set up a tftp server as a bad thing - Quite the
>> opposite really. There is plenty of network code in U-Boot that will
>> benefit from testing under the sandbox target because it is much easier to
>> debug. And there will be minimal impact on U-Boot code (just a sandbox
>> 'Ethernet' driver is all that will be needed)
>>
>> I am reminded of when the R&D department that developed to eNET board were
>> doing the firmware development before the first prototypes were available -
>> The created a HAL which used pcap from memory. I wasn't part of that team,
>> so I really don't know the details...
>
> Yes, sounds good, will put you down for the network testing part :-)

Arghh - Rope, meet neck. Neck, meet rope ;)

>
> I will work on a basic patch set and try to get it to the list early next week.
>

Sounds good

Regards,

Graeme