Re: Mounting a in-ROM filesystem efficiently

Re: Mounting a in-ROM filesystem efficiently

[Richard B. Johnson]

On Fri, 14 Dec 2001, Dominik Kubla wrote:

> On Thu, Dec 13, 2001 at 01:34:45PM -0500, Richard B. Johnson wrote:
> > 
> > Well RAM is a hell of a lot cheaper than NVRAM. If you don't have
> > the required RAM on your box, the hardware engineers screwed up
> > and have to be "educated" preferably with an axe in the parking-lot.
> > 
> 
> What about security issues?  I can imagine quite a few scenarios where
> you would want to insure that you run untampered binaries. (eg. use
> ROM instead of the usual CD-ROM or read-only FD to run your security
> critical application.)
> 

I never even implied that you would run CD-ROM or FD in embedded
applications. The stuff that runs comes-from ROM, actually NVRAM so
in can be written/updated in production. However, EIP software
(Execute In Place) has always been a dog and, even Ethernet switches
3COM, Cisco, etc., don't run that way. You need to get the stuff
that runs into RAM.

Security isn't a problem with embedded systems because everything
that could possibly be done is handled with a "monitor". There is
no shell. If there is no way to execute some foreign executable,
you don't have a security issue unless some dumb alleged software
engineer added some back-doors to the monitor.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
 Santa Claus is coming to town...
          He knows if you've been sleeping,
             He knows if you're awake;
          He knows if you've been bad or good,
             So he must be Attorney General Ashcroft.

Re: Mounting a in-ROM filesystem efficiently

[Helge Hafting]

"Richard B. Johnson" wrote:

> 
> Security isn't a problem with embedded systems because everything
> that could possibly be done is handled with a "monitor". There is
> no shell. If there is no way to execute some foreign executable,
> you don't have a security issue unless some dumb alleged software
> engineer added some back-doors to the monitor.

A hacker don't need a /bin/sh or any other onboard software
to exploit some security flaw.  Assume someone discover that
your embedded box is vulnerable to a buffer overflow attack
of the type usually used to get a root shell.  Then they
discover that running /bin/sh don't work.  What to do?  They
simply put a simple little shell _in_ the buffer overflow
code itself.  A hacker don't need to call anything, all he need
can be downloaded as part of the exploit code.  

If the room for exploit code is thight - use a two-stage approach.
The exploit then consists of code that download the rest of the
code into some other RAM outside the tiny buffer.

No "dangerous" utilities on board doesn't mean the box is safe at
all.  The buffer overflow code could contain code for 
continuing the attack on other boxes, or anything else.

Helge Hafting

Re: Mounting a in-ROM filesystem efficiently

[Richard B. Johnson]

On Tue, 18 Dec 2001, Helge Hafting wrote:

> "Richard B. Johnson" wrote:
> 
> > 
> > Security isn't a problem with embedded systems because everything
> > that could possibly be done is handled with a "monitor". There is
> > no shell. If there is no way to execute some foreign executable,
> > you don't have a security issue unless some dumb alleged software
> > engineer added some back-doors to the monitor.
> 
> A hacker don't need a /bin/sh or any other onboard software
> to exploit some security flaw.  Assume someone discover that
> your embedded box is vulnerable to a buffer overflow attack
> of the type usually used to get a root shell.  Then they
> discover that running /bin/sh don't work.  What to do?  They
> simply put a simple little shell _in_ the buffer overflow
> code itself.  A hacker don't need to call anything, all he need
> can be downloaded as part of the exploit code.  
> 
> If the room for exploit code is thight - use a two-stage approach.
> The exploit then consists of code that download the rest of the
> code into some other RAM outside the tiny buffer.
> 
> No "dangerous" utilities on board doesn't mean the box is safe at
> all.  The buffer overflow code could contain code for 
> continuing the attack on other boxes, or anything else.
> 
> Helge Hafting


   You apparently don't know what an embedded system does.
   It is a device that uses a processor to perform some
   designed functions. It cannot do something that it
   was not designed to do although it can certainly fail
   to do what it was designed to do.

   If you want to break it, it's easier to hit it with a
   hammer or an axe. There is not any capability to "break in".
   Even if there was, what could you do? Shut off a motor?
   Screw up the ignition timing? Put porn pictures into
   medical images?

   Most embedded systems don't have network capabilities.
   Those that do are like your network printer. It's possible
   to send it junk. It's possible to send 20,000 form-feeds
   so it runs out of paper, but basically, you are just
   creating havoc with something you, or your company, owns.

   There is no way that you can teach your Hewlett-Packard
   printer to become a network rogue and break into the
   Pentagon --regardless of what you send it.

   Embedded systems that perform critical functions such
   as FMS (Flight Management Systems) have a reset button.
   If it's screwing up, and they often do, the pilot not
   flying says some four-letter prayers to be picked up
   by the cockpit microphone, hits the reset switch, waits
   for it to re-boot, and enters everything from scratch
   again --usually the entire day's flight-plan routes,
   taking nearly 1,000 key-strokes. Now, that's an embedded
   system.
 

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (797.90 BogoMips).
 Santa Claus is coming to town...
          He knows if you've been sleeping,
             He knows if you're awake;
          He knows if you've been bad or good,
             So he must be Attorney General Ashcroft.

Re: Mounting a in-ROM filesystem efficiently

[Alan Cox]

>    There is no way that you can teach your Hewlett-Packard
>    printer to become a network rogue and break into the
>    Pentagon --regardless of what you send it.

Really. Tell that to the people who already have a full Linux system running
on HP laser printers

Re: Mounting a in-ROM filesystem efficiently

[Mr. James W. Laferriere]

	Hello Alan ,  Got a URL ?-)  Verry interested in this . Tia ,  JimL

On Tue, 18 Dec 2001, Alan Cox wrote:

> >    There is no way that you can teach your Hewlett-Packard
> >    printer to become a network rogue and break into the
> >    Pentagon --regardless of what you send it.
>
> Really. Tell that to the people who already have a full Linux system running
> on HP laser printers
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

       +------------------------------------------------------------------+
       | James   W.   Laferriere | System    Techniques | Give me VMS     |
       | Network        Engineer |     P.O. Box 854     |  Give me Linux  |
       | babydr@baby-dragons.com | Coudersport PA 16915 |   only  on  AXP |
       +------------------------------------------------------------------+

Re: Mounting a in-ROM filesystem efficiently

[H. Peter Anvin]

Followup to:  <200112181409.fBIE9cV15431@pinkpanther.swansea.linux.org.uk>
By author:    Alan Cox <alan@lxorguk.ukuu.org.uk>
In newsgroup: linux.dev.kernel
>
> >    There is no way that you can teach your Hewlett-Packard
> >    printer to become a network rogue and break into the
> >    Pentagon --regardless of what you send it.
> 
> Really. Tell that to the people who already have a full Linux system running
> on HP laser printers
> 

My home HP laser printer has a web server running on it -- to save the
couple of $$ it would cost to put an alphanumeric display on it.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Mounting a in-ROM filesystem efficiently

[Kent Borg]

On Tue, Dec 18, 2001 at 09:00:58AM -0500, Richard B. Johnson wrote:
> On Tue, 18 Dec 2001, Helge Hafting wrote:
> > A hacker don't need a /bin/sh or any other onboard software
> > to exploit some security flaw.
[...]
>    You apparently don't know what an embedded system does.
>    It is a device that uses a processor to perform some
>    designed functions. It cannot do something that it
>    was not designed to do although it can certainly fail
>    to do what it was designed to do.

If it contains a CPU (that is designed to run code) and RAM (that is
designed to store code) and you can trick the CPU in running code you
tricked it into putting in the RAM, then it can do anything it wants
with the other hardware available, anything it is designed to do.

>    If you want to break it, it's easier to hit it with a
>    hammer or an axe. There is not any capability to "break in".
>    Even if there was, what could you do? Shut off a motor?
>    Screw up the ignition timing? Put porn pictures into
>    medical images?

Or, be a proxy on the inside of J. Random Paranoid Corporate Network
that can relay stuff, snoop, sniff interesting stuff that is being
printed, etc.  I mean, how about establishing an http connection out
to a computer controlled by the Bad Guy, and over that he tunnels
whatever he wants?  And maybe it gets setup by putting some rogue
Postscript in a file he somehow conspires to be printed.

>    Most embedded systems don't have network capabilities.

Certainly if the embedded system has extremely limited IO (Coke
vending machine?) there are limited opportunities for exploiting
things like buffer overflows.

But even Coke machines are starting to get IO: to report inventory and
apparent function to the vending company, some to even allow payment
via a cellphone.  Don't think that embedded == isolated, for it is
becoming less and less true.

My cellphone has embedded software in it, it can receive e-mail.
(There was a recent story of a mal formed e-mail message that would
hard-crash some Nokia phones such even a power cycle wouldn't fix it!)

>    There is no way that you can teach your Hewlett-Packard
>    printer to become a network rogue and break into the
>    Pentagon --regardless of what you send it.

Boy, are you ever complacent.  Just because HP manages to largely
obscure the details of its internal CPU and RAM doesn't mean it ain't
there and potentially exploitable.

I have a friend who used to be really into desktop publishing,
Illustrator, his font collection, etc.  He frequently made the
distinction between a "hardware RIP" and a "software RIP".  His point
was actually that the embedded systems were better productized, but I
still corrected him and said they were both software, one was simply
embedded.  He never seemed to get the point, and you don't either.


-kb

Re: Mounting a in-ROM filesystem efficiently

[Herman Oosthuysen]

----- Original Message -----
From: Richard B. Johnson <root@chaos.analogic.com>
To: Helge Hafting <helgehaf@idb.hist.no>
Cc: <linux-kernel@vger.kernel.org>
Sent: Tuesday, December 18, 2001 7:00 AM
Subject: Re: Mounting a in-ROM filesystem efficiently


> On Tue, 18 Dec 2001, Helge Hafting wrote:
>
> > "Richard B. Johnson" wrote:
> >
> > >
> > > Security isn't a problem with embedded systems because everything
> > > that could possibly be done is handled with a "monitor". There is
> > > no shell. If there is no way to execute some foreign executable,
> > > you don't have a security issue unless some dumb alleged software
> > > engineer added some back-doors to the monitor.
> >
...
>
>    Embedded systems that perform critical functions such
>    as FMS (Flight Management Systems) have a reset button.
>    If it's screwing up, and they often do, the pilot not
>    flying says some four-letter prayers to be picked up
>    by the cockpit microphone, hits the reset switch, waits
>    for it to re-boot, and enters everything from scratch
>    again --usually the entire day's flight-plan routes,
>    taking nearly 1,000 key-strokes. Now, that's an embedded
>    system.
>
>
> Cheers,
> Dick Johnson
>
> Penguin : Linux version 2.4.1 on an i686 machine (797.90 BogoMips).
>  Santa Claus is coming to town...
>           He knows if you've been sleeping,
>              He knows if you're awake;
>           He knows if you've been bad or good,
>              So he must be Attorney General Ashcroft.
>
....
Hmm, there are lots of embedded things with network capabilities.  Think of
wireless access points, network routers and switches.  Even a network
printer can cause network chaos if it would start to chatter on the net.
Many of these things run FTP or HTTP servers for management purposes.

However, figuring out how to hack into a non-standard piece of hardware
using an unknown processor is a non-trivial problem - it is very difficult
even if you designed the thing yourself and has all the info, but could
therefore be done by a disgruntled ex-employee.

Exploiting well known network services  such as FTP and HTTP for dark
purposes certainly should be a serious concern to manufactureres of these
devices.
--
Herman Oosthuysen
Herman@WirelessNetworksInc.com
Suite 300, #3016, 5th Ave NE,
Calgary, Alberta, T2A 6K4, Canada
Phone: (403) 569-5688, Fax: (403) 235-3965

Mounting a in-ROM filesystem efficiently

[Thomas Capricelli]

Hello,

I'm looking for a way to put a filesystem into ROM.
Seems pretty trivial, isn't it ?

My understanding is (the way initrd does, and the way I do as of today)
* create a RAMDISK
* loads the data into ramdisk
* mount the ramdisk

problem is that I don't want to waste the RAM as the data in the ROM is 
already in the address space. (it's an embedded system, btw)

Speed is not an issue here. ROM access might be slower than RAM, it will 
always be so much quicker than a disk access. (wrong?)

Ideally, i would give address/length of the fs in ROM to a function, and I 
would get a ramdisk configured to read its data exactly there, and not in 
ram.

Any hint ?

I've tried to look in the different options from mainstream kernels and 
embedded-oriented kernels whithout success.


thanx,
Thomas
ps : i'm subscribed to lkml, no need to cc:

-- 
Thomas Capricelli <orzel@kde.org>
boson.eu.org, kvim, zetalinux

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

I have maintained, on and off, a patch to crafms that supports traditional
cramfs decompress-and-read/run-from-RAM, plus direct mmaping with no
decompression and read/run straight out of ROM:

    http://www.ltc.com/~brad/mips/cramfs-linear-root-xip-linux-2.4.9-2.diff

It includes a modification to mkcramfs to compress or not compress
individual files based on their +t mode setting.  Mkcramfs will leave +t
uncompressed in the cramfs image and cramfs will directly mmap them.

Please note that the patch is against 2.4.9.  I haven't tried to use it
since then.

Regards,
Brad

----- Original Message -----
From: "Thomas Capricelli" <orzel@kde.org>
To: <linux-kernel@vger.kernel.org>
Sent: Thursday, December 13, 2001 11:02 AM
Subject: Mounting a in-ROM filesystem efficiently


>
>
> Hello,
>
> I'm looking for a way to put a filesystem into ROM.
> Seems pretty trivial, isn't it ?
>
> My understanding is (the way initrd does, and the way I do as of today)
> * create a RAMDISK
> * loads the data into ramdisk
> * mount the ramdisk
>
> problem is that I don't want to waste the RAM as the data in the ROM is
> already in the address space. (it's an embedded system, btw)
>
> Speed is not an issue here. ROM access might be slower than RAM, it will
> always be so much quicker than a disk access. (wrong?)
>
> Ideally, i would give address/length of the fs in ROM to a function, and I
> would get a ramdisk configured to read its data exactly there, and not in
> ram.
>
> Any hint ?
>
> I've tried to look in the different options from mainstream kernels and
> embedded-oriented kernels whithout success.
>
>
> thanx,
> Thomas
> ps : i'm subscribed to lkml, no need to cc:
>
> --
> Thomas Capricelli <orzel@kde.org>
> boson.eu.org, kvim, zetalinux-
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Mounting a in-ROM filesystem efficiently

[Thomas Capricelli]

Thanks for the patch. I've thought about doing that, but it seemed to me it 
was something so much important that it would already be in some official 
kernel. 

Does it mean that NONE of the existing embedded linux is able to use a ROM 
directly as a filesystem ?? (either root fs or not)

I'm astonished. Is there some embedded-specific mailing list I'm not aware of?


Thanx,
Thomas


On Thursday 13 December 2001 17:22, Bradley D. LaRonde wrote:
> I have maintained, on and off, a patch to crafms that supports traditional
> cramfs decompress-and-read/run-from-RAM, plus direct mmaping with no
> decompression and read/run straight out of ROM:
>
>     http://www.ltc.com/~brad/mips/cramfs-linear-root-xip-linux-2.4.9-2.diff


> > I'm looking for a way to put a filesystem into ROM.
> > Seems pretty trivial, isn't it ?
> >
> > My understanding is (the way initrd does, and the way I do as of today)
> > * create a RAMDISK
> > * loads the data into ramdisk
> > * mount the ramdisk
> >
> > problem is that I don't want to waste the RAM as the data in the ROM is
> > already in the address space. (it's an embedded system, btw)
> >
> > Speed is not an issue here. ROM access might be slower than RAM, it will
> > always be so much quicker than a disk access. (wrong?)
> >
> > Ideally, i would give address/length of the fs in ROM to a function, and
> > I would get a ramdisk configured to read its data exactly there, and not
> > in ram.
> >
> > Any hint ?
> >
> > I've tried to look in the different options from mainstream kernels and
> > embedded-oriented kernels whithout success.

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message -----
From: "Thomas Capricelli" <orzel@kde.org>
To: <linux-kernel@vger.kernel.org>
Sent: Thursday, December 13, 2001 11:41 AM
Subject: Re: Mounting a in-ROM filesystem efficiently


> Does it mean that NONE of the existing embedded linux is able to use a ROM
> directly as a filesystem ?? (either root fs or not)

There is also the MTD block driver.

> I'm astonished. Is there some embedded-specific mailing list I'm not aware
of?

I don't think so.

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[Richard B. Johnson]

On Thu, 13 Dec 2001, Bradley D. LaRonde wrote:

> ----- Original Message -----
> From: "Thomas Capricelli" <orzel@kde.org>
> To: <linux-kernel@vger.kernel.org>
> Sent: Thursday, December 13, 2001 11:41 AM
> Subject: Re: Mounting a in-ROM filesystem efficiently
> 
> 
> > Does it mean that NONE of the existing embedded linux is able to use a ROM
> > directly as a filesystem ?? (either root fs or not)
> 

Generally, ROM based stuff is compressed before being written to
NVRAM. It's uncompressed into a RAM-Disk and the RAM-Disk is mounted.

That way, you can use, say, 2 megabytes of NVRAM to get a 10 to 20
megabyte root file-system. This also allows /tmp and /var/log to be
writable, which is a great help because the development environment 
closely approximates the run-time environment.

FYI, generally NVRAM access is sooooo slow. I don't think you'd
like to use it directly as a file-system and access-time will be
a problem unless you modify the kernel. 

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
 Santa Claus is coming to town...
          He knows if you've been sleeping,
             He knows if you're awake;
          He knows if you've been bad or good,
             So he must be Attorney General Ashcroft.

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message ----- 
From: "Richard B. Johnson" <root@chaos.analogic.com>
To: "Bradley D. LaRonde" <brad@ltc.com>
Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
Sent: Thursday, December 13, 2001 1:02 PM
Subject: Re: Mounting a in-ROM filesystem efficiently


> Generally, ROM based stuff is compressed before being written to
> NVRAM. It's uncompressed into a RAM-Disk and the RAM-Disk is mounted.
> 
> That way, you can use, say, 2 megabytes of NVRAM to get a 10 to 20
> megabyte root file-system. This also allows /tmp and /var/log to be
> writable, which is a great help because the development environment 
> closely approximates the run-time environment.

That's perfect if you have plenty of RAM to spare.

> FYI, generally NVRAM access is sooooo slow. I don't think you'd
> like to use it directly as a file-system and access-time will be
> a problem unless you modify the kernel. 

Modify the kernel how?

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[Richard B. Johnson]

On Thu, 13 Dec 2001, Bradley D. LaRonde wrote:
[SNIPPED...]

> Sent: Thursday, December 13, 2001 1:02 PM Subject: Re: Mounting a in-ROM
> filesystem efficiently
> 
> 
> > Generally, ROM based stuff is compressed before being written to >
> NVRAM. It's uncompressed into a RAM-Disk and the RAM-Disk is mounted.  >
> > That way, you can use, say, 2 megabytes of NVRAM to get a 10 to 20 >
> megabyte root file-system. This also allows /tmp and /var/log to be >
> writable, which is a great help because the development environment >
> closely approximates the run-time environment. 
> 
> That's perfect if you have plenty of RAM to spare. 
> 

Well RAM is a hell of a lot cheaper than NVRAM. If you don't have
the required RAM on your box, the hardware engineers screwed up
and have to be "educated" preferably with an axe in the parking-lot.

The machine configuration should NOT have NVRAM addressed as a
bunch of RAM occupying a lot of address space. It should be
an easily-decodable chunk (0x1000, 0x2000, 0x4000, 0x8000, etc.)
that is paged, under software control. The base address should
be wherever memory-mapped hardware for your architecture usually
is.

> > FYI, generally NVRAM access is sooooo slow. I don't think you'd
> > like to use it directly as a file-system and access-time will be
> > a problem unless you modify the kernel. 
> 
> Modify the kernel how?
>

You have to keep it from failing when it tries to update the access-time
of files/directories. This means that your read/only NVRAM/ROM must
either pretend that it can be written or the driver for the hardware
has to do the same.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
 Santa Claus is coming to town...
          He knows if you've been sleeping,
             He knows if you're awake;
          He knows if you've been bad or good,
             So he must be Attorney General Ashcroft.

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message -----
From: "Richard B. Johnson" <root@chaos.analogic.com>
To: "Bradley D. LaRonde" <brad@ltc.com>
Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
Sent: Thursday, December 13, 2001 1:34 PM
Subject: Re: Mounting a in-ROM filesystem efficiently


> On Thu, 13 Dec 2001, Bradley D. LaRonde wrote:
> [SNIPPED...]
>
> > Sent: Thursday, December 13, 2001 1:02 PM Subject: Re: Mounting a in-ROM
> > filesystem efficiently
> >
> >
> > > Generally, ROM based stuff is compressed before being written to >
> > NVRAM. It's uncompressed into a RAM-Disk and the RAM-Disk is mounted.  >
> > > That way, you can use, say, 2 megabytes of NVRAM to get a 10 to 20 >
> > megabyte root file-system. This also allows /tmp and /var/log to be >
> > writable, which is a great help because the development environment >
> > closely approximates the run-time environment.
> >
> > That's perfect if you have plenty of RAM to spare.
> >
>
> Well RAM is a hell of a lot cheaper than NVRAM. If you don't have
> the required RAM on your box, the hardware engineers screwed up
> and have to be "educated" preferably with an axe in the parking-lot.

As I mentioned before, there may be other-than-cost considerations for
choosing the amount of RAM on a box.  For example, low power consumption on
portable devices.  For another example, a huge ROM database that doesn't
need to be in RAM all at once.

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[Richard B. Johnson]

On Thu, 13 Dec 2001, Bradley D. LaRonde wrote:

> ----- Original Message -----
> From: "Richard B. Johnson" <root@chaos.analogic.com>
> To: "Bradley D. LaRonde" <brad@ltc.com>
> Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
> Sent: Thursday, December 13, 2001 1:34 PM
> Subject: Re: Mounting a in-ROM filesystem efficiently
> 
> 
> > On Thu, 13 Dec 2001, Bradley D. LaRonde wrote:
> > [SNIPPED...]
> >
> > > Sent: Thursday, December 13, 2001 1:02 PM Subject: Re: Mounting a in-ROM
> > > filesystem efficiently
> > >
> > >
> > > > Generally, ROM based stuff is compressed before being written to >
> > > NVRAM. It's uncompressed into a RAM-Disk and the RAM-Disk is mounted.  >
> > > > That way, you can use, say, 2 megabytes of NVRAM to get a 10 to 20 >
> > > megabyte root file-system. This also allows /tmp and /var/log to be >
> > > writable, which is a great help because the development environment >
> > > closely approximates the run-time environment.
> > >
> > > That's perfect if you have plenty of RAM to spare.
> > >
> >
> > Well RAM is a hell of a lot cheaper than NVRAM. If you don't have
> > the required RAM on your box, the hardware engineers screwed up
> > and have to be "educated" preferably with an axe in the parking-lot.
> 
> As I mentioned before, there may be other-than-cost considerations for
> choosing the amount of RAM on a box.  For example, low power consumption on
> portable devices.  For another example, a huge ROM database that doesn't
> need to be in RAM all at once.
> 
> Regards,
> Brad
> 

Then you make a block-device device-driver that extracts and uncompresses
each read from ROM/NVRAM upon demand. It pretends to write. The actual
data-storage device is still paged and it only writes to the caller's
buffer so it doesn't use any RAM for storage.

When writing the compressed NVRAM, one has to use only allocation-unit
sizes, which reduces the amount of compression possible. Basically,
if you have 2000 "blocks", you need to compress 2000 individual blocks
and keep an uncompressed list of each block offset somewhere. This
complication is one of the many reasons why the general solution
is to un-compress everything once into RAM on startup. 

There are many arguments, but I don't think power consumption is
one of them. Whatever they use for RAM on the palm machines allows
the machines to run a week on 4 'aa' -size batteries. Maybe they
grab kinetic energy from keystrokes using flea-generators ^;).

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
 Santa Claus is coming to town...
          He knows if you've been sleeping,
             He knows if you're awake;
          He knows if you've been bad or good,
             So he must be Attorney General Ashcroft.

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message -----
From: "Richard B. Johnson" <root@chaos.analogic.com>
To: "Bradley D. LaRonde" <brad@ltc.com>
Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
Sent: Thursday, December 13, 2001 2:41 PM
Subject: Re: Mounting a in-ROM filesystem efficiently


> On Thu, 13 Dec 2001, Bradley D. LaRonde wrote:
>
> > ----- Original Message -----
> > From: "Richard B. Johnson" <root@chaos.analogic.com>
> > To: "Bradley D. LaRonde" <brad@ltc.com>
> > Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
> > Sent: Thursday, December 13, 2001 1:34 PM
> > Subject: Re: Mounting a in-ROM filesystem efficiently
> > > Well RAM is a hell of a lot cheaper than NVRAM. If you don't have
> > > the required RAM on your box, the hardware engineers screwed up
> > > and have to be "educated" preferably with an axe in the parking-lot.
> >
> > As I mentioned before, there may be other-than-cost considerations for
> > choosing the amount of RAM on a box.  For example, low power consumption
on
> > portable devices.  For another example, a huge ROM database that doesn't
> > need to be in RAM all at once.
> >
> > Regards,
> > Brad
> >
>
> Then you make a block-device device-driver that extracts and uncompresses
> each read from ROM/NVRAM upon demand. It pretends to write. The actual
> data-storage device is still paged and it only writes to the caller's
> buffer so it doesn't use any RAM for storage.

Or you could use cramfs + the patch that I mentioned a few e-mails ago.  :-)

> There are many arguments, but I don't think power consumption is
> one of them. Whatever they use for RAM on the palm machines allows
> the machines to run a week on 4 'aa' -size batteries. Maybe they
> grab kinetic energy from keystrokes using flea-generators ^;).

No flea-generators that I know of.  :-)

SDRAM, even in self-refresh mode, does draw considerable current.  But then
again so does decompressing stuff from ROM all the time.

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[Pavel Machek]

Hi!

> > There are many arguments, but I don't think power consumption is
> > one of them. Whatever they use for RAM on the palm machines allows
> > the machines to run a week on 4 'aa' -size batteries. Maybe they
> > grab kinetic energy from keystrokes using flea-generators ^;).
> 
> No flea-generators that I know of.  :-)
> 
> SDRAM, even in self-refresh mode, does draw considerable current.  But then
> again so does decompressing stuff from ROM all the time.

Well.. at least decompressing from rom does not eat power when not doing
*something*.

-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: Mounting a in-ROM filesystem efficiently

[Catalin Marinas]

On Thu, 13 Dec 2001, Thomas Capricelli wrote:

> I'm astonished. Is there some embedded-specific mailing list I'm not aware of?

Try comp.os.linux.embedded

-- 
Catalin


_____________________________________________________________________
This message has been checked for all known viruses by the 
MessageLabs Virus Scanning Service. For further information visit
http://www.messagelabs.com/stats.asp

Re: Mounting a in-ROM filesystem efficiently

[David Woodhouse]

orzel@kde.org said:
>  Does it mean that NONE of the existing embedded linux is able to use
> a ROM  directly as a filesystem ?? (either root fs or not) 

Out of the box, no. XIP isn't that interesting. Most boxes have flash, and 
flash is more expensive than RAM - so compression is more useful than XIP 
in many cases. Obviously the two are mutually exclusive.

Writing to an XIP filesystem is fairly hard too.

--
dwmw2

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message -----
From: "David Woodhouse" <dwmw2@infradead.org>
To: "Thomas Capricelli" <orzel@kde.org>
Cc: <linux-kernel@vger.kernel.org>
Sent: Thursday, December 13, 2001 12:49 PM
Subject: Re: Mounting a in-ROM filesystem efficiently

> orzel@kde.org said:
> >  Does it mean that NONE of the existing embedded linux is able to use
> > a ROM  directly as a filesystem ?? (either root fs or not)
>
> Out of the box, no. XIP isn't that interesting. Most boxes have flash, and
> flash is more expensive than RAM

Dollar-wise, but not power-wise.

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[H. Peter Anvin]

Followup to:  <20011213160007.D998D23CCB@persephone.dmz.logatique.fr>
By author:    Thomas Capricelli <orzel@kde.org>
In newsgroup: linux.dev.kernel
> 
> Hello,
> 
> I'm looking for a way to put a filesystem into ROM.
> Seems pretty trivial, isn't it ?
> 
> My understanding is (the way initrd does, and the way I do as of today)
> * create a RAMDISK
> * loads the data into ramdisk
> * mount the ramdisk
> 
> problem is that I don't want to waste the RAM as the data in the ROM is 
> already in the address space. (it's an embedded system, btw)
> 
> Speed is not an issue here. ROM access might be slower than RAM, it will 
> always be so much quicker than a disk access. (wrong?)
> 

Frequently wrong.  Depending on the interface, ROM can be just
unbelievably slow.

> Ideally, i would give address/length of the fs in ROM to a function, and I 
> would get a ramdisk configured to read its data exactly there, and not in 
> ram.

The right thing for you to do is to write a block device driver, and
then mount that block device like any order device.  Your in-use data
will be copied to RAM (i.e. cached), but it can be dropped and
re-fetched as necessary.  This should be the desired behaviour.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message -----
From: "H. Peter Anvin" <hpa@zytor.com>
To: <linux-kernel@vger.kernel.org>
Sent: Thursday, December 13, 2001 3:38 PM
Subject: Re: Mounting a in-ROM filesystem efficiently


> Followup to:  <20011213160007.D998D23CCB@persephone.dmz.logatique.fr>
> By author:    Thomas Capricelli <orzel@kde.org>
> In newsgroup: linux.dev.kernel
>
> > Ideally, i would give address/length of the fs in ROM to a function, and
I
> > would get a ramdisk configured to read its data exactly there, and not
in
> > ram.
>
> The right thing for you to do is to write a block device driver, and
> then mount that block device like any order device.  Your in-use data
> will be copied to RAM (i.e. cached), but it can be dropped and
> re-fetched as necessary.  This should be the desired behaviour.

That's one way (q.v. MTD block device), but it adds an unecessary block
layer.

The cramfs patch I mentioned shuns the block layer and instead allows the fs
to read directly from ROM.

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[David Woodhouse]

brad@ltc.com said:
>  have maintained, on and off, a patch to crafms that supports
> traditional cramfs decompress-and-read/run-from-RAM, plus direct
> mmaping with no decompression and read/run straight out of ROM:

> http://www.ltc.com/~brad/mips/cramfs-linear-root-xip-linux-2.4.9-2.diff 

+	if (remap_page_range(vma->vm_start, KSEG0ADDR(address), length,
+			     vma->vm_page_prot))


Cute, but not very generic. The approach I was contemplating was to 
allocate a set of 'struct page's for the pages containing XIP data, then 
add those pages to the page cache manually on read_inode(). 

It's a shame that ->readpage() doesn't get to say 'actually I used my own 
page for that, I didn't want one allocated for me'.

Extending the MTD API to return a set of pages representing a particular
device, and handling the locking so that we don't try to write to the chips
while pages are mapped, will also be necessary if we want to do it with
flash chips that are used for anything else.

--
dwmw2

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message -----
From: "David Woodhouse" <dwmw2@infradead.org>
To: "Bradley D. LaRonde" <brad@ltc.com>
Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
Sent: Friday, December 14, 2001 4:45 AM
Subject: Re: Mounting a in-ROM filesystem efficiently


> brad@ltc.com said:
> >  have maintained, on and off, a patch to crafms that supports
> > traditional cramfs decompress-and-read/run-from-RAM, plus direct
> > mmaping with no decompression and read/run straight out of ROM:
>
> > http://www.ltc.com/~brad/mips/cramfs-linear-root-xip-linux-2.4.9-2.diff
>
> + if (remap_page_range(vma->vm_start, KSEG0ADDR(address), length,
> +      vma->vm_page_prot))
>
>
> Cute, but not very generic.

Yes, sorry, I should have mentioned that MIPSism in there.  I haven't tried,
but I imagine it wouldn't be that hard to adjust for other platforms.

(In case anyone isn't familiar with KSEG0ADDR on MIPS, it takes a physical
address as it's only paramter and returns a cacheable virtual address in
"kernel address segment 0" (0x80000000 - 0xafffffff) which is hardwire 1:1
mapped with physical address space).

> The approach I was contemplating was to
> allocate a set of 'struct page's for the pages containing XIP data, then
> add those pages to the page cache manually on read_inode().
>
> It's a shame that ->readpage() doesn't get to say 'actually I used my own
> page for that, I didn't want one allocated for me'.

That sounds nice, but I cannot imagine how much trouble it would be to
implement.

> Extending the MTD API to return a set of pages representing a particular
> device, and handling the locking so that we don't try to write to the
chips
> while pages are mapped, will also be necessary if we want to do it with
> flash chips that are used for anything else.

Actually, I've used that patch on a system that had a cramfs/xip and a jffs
partition on the same flash chip where the kernel was running xip out of
flash.  :-)

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[David Woodhouse]

brad@ltc.com said:
>  That sounds nice, but I cannot imagine how much trouble it would be
> to implement.

Adding the pages to the page cache on read_inode() is fairly simple. Hacking 
the kernel so that readpage() can provide its own page less so.

>  Actually, I've used that patch on a system that had a cramfs/xip and
> a jffs partition on the same flash chip where the kernel was running
> xip out of flash.  :-) 

S'OK if you have the right type of flash chips, I suppose :)

--
dwmw2

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message ----- 
From: "David Woodhouse" <dwmw2@infradead.org>
To: "Bradley D. LaRonde" <brad@ltc.com>
Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
Sent: Friday, December 14, 2001 11:51 AM
Subject: Re: Mounting a in-ROM filesystem efficiently 

> >  Actually, I've used that patch on a system that had a cramfs/xip and
> > a jffs partition on the same flash chip where the kernel was running
> > xip out of flash.  :-) 
> 
> S'OK if you have the right type of flash chips, I suppose :)

128 Mbit Intel StrataFlash.

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[David Woodhouse]

brad@ltc.com said:
>  128 Mbit Intel StrataFlash. 

Oh - I thought those started returning status bits from all addresses when
ever you were writing or erasing any part of them?

--
dwmw2

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message -----
From: "David Woodhouse" <dwmw2@infradead.org>
To: "Bradley D. LaRonde" <brad@ltc.com>
Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
Sent: Friday, December 14, 2001 12:03 PM
Subject: Re: Mounting a in-ROM filesystem efficiently


> brad@ltc.com said:
> >  128 Mbit Intel StrataFlash.
>
> Oh - I thought those started returning status bits from all addresses when
> ever you were writing or erasing any part of them?

They do.  On that system just the low-level flash write code was kept in
RAM, but the rest of the kernel was XIP from flash.

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[David Woodhouse]

brad@ltc.com said:
>  They do.  On that system just the low-level flash write code was kept
> in RAM, but the rest of the kernel was XIP from flash. 

Oh, right - so you run from RAM and disable interrupts while you're writing 
or erasing? That can work, but isn't an approach I really want to encourage :)



--
dwmw2

Re: Mounting a in-ROM filesystem efficiently

[Bradley D. LaRonde]

----- Original Message -----
From: "David Woodhouse" <dwmw2@infradead.org>
To: "Bradley D. LaRonde" <brad@ltc.com>
Cc: "Thomas Capricelli" <orzel@kde.org>; <linux-kernel@vger.kernel.org>
Sent: Friday, December 14, 2001 12:16 PM
Subject: Re: Mounting a in-ROM filesystem efficiently


>
> brad@ltc.com said:
> >  They do.  On that system just the low-level flash write code was kept
> > in RAM, but the rest of the kernel was XIP from flash.
>
> Oh, right - so you run from RAM and disable interrupts while you're
writing
> or erasing?

Correct.

Regards,
Brad

Re: Mounting a in-ROM filesystem efficiently

[Christoph Rohland]

Hi David,

On Fri, 14 Dec 2001, David Woodhouse wrote:
> Adding the pages to the page cache on read_inode() is fairly
> simple. Hacking the kernel so that readpage() can provide its own
> page less so.

But would make mm/shmem.c much simpler while adding functionality.

Greetings
		Christoph

Re: Mounting a in-ROM filesystem efficiently

[Eric W. Biederman]

David Woodhouse <dwmw2@infradead.org> writes:

> brad@ltc.com said:
> >  That sounds nice, but I cannot imagine how much trouble it would be
> > to implement.
> 
> Adding the pages to the page cache on read_inode() is fairly simple. Hacking 
> the kernel so that readpage() can provide its own page less so.

Well the generic solution is to simply skip readpage and provide (for your fs)
your own versions of generic_file_read and filemap_nopage.  At least
if you want to do it on demand...

Eric