XD/smartmedia - how to implement it right?

XD/smartmedia - how to implement it right?

[Maxim Levitsky]

Hi,

This thread is meant to continue discussion about XD/smartmedia support
in linux.

I CC'd few peoples that participated in original discussion.

Alex Dubov attempted to write a driver for jmicron device (I have it
btw), but this was rejected due to code duplication.

I successfully reverse engineered (in my free time) the XD card reader
found on my laptop (ricoh based one)

I would like to know how you think to implement the subsystem support
right.

I would go with mmc subsystem + FTL, but according to original thread,
there are numerous problems with this approach.

Any advice is appreciated.

Best regards,
Maxim Levitsky

Re: XD/smartmedia - how to implement it right?

[Jörn Engel]

On Sat, 21 November 2009 02:25:44 +0200, Maxim Levitsky wrote:
> 
> This thread is meant to continue discussion about XD/smartmedia support
> in linux.
> 
> I CC'd few peoples that participated in original discussion.
> 
> Alex Dubov attempted to write a driver for jmicron device (I have it
> btw), but this was rejected due to code duplication.
> 
> I successfully reverse engineered (in my free time) the XD card reader
> found on my laptop (ricoh based one)
> 
> I would like to know how you think to implement the subsystem support
> right.
> 
> I would go with mmc subsystem + FTL, but according to original thread,
> there are numerous problems with this approach.
> 
> Any advice is appreciated.

A novel approach might be to just send the code as a patch.  Last time
around noone seemed to have had a real look.

The "mtd uses blocking calls" argument I can help with.  For some other
project I've added non-blocking calls to mtd[1].

Biggest problems I see are a) the userspace interface and b) allowing
both raw flash access through MTD and block device access with an FTL.

[1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html

Jörn

-- 
The wise man seeks everything in himself; the ignorant man tries to get
everything from somebody else.
-- unknown

Re: XD/smartmedia - how to implement it right?

[Maxim Levitsky]

On Sat, 2009-11-21 at 11:25 +0100, Jörn Engel wrote: 
> On Sat, 21 November 2009 02:25:44 +0200, Maxim Levitsky wrote:
> > 
> > This thread is meant to continue discussion about XD/smartmedia support
> > in linux.
> > 
> > I CC'd few peoples that participated in original discussion.
> > 
> > Alex Dubov attempted to write a driver for jmicron device (I have it
> > btw), but this was rejected due to code duplication.
> > 
> > I successfully reverse engineered (in my free time) the XD card reader
> > found on my laptop (ricoh based one)
> > 
> > I would like to know how you think to implement the subsystem support
> > right.
> > 
> > I would go with mmc subsystem + FTL, but according to original thread,
> > there are numerous problems with this approach.
> > 
> > Any advice is appreciated.
> 
> A novel approach might be to just send the code as a patch.  Last time
> around noone seemed to have had a real look.
> 
> The "mtd uses blocking calls" argument I can help with.  For some other
> project I've added non-blocking calls to mtd[1].
> 
> Biggest problems I see are a) the userspace interface and b) allowing
> both raw flash access through MTD and block device access with an FTL.
> 
> [1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
> 
> Jörn


I don't have much to say right now, I will study the sources for a
while, and then continue the effort.


Best regards,
Maxim Levitsky

Plan for adding XD support in mtd layer

[Maxim Levitsky]

Here is my plan for adding the XD format into MTD subsystem.
Please review.

New additions are marked with '->'


Chip driver:

no problems with that, in fact ricoh controller I reverse engineered is
very similar conceptually
to jmicron.
I can take original jmicron driver almost verbatim.
This will live in new sub folder of mtd and call into base XD driver


Base XD mtd driver:

* No way to tell that readsize = 512, but write & erase size isn't.
Not such a big deal, since higher level FTL driver can hardcode that
assumption, and access via mtdblk isn't necessary, since FTL will
replace it for XD, and the raw access will be done through mtdchar.

* it is possible to export 'extra' via oob. FTL will use that, and will
be available to user via char device

* No need to export CIF block, it will be available for user via char
device by scanning the device (using a new utility)


-> I need new mtd_info member for at least for card identification, and
will be nice to have zone/block/page size
This will be used by FTL and mtdchar to be exported to user space


FTL driver:

I will write new XD/smartmedia FTL driver.

* Will read ID/type from device and do all the work based on that,
(figure page size/zone count/block size, etc...) or will get that info
from mtd driver 

* Will cache read/writes (this will overlap with mtdblk)

* Will provide normal block level access to filesystems, and user.
(using common block device of course)

* No debugging information or/and device specific ioctls.


Char driver:

* Already exists, and I will need to modify it, so I could read ID/media
type
Also would be nice to pass info about sizes (zones/pages/blocks) to
userspace 
If not, I will put translation table to userspace utility based on media
ID


Userspace utility:
* Will allow user to read FTL maps (based on oob), Media ID, CIF, all
using char device
* Will also allow low level format.



Best regards,
Maxim Levitsky

Re: Plan for adding XD support in mtd layer

[Jörn Engel]

On Wed, 25 November 2009 01:50:10 +0200, Maxim Levitsky wrote:
> 
> Chip driver:
> 
> no problems with that, in fact ricoh controller I reverse engineered is
> very similar conceptually
> to jmicron.
> I can take original jmicron driver almost verbatim.
> This will live in new sub folder of mtd and call into base XD driver

Ok.  Is the code available somewhere or can you send it?  And what
devices would one have to buy to test it?

> Base XD mtd driver:
> 
> * No way to tell that readsize = 512, but write & erase size isn't.
> Not such a big deal, since higher level FTL driver can hardcode that
> assumption, and access via mtdblk isn't necessary, since FTL will
> replace it for XD, and the raw access will be done through mtdchar.
> 
> * it is possible to export 'extra' via oob. FTL will use that, and will
> be available to user via char device
> 
> * No need to export CIF block, it will be available for user via char
> device by scanning the device (using a new utility)
> 
> 
> -> I need new mtd_info member for at least for card identification, and
> will be nice to have zone/block/page size
> This will be used by FTL and mtdchar to be exported to user space

Block and page size already exist as mtd->erasesize and mtd->writesize
respectively.  I'm not sure whether the zone size should become part of
the mtd interface - it looks more like a property of the ftl.

> FTL driver:
> 
> I will write new XD/smartmedia FTL driver.
> 
> * Will read ID/type from device and do all the work based on that,
> (figure page size/zone count/block size, etc...) or will get that info
> from mtd driver 
> 
> * Will cache read/writes (this will overlap with mtdblk)
> 
> * Will provide normal block level access to filesystems, and user.
> (using common block device of course)
> 
> * No debugging information or/and device specific ioctls.
> 
> 
> Char driver:
> 
> * Already exists, and I will need to modify it, so I could read ID/media
> type
> Also would be nice to pass info about sizes (zones/pages/blocks) to
> userspace 
> If not, I will put translation table to userspace utility based on media
> ID

What userspace programs would need this information?  I can imagine some
sort of xd_format or mkxd in case the card has been corrupted or used
raw.  Anything else?

> Userspace utility:
> * Will allow user to read FTL maps (based on oob), Media ID, CIF, all
> using char device

What would this be needed for?

> * Will also allow low level format.

Jörn

-- 
Beware of bugs in the above code; I have only proved it correct, but
not tried it.
-- Donald Knuth

Re: Plan for adding XD support in mtd layer

[Maxim Levitsky]

On Wed, 2009-11-25 at 11:40 +0100, Jörn Engel wrote: 
> On Wed, 25 November 2009 01:50:10 +0200, Maxim Levitsky wrote:
> > 
> > Chip driver:
> > 
> > no problems with that, in fact ricoh controller I reverse engineered is
> > very similar conceptually
> > to jmicron.
> > I can take original jmicron driver almost verbatim.
> > This will live in new sub folder of mtd and call into base XD driver
> 
> Ok.  Is the code available somewhere or can you send it?  And what
> devices would one have to buy to test it?
Currently jmicron support is found in aspire one.
My acer has ricoh device I know how works.
Alex dubov said that TI has interest in a driver.
As soon as core is written and accepted, writing drivers will be very
easy if one has documentation.

I also plan to write a 'dummy' driver to test subsystem against.

Other that work of Alex Dubov, I need to write the code, but I don't
want to repeat his mistake and write code that isn't accepted on basis
of lack for fitness in existing subsystems.

> 
> > Base XD mtd driver:
> > 
> > * No way to tell that readsize = 512, but write & erase size isn't.
> > Not such a big deal, since higher level FTL driver can hardcode that
> > assumption, and access via mtdblk isn't necessary, since FTL will
> > replace it for XD, and the raw access will be done through mtdchar.
> > 
> > * it is possible to export 'extra' via oob. FTL will use that, and will
> > be available to user via char device
> > 
> > * No need to export CIF block, it will be available for user via char
> > device by scanning the device (using a new utility)
> > 
> > 
> > -> I need new mtd_info member for at least for card identification, and
> > will be nice to have zone/block/page size
> > This will be used by FTL and mtdchar to be exported to user space
> 
> Block and page size already exist as mtd->erasesize and mtd->writesize
> respectively.  I'm not sure whether the zone size should become part of
> the mtd interface - it looks more like a property of the ftl.
You didn't understand me.
I can't nether write nor erase individual blocks (writing is technically
possible, but is disallowed by recent XD spec (this is what Alex Dubov
says)

However I can _read_ individual sectors
sector is 512 bytes, block is usually around 32 sectors.


> 
> > FTL driver:
> > 
> > I will write new XD/smartmedia FTL driver.
> > 
> > * Will read ID/type from device and do all the work based on that,
> > (figure page size/zone count/block size, etc...) or will get that info
> > from mtd driver 
> > 
> > * Will cache read/writes (this will overlap with mtdblk)
> > 
> > * Will provide normal block level access to filesystems, and user.
> > (using common block device of course)
> > 
> > * No debugging information or/and device specific ioctls.
> > 
> > 
> > Char driver:
> > 
> > * Already exists, and I will need to modify it, so I could read ID/media
> > type
> > Also would be nice to pass info about sizes (zones/pages/blocks) to
> > userspace 
> > If not, I will put translation table to userspace utility based on media
> > ID
> 
> What userspace programs would need this information?  I can imagine some
> sort of xd_format or mkxd in case the card has been corrupted or used
> raw.  Anything else?

I can imagine a format application, a rescue application, and an debug
application that will show how healthy is the card.

There are actually two pieces of the data.
One is an ID, a serial number that can be used for example by udev for
permanent mapping.

Other is the media type, and this has to be used to know the sizes of
sectors, zones, pages, how many spares there are etc...



> 
> > Userspace utility:
> > * Will allow user to read FTL maps (based on oob), Media ID, CIF, all
> > using char device
> 
> What would this be needed for?
A rescue operation, of just for curiosity...
Also to edit CIF to bypass vendor lock-in as explained in Wikipedia
article

Access via mtdblock can be used to use the card as raw device too.

ECC corecting will be done in the base mtd driver, but userspace
application will be able to determine if block is good or bad.
If it is good, it can use oob information to determine of ECC correction
was applied or not, and report overall health of the card.


Thus I think that XD can be integrated in the core quite easily with
only minor modifications.


Best regards,
Maxim Levitsky

Re: Plan for adding XD support in mtd layer

[Jörn Engel]

Hello Maxim

On Wed, 25 November 2009 15:20:58 +0200, Maxim Levitsky wrote:
>
> Currently jmicron support is found in aspire one.
> My acer has ricoh device I know how works.

Those are card readers built into notebooks, right?  How are they
attached?  Via PCI or USB?

> Other that work of Alex Dubov, I need to write the code, but I don't
> want to repeat his mistake and write code that isn't accepted on basis
> of lack for fitness in existing subsystems.


> > > -> I need new mtd_info member for at least for card identification, and
> > > will be nice to have zone/block/page size
> > > This will be used by FTL and mtdchar to be exported to user space
> > 
> > Block and page size already exist as mtd->erasesize and mtd->writesize
> > respectively.  I'm not sure whether the zone size should become part of
> > the mtd interface - it looks more like a property of the ftl.
> You didn't understand me.
> I can't nether write nor erase individual blocks (writing is technically
> possible, but is disallowed by recent XD spec (this is what Alex Dubov
> says)
> 
> However I can _read_ individual sectors
> sector is 512 bytes, block is usually around 32 sectors.

It has been a while since I looked at the spec, but I believe there are
two parts to it.  The flash hardware allows block erases, sector reads
and sector writes.  On top of that you have the FTL that adds further
restrictions.

If you use you xD card without the FTL, as a raw MTD device, you can do
almost anything you want.  Format it with JFFS2, for example.  But if
used _with_ the FTL, that card now has a corrupt format and may be
unusable.

Or is the FTL already implemented in the card reader hardware?

> > What userspace programs would need this information?  I can imagine some
> > sort of xd_format or mkxd in case the card has been corrupted or used
> > raw.  Anything else?
> 
> I can imagine a format application, a rescue application, and an debug
> application that will show how healthy is the card.

Format and rescue certainly make sense.  I'm not sure how a debug
application would measure the health, but maybe that's possible as well.

> There are actually two pieces of the data.
> One is an ID, a serial number that can be used for example by udev for
> permanent mapping.

Ok, makes sense.

> Other is the media type, and this has to be used to know the sizes of
> sectors, zones, pages, how many spares there are etc...

That would be available when using mtdchar or mtdblock to read the
device.  Which ought to be enough for format, rescue and debug, I
believe.

> ECC corecting will be done in the base mtd driver, but userspace
> application will be able to determine if block is good or bad.
> If it is good, it can use oob information to determine of ECC correction
> was applied or not, and report overall health of the card.

With mtdchar it is possible to do ECC correction in userspace as well.
Not that I would consider it a good idea, but the information is
certainly available and can be controlled if a compelling reason can be
found.

Jörn

-- 
Invincibility is in oneself, vulnerability is in the opponent.
-- Sun Tzu

Re: Plan for adding XD support in mtd layer

[Maxim Levitsky]

On Wed, 2009-11-25 at 16:34 +0100, Jörn Engel wrote: 
> Hello Maxim
> 
> On Wed, 25 November 2009 15:20:58 +0200, Maxim Levitsky wrote:
> >
> > Currently jmicron support is found in aspire one.
> > My acer has ricoh device I know how works.
> 
> Those are card readers built into notebooks, right?  How are they
> attached?  Via PCI or USB?
Mostly in notebooks.
External readers usually implement standard mass-storage protocol.

They are usually PCI , however some USB devices work the same, for
example
alluda device that currently has two drivers in the kernel, one mtd for
raw access, and another self-contained USB driver that does FTL and
exposes device as an SCSI device.

> 
> > Other that work of Alex Dubov, I need to write the code, but I don't
> > want to repeat his mistake and write code that isn't accepted on basis
> > of lack for fitness in existing subsystems.
> 
> 
> > > > -> I need new mtd_info member for at least for card identification, and
> > > > will be nice to have zone/block/page size
> > > > This will be used by FTL and mtdchar to be exported to user space
> > > 
> > > Block and page size already exist as mtd->erasesize and mtd->writesize
> > > respectively.  I'm not sure whether the zone size should become part of
> > > the mtd interface - it looks more like a property of the ftl.
> > You didn't understand me.
> > I can't nether write nor erase individual blocks (writing is technically
> > possible, but is disallowed by recent XD spec (this is what Alex Dubov
> > says)
> > 
> > However I can _read_ individual sectors
> > sector is 512 bytes, block is usually around 32 sectors.
> 
> It has been a while since I looked at the spec, but I believe there are
> two parts to it.  The flash hardware allows block erases, sector reads
> and sector writes.  On top of that you have the FTL that adds further
> restrictions.
if this is the case, then no problems.
Alex said something about firmware bugs though, that I belive is
connected to raw block writes.

I also know that driver for my ricoh based controller does always block
based cow writes.

when I say block I mean area that can be erased independently, and it
consists of several 512 byte sectors.
(Very old incarnation of smartmedia format had 256 byte sectors that
were tied together in pairs)

> 
> If you use you xD card without the FTL, as a raw MTD device, you can do
> almost anything you want.  Format it with JFFS2, for example.  But if
> used _with_ the FTL, that card now has a corrupt format and may be
> unusable.
> 
> Or is the FTL already implemented in the card reader hardware?
No, I have to implement it.
And yes, XD can give ready to use testing ground for flash based file
systems, like JFFS2.

> 
> > > What userspace programs would need this information?  I can imagine some
> > > sort of xd_format or mkxd in case the card has been corrupted or used
> > > raw.  Anything else?
> > 
> > I can imagine a format application, a rescue application, and an debug
> > application that will show how healthy is the card.
> 
> Format and rescue certainly make sense.  I'm not sure how a debug
> application would measure the health, but maybe that's possible as well.
I can count bad sectors, ecc corrected sectors, see how many spares
remain in each block, etc...

> 
> > There are actually two pieces of the data.
> > One is an ID, a serial number that can be used for example by udev for
> > permanent mapping.
> 
> Ok, makes sense.
> 
> > Other is the media type, and this has to be used to know the sizes of
> > sectors, zones, pages, how many spares there are etc...
> 
> That would be available when using mtdchar or mtdblock to read the
> device.  Which ought to be enough for format, rescue and debug, I
> believe.

Yes this is enough, but how I expose media type in the mtd driver, there
is a field for that I missed?

And same question for the ID.

Note that media type /ID is _not_ stored on the media (well it is but is
not accessible that way)
There is need for special command to read it from the device.

> 
> > ECC corecting will be done in the base mtd driver, but userspace
> > application will be able to determine if block is good or bad.
> > If it is good, it can use oob information to determine of ECC correction
> > was applied or not, and report overall health of the card.
> 
> With mtdchar it is possible to do ECC correction in userspace as well.
> Not that I would consider it a good idea, but the information is
> certainly available and can be controlled if a compelling reason can be
> found.
Exactly.


Best regards,
Maxim Levitsky

Re: Plan for adding XD support in mtd layer

[Jörn Engel]

On Wed, 25 November 2009 18:17:46 +0200, Maxim Levitsky wrote:
> On Wed, 2009-11-25 at 16:34 +0100, Jörn Engel wrote: 
> > 
> > Those are card readers built into notebooks, right?  How are they
> > attached?  Via PCI or USB?
> Mostly in notebooks.
> External readers usually implement standard mass-storage protocol.

Ok.  I guess I won't buy a new notebook just to test a driver then.

> They are usually PCI , however some USB devices work the same, for
> example
> alluda device that currently has two drivers in the kernel, one mtd for
> raw access, and another self-contained USB driver that does FTL and
> exposes device as an SCSI device.

Yes.  I've written the mtd driver for that one.

> > It has been a while since I looked at the spec, but I believe there are
> > two parts to it.  The flash hardware allows block erases, sector reads
> > and sector writes.  On top of that you have the FTL that adds further
> > restrictions.
> if this is the case, then no problems.
> Alex said something about firmware bugs though, that I belive is
> connected to raw block writes.
> 
> I also know that driver for my ricoh based controller does always block
> based cow writes.

If you only ever want to use the device with the FTL on top, that is
perfectly acceptable.  But it is an FTL restriction, not part of the
hardware.  Apart from firmware bugs, of course.

> when I say block I mean area that can be erased independently, and it
> consists of several 512 byte sectors.
> (Very old incarnation of smartmedia format had 256 byte sectors that
> were tied together in pairs)

Some also had 256 or 512 blocks per zone.  These days everything seems
to have 512+16 Byte sectors, 16 sectors per block, 1024 physical (1000
logical) blocks per zone and as many zones as the device can fit.  Or at
least that was the case two years ago.

> > Format and rescue certainly make sense.  I'm not sure how a debug
> > application would measure the health, but maybe that's possible as well.
> I can count bad sectors, ecc corrected sectors, see how many spares
> remain in each block, etc...

Iirc the number of bad blocks never changes.  Either a block is
factory-marked or it will stay in use forever.  A smart FTL could try to
store the erase count or the number of correctable/uncorrectable errors
for each block somewhere, but I don't believe it is done - or could be
done while staying compatible with the xD spec.

So that pretty much leaves the number of correctable/uncorrectable
errors currently on the device.

> > That would be available when using mtdchar or mtdblock to read the
> > device.  Which ought to be enough for format, rescue and debug, I
> > believe.
> 
> Yes this is enough, but how I expose media type in the mtd driver, there
> is a field for that I missed?
> 
> And same question for the ID.
> 
> Note that media type /ID is _not_ stored on the media (well it is but is
> not accessible that way)
> There is need for special command to read it from the device.

Right.  I forgot about that one.  In fact, I tried fairly hard to
forget. ;)

According to FMTP140E.PDF, CIS and IDI are stored in sector 0 or the
first non-defective block.  In other words, it can be read and written
normally, without any magic.  Certainly not the atrocities I tried not
to remember.

And even if there are some atrocities hidden, we can abtract them out in
the driver.  When reading/writing block 0, just do the magic
transparently.

Jörn

-- 
There are two ways of constructing a software design: one way is to make
it so simple that there are obviously no deficiencies, and the other is
to make it so complicated that there are no obvious deficiencies.
-- C. A. R. Hoare

Re: Plan for adding XD support in mtd layer

[Maxim Levitsky]

On Wed, 2009-11-25 at 21:59 +0100, Jörn Engel wrote: 
> On Wed, 25 November 2009 18:17:46 +0200, Maxim Levitsky wrote:
> > On Wed, 2009-11-25 at 16:34 +0100, Jörn Engel wrote: 
> > > 
> > > Those are card readers built into notebooks, right?  How are they
> > > attached?  Via PCI or USB?
> > Mostly in notebooks.
> > External readers usually implement standard mass-storage protocol.
> 
> Ok.  I guess I won't buy a new notebook just to test a driver then.
> 
> > They are usually PCI , however some USB devices work the same, for
> > example
> > alluda device that currently has two drivers in the kernel, one mtd for
> > raw access, and another self-contained USB driver that does FTL and
> > exposes device as an SCSI device.
> 
> Yes.  I've written the mtd driver for that one.
> 
> > > It has been a while since I looked at the spec, but I believe there are
> > > two parts to it.  The flash hardware allows block erases, sector reads
> > > and sector writes.  On top of that you have the FTL that adds further
> > > restrictions.
> > if this is the case, then no problems.
> > Alex said something about firmware bugs though, that I belive is
> > connected to raw block writes.
> > 
> > I also know that driver for my ricoh based controller does always block
> > based cow writes.
> 
> If you only ever want to use the device with the FTL on top, that is
> perfectly acceptable.  But it is an FTL restriction, not part of the
> hardware.  Apart from firmware bugs, of course.
Exactly.

> 
> > when I say block I mean area that can be erased independently, and it
> > consists of several 512 byte sectors.
> > (Very old incarnation of smartmedia format had 256 byte sectors that
> > were tied together in pairs)
> 
> Some also had 256 or 512 blocks per zone.  These days everything seems
> to have 512+16 Byte sectors, 16 sectors per block, 1024 physical (1000
> logical) blocks per zone and as many zones as the device can fit.  Or at
> least that was the case two years ago.
Yes that the same I see.

> 
> > > Format and rescue certainly make sense.  I'm not sure how a debug
> > > application would measure the health, but maybe that's possible as well.
> > I can count bad sectors, ecc corrected sectors, see how many spares
> > remain in each block, etc...
> 
> Iirc the number of bad blocks never changes.  Either a block is
> factory-marked or it will stay in use forever.  A smart FTL could try to
> store the erase count or the number of correctable/uncorrectable errors
> for each block somewhere, but I don't believe it is done - or could be
> done while staying compatible with the xD spec.

> 
> So that pretty much leaves the number of correctable/uncorrectable
> errors currently on the device.
Exactly.

> 
> > > That would be available when using mtdchar or mtdblock to read the
> > > device.  Which ought to be enough for format, rescue and debug, I
> > > believe.
> > 
> > Yes this is enough, but how I expose media type in the mtd driver, there
> > is a field for that I missed?
> > 
> > And same question for the ID.
> > 
> > Note that media type /ID is _not_ stored on the media (well it is but is
> > not accessible that way)
> > There is need for special command to read it from the device.
> 
> Right.  I forgot about that one.  In fact, I tried fairly hard to
> forget. ;)
> 
> According to FMTP140E.PDF, CIS and IDI are stored in sector 0 or the
> first non-defective block.  In other words, it can be read and written
> normally, without any magic.  Certainly not the atrocities I tried not
> to remember.
I wish I had the spec.
I don't know the format of the CIS, and I know that special command is
used to query the ID.
CIS these days isn't used for anything and I hoped not to touch that
thing.

I wont argue on that one with you, but why not to add the new field.
It is quite generic, it could be card vendor information or so?
Few bits of custom information to assign for each card no?


I am taking about these:

struct xd_card_id1 {
unsigned char maker_code;
unsigned char device_code;
unsigned char option_code1;
unsigned char option_code2;
} __attribute__((packed));

struct xd_card_id2 {
unsigned char characteristics_code;
#define XD_CARD_CELL_TYPE_MASK   0xc0
#define XD_CARD_MULTI_BLOCK_MASK 0x30

unsigned char vendor_code1;
unsigned char size_code;
unsigned char vendor_code2;
unsigned char vendor_code3;
} __attribute__((packed));

struct xd_card_id3 {
unsigned char vendor_code1;
unsigned char vendor_code2;
unsigned char id_code;
unsigned char vendor_code3;
} __attribute__((packed));


Are these 32 bit values copied from CIF?

These values are read using commands:

XD_CARD_CMD_ID1         = 0x90,
XD_CARD_CMD_ID2         = 0x91,
XD_CARD_CMD_ID3         = 0x9a


I need especially the xd_card_id1.device_code because this specifies the
media type, but xd_card_id3.id_code is used to detemine if media is
smartmedia or XD.

Best regards,
Maxim Levitsky

Re: Plan for adding XD support in mtd layer

[Jörn Engel]

On Thu, 26 November 2009 01:22:45 +0200, Maxim Levitsky wrote:
> > 
> > According to FMTP140E.PDF, CIS and IDI are stored in sector 0 or the
> > first non-defective block.  In other words, it can be read and written
> > normally, without any magic.  Certainly not the atrocities I tried not
> > to remember.
> I wish I had the spec.

It used to be semi-free.  You could download it for your name and
address.  But I believe I'm not legally able to forward it to you and
the official site seems to be down.  Most of the information can also
be found here:
http://www.win.tue.nl/~aeb/linux/smartmedia/SmartMedia_Format.pdf
And there seems to be a copy available at pudn:
http://en.pudn.com/downloads137/doc/fileformat/detail585996_en.html

> I don't know the format of the CIS, and I know that special command is
> used to query the ID.
> CIS these days isn't used for anything and I hoped not to touch that
> thing.

:)

> I wont argue on that one with you, but why not to add the new field.
> It is quite generic, it could be card vendor information or so?
> Few bits of custom information to assign for each card no?
> 
> 
> I am taking about these:
> 
> struct xd_card_id1 {
> unsigned char maker_code;
> unsigned char device_code;
> unsigned char option_code1;
> unsigned char option_code2;
> } __attribute__((packed));
> 
> struct xd_card_id2 {
> unsigned char characteristics_code;
> #define XD_CARD_CELL_TYPE_MASK   0xc0
> #define XD_CARD_MULTI_BLOCK_MASK 0x30
> 
> unsigned char vendor_code1;
> unsigned char size_code;
> unsigned char vendor_code2;
> unsigned char vendor_code3;
> } __attribute__((packed));
> 
> struct xd_card_id3 {
> unsigned char vendor_code1;
> unsigned char vendor_code2;
> unsigned char id_code;
> unsigned char vendor_code3;
> } __attribute__((packed));
> 
> 
> Are these 32 bit values copied from CIF?
> 
> These values are read using commands:
> 
> XD_CARD_CMD_ID1         = 0x90,
> XD_CARD_CMD_ID2         = 0x91,
> XD_CARD_CMD_ID3         = 0x9a
> 
> 
> I need especially the xd_card_id1.device_code because this specifies the
> media type, but xd_card_id3.id_code is used to detemine if media is
> smartmedia or XD.

If you add this information to a struct xD_card and have mtd->priv point
to said structure, wouldn't that be enough for your purposes?

Jörn

-- 
The cheapest, fastest and most reliable components of a computer
system are those that aren't there.
-- Gordon Bell, DEC labratories

Re: Plan for adding XD support in mtd layer

[Maxim Levitsky]

On Thu, 2009-11-26 at 09:27 +0100, Jörn Engel wrote: 
> On Thu, 26 November 2009 01:22:45 +0200, Maxim Levitsky wrote:
> > > 
> > > According to FMTP140E.PDF, CIS and IDI are stored in sector 0 or the
> > > first non-defective block.  In other words, it can be read and written
> > > normally, without any magic.  Certainly not the atrocities I tried not
> > > to remember.
> > I wish I had the spec.
> 
> It used to be semi-free.  You could download it for your name and
> address.  But I believe I'm not legally able to forward it to you and
> the official site seems to be down.  Most of the information can also
> be found here:
> http://www.win.tue.nl/~aeb/linux/smartmedia/SmartMedia_Format.pdf
This one I have, and there is a brief description of the CIS format,
which seems to indicate that its contents are predefined.

> And there seems to be a copy available at pudn:
> http://en.pudn.com/downloads137/doc/fileformat/detail585996_en.html
> 
> > I don't know the format of the CIS, and I know that special command is
> > used to query the ID.
> > CIS these days isn't used for anything and I hoped not to touch that
> > thing.
> 
> :)
> 
> > I wont argue on that one with you, but why not to add the new field.
> > It is quite generic, it could be card vendor information or so?
> > Few bits of custom information to assign for each card no?
> > 
> > 
> > I am taking about these:
> > 
> > struct xd_card_id1 {
> > unsigned char maker_code;
> > unsigned char device_code;
> > unsigned char option_code1;
> > unsigned char option_code2;
> > } __attribute__((packed));
> > 
> > struct xd_card_id2 {
> > unsigned char characteristics_code;
> > #define XD_CARD_CELL_TYPE_MASK   0xc0
> > #define XD_CARD_MULTI_BLOCK_MASK 0x30
> > 
> > unsigned char vendor_code1;
> > unsigned char size_code;
> > unsigned char vendor_code2;
> > unsigned char vendor_code3;
> > } __attribute__((packed));
> > 
> > struct xd_card_id3 {
> > unsigned char vendor_code1;
> > unsigned char vendor_code2;
> > unsigned char id_code;
> > unsigned char vendor_code3;
> > } __attribute__((packed));
> > 
> > 
> > Are these 32 bit values copied from CIF?
> > 
> > These values are read using commands:
> > 
> > XD_CARD_CMD_ID1         = 0x90,
> > XD_CARD_CMD_ID2         = 0x91,
> > XD_CARD_CMD_ID3         = 0x9a
> > 
> > 
> > I need especially the xd_card_id1.device_code because this specifies the
> > media type, but xd_card_id3.id_code is used to detemine if media is
> > smartmedia or XD.
> 
> If you add this information to a struct xD_card and have mtd->priv point
> to said structure, wouldn't that be enough for your purposes?

I am afraid not.


My plan is to create one 'meta' mtd driver for all XD controllers, and
plug in small drivers.

The meta driver and small driver will share the ->priv structure.

I need a way to pass the ID from mtd driver to FTL, and to userspace via
mtdchar.


Best regards,
Maxim Levitsky

Re: Plan for adding XD support in mtd layer

[Jörn Engel]

On Thu, 26 November 2009 15:02:54 +0200, Maxim Levitsky wrote:
>
> This one I have, and there is a brief description of the CIS format,
> which seems to indicate that its contents are predefined.

That is my understanding, yes.  Essentially an xD card is just nand
flash in cheap packaging with the first block filled by CIS/IDI.

> I am afraid not.
> 
> 
> My plan is to create one 'meta' mtd driver for all XD controllers, and
> plug in small drivers.
> 
> The meta driver and small driver will share the ->priv structure.
> 
> I need a way to pass the ID from mtd driver to FTL, and to userspace via
> mtdchar.

You can also add a ->priv field to struct xD_data.  Or you can have it
contain the mtd_info, and use container_of to access the xD information.

But those are details.  Feel free to write the code either way.  We can
sort out which method to use later.

Jörn

-- 
Audacity augments courage; hesitation, fear.
-- Publilius Syrus

Re: Plan for adding XD support in mtd layer

[Maxim Levitsky]

On Thu, 2009-11-26 at 14:42 +0100, Jörn Engel wrote: 
> On Thu, 26 November 2009 15:02:54 +0200, Maxim Levitsky wrote:
> >
> > This one I have, and there is a brief description of the CIS format,
> > which seems to indicate that its contents are predefined.
> 
> That is my understanding, yes.  Essentially an xD card is just nand
> flash in cheap packaging with the first block filled by CIS/IDI.
> 
> > I am afraid not.
> > 
> > 
> > My plan is to create one 'meta' mtd driver for all XD controllers, and
> > plug in small drivers.
> > 
> > The meta driver and small driver will share the ->priv structure.
> > 
> > I need a way to pass the ID from mtd driver to FTL, and to userspace via
> > mtdchar.
> 
> You can also add a ->priv field to struct xD_data.  Or you can have it
> contain the mtd_info, and use container_of to access the xD information.
> 
> But those are details.  Feel free to write the code either way.  We can
> sort out which method to use later.

Thanks, no more questions, I will publish code as soon as I write one.

Best regards,
Maxim Levitsky


> 
> Jörn
> 

Re: XD/smartmedia - how to implement it right?

[Alex Dubov]

> 
> The "mtd uses blocking calls" argument I can help
> with.  For some other
> project I've added non-blocking calls to mtd[1].
> 
> Biggest problems I see are a) the userspace interface and
> b) allowing
> both raw flash access through MTD and block device access
> with an FTL.
> 
> [1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
> 
> Jörn

I actually tried several approaches to this problem and I think, I have a good idea about how to create a fully asynchronous mtd driver, which can handle different device models. Unfortunately, I had other commitments so I couldn't advance my mtd stuff.

Recently I was going to go back to it, but then I was asked to implement some additional functionality in rapidio subsystem, which I'll have to take care of first.



      

Re: XD/smartmedia - how to implement it right?

[Maxim Levitsky]

On Fri, 2009-11-27 at 23:22 -0800, Alex Dubov wrote: 
> > 
> > The "mtd uses blocking calls" argument I can help
> > with.  For some other
> > project I've added non-blocking calls to mtd[1].
> > 
> > Biggest problems I see are a) the userspace interface and
> > b) allowing
> > both raw flash access through MTD and block device access
> > with an FTL.
> > 
> > [1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
> > 
> > Jörn
> 
> I actually tried several approaches to this problem and I think, I have a good idea about how to create a fully asynchronous mtd driver, which can handle different device models. Unfortunately, I had other commitments so I couldn't advance my mtd stuff.
> 
> Recently I was going to go back to it, but then I was asked to implement some additional functionality in rapidio subsystem, which I'll have to take care of first.

Hi, 

First of all, thank you very much for your contributions.
Could you explain, why we need an asynchronous mtd driver?

Also, as I understand the command interface more and more, it seems that
'magically' xD card had same interface as standard NAND flash chip.

I think I can implement the driver for each controller just like an nand
chip driver.

Best regards,
Maxim Levitsky

Re: XD/smartmedia - how to implement it right?

[Alex Dubov]

--- On Sat, 11/28/09, Maxim Levitsky <maximlevitsky@gmail.com> wrote:

> From: Maxim Levitsky <maximlevitsky@gmail.com>
> Subject: Re: XD/smartmedia - how to implement it right?
> To: "Alex Dubov" <oakad@yahoo.com>
> Cc: "Jörn Engel" <joern@logfs.org>, "linux-kernel" <linux-kernel@vger.kernel.org>, arnd@arndb.de, tglx@linutonix.de
> Date: Saturday, November 28, 2009, 2:36 AM
> On Fri, 2009-11-27 at 23:22 -0800,
> Alex Dubov wrote: 
> > > 
> > > The "mtd uses blocking calls" argument I can
> help
> > > with.  For some other
> > > project I've added non-blocking calls to mtd[1].
> > > 
> > > Biggest problems I see are a) the userspace
> interface and
> > > b) allowing
> > > both raw flash access through MTD and block
> device access
> > > with an FTL.
> > > 
> > > [1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
> > > 
> > > Jörn
> > 
> > I actually tried several approaches to this problem
> and I think, I have a good idea about how to create a fully
> asynchronous mtd driver, which can handle different device
> models. Unfortunately, I had other commitments so I couldn't
> advance my mtd stuff.
> > 
> > Recently I was going to go back to it, but then I was
> asked to implement some additional functionality in rapidio
> subsystem, which I'll have to take care of first.
> 
> Hi, 
> 
> First of all, thank you very much for your contributions.
> Could you explain, why we need an asynchronous mtd driver?

Because it's the future. ;-)

> 
> Also, as I understand the command interface more and more,
> it seems that
> 'magically' xD card had same interface as standard NAND
> flash chip.
> 
> I think I can implement the driver for each controller just
> like an nand
> chip driver.
> 

xD spec, first of all, is about flash-based ordinary block devices.
All the hard problems are in good r/w FTL implementation, which is
currently mostly absent from the MTD.



      

Re: XD/smartmedia - how to implement it right?

[Maxim Levitsky]

On Mon, 2009-11-30 at 04:35 -0800, Alex Dubov wrote: 
> 
> --- On Sat, 11/28/09, Maxim Levitsky <maximlevitsky@gmail.com> wrote:
> 
> > From: Maxim Levitsky <maximlevitsky@gmail.com>
> > Subject: Re: XD/smartmedia - how to implement it right?
> > To: "Alex Dubov" <oakad@yahoo.com>
> > Cc: "Jörn Engel" <joern@logfs.org>, "linux-kernel" <linux-kernel@vger.kernel.org>, arnd@arndb.de, tglx@linutonix.de
> > Date: Saturday, November 28, 2009, 2:36 AM
> > On Fri, 2009-11-27 at 23:22 -0800,
> > Alex Dubov wrote: 
> > > > 
> > > > The "mtd uses blocking calls" argument I can
> > help
> > > > with.  For some other
> > > > project I've added non-blocking calls to mtd[1].
> > > > 
> > > > Biggest problems I see are a) the userspace
> > interface and
> > > > b) allowing
> > > > both raw flash access through MTD and block
> > device access
> > > > with an FTL.
> > > > 
> > > > [1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
> > > > 
> > > > Jörn
> > > 
> > > I actually tried several approaches to this problem
> > and I think, I have a good idea about how to create a fully
> > asynchronous mtd driver, which can handle different device
> > models. Unfortunately, I had other commitments so I couldn't
> > advance my mtd stuff.
> > > 
> > > Recently I was going to go back to it, but then I was
> > asked to implement some additional functionality in rapidio
> > subsystem, which I'll have to take care of first.
> > 
> > Hi, 
> > 
> > First of all, thank you very much for your contributions.
> > Could you explain, why we need an asynchronous mtd driver?
> 
> Because it's the future. ;-)
Although I like the xD format, it has no future,
Everyone migrated to SD now.

Reasons are very simple:

1 - xD hardcodes device sizes, thus if somebody makes say 8GB card, many
readers couldn't use it.
I know that windows driver I was dealing with supports sizes up to 2GB


2 - xD is a nand card, but with 8-bit bus.
Newer nand chips seem to favor 16 bit bus or more.
Obviously, if they were to implement that it would be physically
incompatible.

3 - embedded cpus are cheap dirt now, and allow to use their, always
'better' FTL strategy, and also hide inner working of the chip.

Memstick pro is also today sort of SD card, but for Sony its their first
priority to keep proprietary standards even if inferior.
I won't be surprised though if this format dies too.

Could you explain why we need an asynchronous interface?
I understand that filesystem/page cache/block cache/and block layer,
already provide caches and asynchronous behavior.

Only maybe reading several pages at time, and writing could benefit, but
many devices don't support that feature.
And again, bulk of the time will be spend writing, and only saving will
be from queuing more work to the device.



> 
> > 
> > Also, as I understand the command interface more and more,
> > it seems that
> > 'magically' xD card had same interface as standard NAND
> > flash chip.
> > 
> > I think I can implement the driver for each controller just
> > like an nand
> > chip driver.
> > 
> 
> xD spec, first of all, is about flash-based ordinary block devices.
> All the hard problems are in good r/w FTL implementation, which is
> currently mostly absent from the MTD.

Its true, and I will write an r/w FTL for xD devices, in same manner as
existing FTL implementations.
It works above the mtd layer btw.



Best regards,
Maxim Levitsky

Re: XD/smartmedia - how to implement it right?

[Maxim Levitsky]

On Mon, 2009-11-30 at 15:58 +0200, Maxim Levitsky wrote: 
> On Mon, 2009-11-30 at 04:35 -0800, Alex Dubov wrote: 
> > 
> > --- On Sat, 11/28/09, Maxim Levitsky <maximlevitsky@gmail.com> wrote:
> > 
> > > From: Maxim Levitsky <maximlevitsky@gmail.com>
> > > Subject: Re: XD/smartmedia - how to implement it right?
> > > To: "Alex Dubov" <oakad@yahoo.com>
> > > Cc: "Jörn Engel" <joern@logfs.org>, "linux-kernel" <linux-kernel@vger.kernel.org>, arnd@arndb.de, tglx@linutonix.de
> > > Date: Saturday, November 28, 2009, 2:36 AM
> > > On Fri, 2009-11-27 at 23:22 -0800,
> > > Alex Dubov wrote: 
> > > > > 
> > > > > The "mtd uses blocking calls" argument I can
> > > help
> > > > > with.  For some other
> > > > > project I've added non-blocking calls to mtd[1].
> > > > > 
> > > > > Biggest problems I see are a) the userspace
> > > interface and
> > > > > b) allowing
> > > > > both raw flash access through MTD and block
> > > device access
> > > > > with an FTL.
> > > > > 
> > > > > [1] http://lists.infradead.org/pipermail/linux-mtd/2009-November/028065.html
> > > > > 
> > > > > Jörn
> > > > 
> > > > I actually tried several approaches to this problem
> > > and I think, I have a good idea about how to create a fully
> > > asynchronous mtd driver, which can handle different device
> > > models. Unfortunately, I had other commitments so I couldn't
> > > advance my mtd stuff.
> > > > 
> > > > Recently I was going to go back to it, but then I was
> > > asked to implement some additional functionality in rapidio
> > > subsystem, which I'll have to take care of first.
> > > 
> > > Hi, 
> > > 
> > > First of all, thank you very much for your contributions.
> > > Could you explain, why we need an asynchronous mtd driver?
> > 
> > Because it's the future. ;-)
> Although I like the xD format, it has no future,
> Everyone migrated to SD now.
> 
> Reasons are very simple:
> 
> 1 - xD hardcodes device sizes, thus if somebody makes say 8GB card, many
> readers couldn't use it.
> I know that windows driver I was dealing with supports sizes up to 2GB
> 
> 
> 2 - xD is a nand card, but with 8-bit bus.
> Newer nand chips seem to favor 16 bit bus or more.
> Obviously, if they were to implement that it would be physically
> incompatible.
> 
> 3 - embedded cpus are cheap dirt now, and allow to use their, always
> 'better' FTL strategy, and also hide inner working of the chip.
> 
> Memstick pro is also today sort of SD card, but for Sony its their first
> priority to keep proprietary standards even if inferior.
> I won't be surprised though if this format dies too.
> 
> Could you explain why we need an asynchronous interface?
> I understand that filesystem/page cache/block cache/and block layer,
> already provide caches and asynchronous behavior.
> 
> Only maybe reading several pages at time, and writing could benefit, but
> many devices don't support that feature.
> And again, bulk of the time will be spend writing, and only saving will
> be from queuing more work to the device.
> 
> 
> 
> > 
> > > 
> > > Also, as I understand the command interface more and more,
> > > it seems that
> > > 'magically' xD card had same interface as standard NAND
> > > flash chip.
> > > 
> > > I think I can implement the driver for each controller just
> > > like an nand
> > > chip driver.
> > > 
> > 
> > xD spec, first of all, is about flash-based ordinary block devices.
> > All the hard problems are in good r/w FTL implementation, which is
> > currently mostly absent from the MTD.
> 
> Its true, and I will write an r/w FTL for xD devices, in same manner as
> existing FTL implementations.
> It works above the mtd layer btw.


Alex, could you do me a favor, and explain why partial block writes
don't work?
What happens if its done?

the FTL layer of course will do full block writes, but its not easy to
do full block writes in mtd layer.

If I was going to do full block writes, I would need to set
page_size = block size, and then loose many features nand mtd supports.


Best regards,
Maxim Levitsky

Re: XD/smartmedia - how to implement it right?

[Jörn Engel]

On Tue, 1 December 2009 01:04:26 +0200, Maxim Levitsky wrote:
> 
> Alex, could you do me a favor, and explain why partial block writes
> don't work?
> What happens if its done?

With the Alauda and the limited set of cards I tried they work.

Jörn

-- 
The story so far:
In the beginning the Universe was created.  This has made a lot
of people very angry and been widely regarded as a bad move.
-- Douglas Adams

Re: XD/smartmedia - how to implement it right?

[Alex Dubov]

--- On Tue, 12/1/09, Jörn Engel <joern@logfs.org> wrote:

> From: Jörn Engel <joern@logfs.org>
> Subject: Re: XD/smartmedia - how to implement it right?
> To: "Maxim Levitsky" <maximlevitsky@gmail.com>
> Cc: "Alex Dubov" <oakad@yahoo.com>, "linux-kernel" <linux-kernel@vger.kernel.org>, arnd@arndb.de, tglx@linutonix.de
> Date: Tuesday, December 1, 2009, 12:22 AM
> On Tue, 1 December 2009 01:04:26
> +0200, Maxim Levitsky wrote:
> > 
> > Alex, could you do me a favor, and explain why partial
> block writes
> > don't work?
> > What happens if its done?
> 
> With the Alauda and the limited set of cards I tried they
> work.
> 
> Jörn
> 

It's a bug particular to jmicron controllers.



      

Re: XD/smartmedia - how to implement it right?

[Maxim Levitsky]

On Tue, 2009-12-01 at 08:10 -0800, Alex Dubov wrote: 
> 
> --- On Tue, 12/1/09, Jörn Engel <joern@logfs.org> wrote:
> 
> > From: Jörn Engel <joern@logfs.org>
> > Subject: Re: XD/smartmedia - how to implement it right?
> > To: "Maxim Levitsky" <maximlevitsky@gmail.com>
> > Cc: "Alex Dubov" <oakad@yahoo.com>, "linux-kernel" <linux-kernel@vger.kernel.org>, arnd@arndb.de, tglx@linutonix.de
> > Date: Tuesday, December 1, 2009, 12:22 AM
> > On Tue, 1 December 2009 01:04:26
> > +0200, Maxim Levitsky wrote:
> > > 
> > > Alex, could you do me a favor, and explain why partial
> > block writes
> > > don't work?
> > > What happens if its done?
> > 
> > With the Alauda and the limited set of cards I tried they
> > work.
> > 
> > Jörn
> > 
> 
> It's a bug particular to jmicron controllers.

Thanks!

So, I will write the driver for my controller using nand subsystem, and
then, I see what to do next.
If you write your mtd driver, I have no objection to port my driver to
yours (I assume that there will be a common core, just like the nand
subsystem, and small drivers.)


And the last question probably:

I know that xD is very similar to nand, but due to things like the
above, maybe full mtd driver is really better.
There will be some code duplication, but on the other hand the driver
could follow the way windows driver works more closely, thus making it
more compatable with xD cards.

So if I or Alex write the full blown mtd driver, there will be no
objections because of code duplication?


Anyway, now I am going to stop talking and start coding (I did quite in
depth review of mtd and nand systems already)


Best regards,
Maxim Levitsky

Re: XD/smartmedia - how to implement it right?

[Pavel Machek]

> > > First of all, thank you very much for your contributions.
> > > Could you explain, why we need an asynchronous mtd driver?
> > 
> > Because it's the future. ;-)
> Although I like the xD format, it has no future,
> Everyone migrated to SD now.
> 
> Reasons are very simple:
> 
> 1 - xD hardcodes device sizes, thus if somebody makes say 8GB card, many
> readers couldn't use it.
> I know that windows driver I was dealing with supports sizes up to 2GB
> 
> 
> 2 - xD is a nand card, but with 8-bit bus.
> Newer nand chips seem to favor 16 bit bus or more.
> Obviously, if they were to implement that it would be physically
> incompatible.
> 
> 3 - embedded cpus are cheap dirt now, and allow to use their, always
> 'better' FTL strategy, and also hide inner working of the chip.

XDs are still used by at least olympus digital cameras. Ouch.

If you have onboard embedded cpu, you can bet it will not work
correctly over powerfail. If you do FTL on host, you have at least
chance to get it right.

								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: XD/smartmedia - how to implement it right?

[Maxim Levitsky]

On Sat, 2009-12-05 at 20:09 +0100, Pavel Machek wrote: 
> > > > First of all, thank you very much for your contributions.
> > > > Could you explain, why we need an asynchronous mtd driver?
> > > 
> > > Because it's the future. ;-)
> > Although I like the xD format, it has no future,
> > Everyone migrated to SD now.
> > 
> > Reasons are very simple:
> > 
> > 1 - xD hardcodes device sizes, thus if somebody makes say 8GB card, many
> > readers couldn't use it.
> > I know that windows driver I was dealing with supports sizes up to 2GB
> > 
> > 
> > 2 - xD is a nand card, but with 8-bit bus.
> > Newer nand chips seem to favor 16 bit bus or more.
> > Obviously, if they were to implement that it would be physically
> > incompatible.
> > 
> > 3 - embedded cpus are cheap dirt now, and allow to use their, always
> > 'better' FTL strategy, and also hide inner working of the chip.
> 
> XDs are still used by at least olympus digital cameras. Ouch.
Well, at least according to wikipedia, Olympus phased out XD support, as
well as Fuji.
This doesn't mean that there will be no users of XD in five following
years, thats why I am writing the driver in first place.


> 
> If you have onboard embedded cpu, you can bet it will not work
> correctly over powerfail. If you do FTL on host, you have at least
> chance to get it right.
Thats correct. In fact we are both right here.
Your viewpoint is based on a view of an user of a sane operation system.
Sure, the whole idea of FTL is evil, the perfect place for wear leveling
is in the filesystem itself, and in addition to that, there are several
technologies that imply finite write cycle count, and that reason
justifies such filesystem even more.

I also use the same sane operation system, so for me its quite nice to
have XD^WNAND removable chip support, and in my eyes the raw access to
flash is very nice advantage.

However, due to limitations of other less sane operation systems,
hardware makers have to support wear leveling ether in the driver or in
the hardware.
This is yet another example, about why some operation systems 'just
work'...
Not that these operation systems are better, but that hardware makers do
everything possible to patch all problems these operation systems have.
Same is true for software generally.


Supporting wear leveling in hardware is easier, because this way, they
can be sure it works well, and not depend on each and every driver
implementation to do that right.

Small hint, you know how the windows driver for my controller handles
COW?
Very simple, every 512 bytes write (only accepts such) turns into whole
erase/write cycle of whole block (32 KB usually).

So, if you write a 32K or larger file , and no matter how clever the OS
is, it will still end in 64 erase cycles...


Best regards,
Maxim Levitsky

Re: XD/smartmedia - how to implement it right?

[Arnd Bergmann]

On Sunday 06 December 2009, Maxim Levitsky wrote:
> Supporting wear leveling in hardware is easier, because this way, they
> can be sure it works well, and not depend on each and every driver
> implementation to do that right.

Assuming that hardware engineers know what they are doing, yes.
Unfortunately the hardware people seem to be just as clueless about
this as the every device driver writer, which actually makes this much
worse. Having bad wear leveling in hardware often means that the
card eats your data, no matter what the driver does. Doing it in software
means that it's broken most of the time but fixable.

	Arnd <><

Re: XD/smartmedia - how to implement it right?

[Maxim Levitsky]

On Tue, 2009-12-01 at 09:22 +0100, Jörn Engel wrote: 
> On Tue, 1 December 2009 01:04:26 +0200, Maxim Levitsky wrote:
> > 
> > Alex, could you do me a favor, and explain why partial block writes
> > don't work?
> > What happens if its done?
> 
> With the Alauda and the limited set of cards I tried they work.
> 
> Jörn
> 
I have few severe problems in writing the driver, but otherwise its is
almost written. I also suspect that I will have to modify mtd core in
several places to achieve what I need to. I guess Alex was right after
all...

First of all, the device I deal with supports 32 bit dma only, as well
as many other pci devices. (I know that for sure because dma register is
32 bit wide....)

Now, mtd_blkdevs doesn't ask block subsystem for 32 bit bounce, and in
addition to that there is no guarantee that my mtd driver won't be
called on unrelated kernel buffer (for example if ftl layer does
caching....)
So I guess, ether I need to try to map the buffer, and then fail back to
copying into preallocated buffer in 32 bit zone, or always use that
buffer.
MMC core seem to handle the dma bouncing inside.


But that minor problem, other problem, I revealed when I began to think
about how properly to guard against races.
Problem is that mtd core doesn't handle hot-plug well.

Suppose, user has mounted filesystem on xd card and pulls it out.
SD/MMC cards handle that case. There might be fs corruption though, but
that is the user responsibility.

Now if anything has open the mtd_blkdevs, it increments the mtd user
count, thus removal of the mtd device will just fail.
Now I really don't know how to handle that property, but I know that
yanking out usb keys, mmc cards, etc works, and all devices that were
created disappear.

I will eventually figure out how to do that right, but it would be great
if you help me a bit.

Best regards,
Maxim Levitsly