linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Moving unmaintained filesystems to staging
@ 2018-04-25 15:46 Matthew Wilcox
  2018-04-25 15:47 ` Christoph Hellwig
                   ` (2 more replies)
  0 siblings, 3 replies; 73+ messages in thread
From: Matthew Wilcox @ 2018-04-25 15:46 UTC (permalink / raw)
  To: linux-fsdevel; +Cc: linux-kernel

Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
developers report bugs in hfs, which they deem a security hole because
Ubuntu attempts to automount an inserted USB device as hfs.

We have no maintainer for hfs, and no likely prospect of anyone stepping
up soon to become hfs maintainer.  I think it's irresponsible of us
to present unmaintained code on an equal basis with filesystems under
active maintenance like ext2.

I therefore propose we move the following filesystems which are explicitly
listed as Orphaned to staging:

affs - Amiga filesystem.
efs - old SGI filesystem predating XFS, used on CDs for a while.
hfs - Mac filesystem.
hfsplus - Mac filesystem.

I further propose we move the following filesystems which have no entry
in MAINTAINERS to staging:

adfs - Acorn filesystem from the 1990s.
minix
qnx6

We have no listed maintainer for isofs.  This is clearly not a candidate
for staging.  Jan Kara appears to be acting as default maintainer,
so I'd like to propose he gets the appropriate credit with an entry
in MAINTAINERS.

romfs also has no listed maintainer.  I'm not sure who to suggest,
but it seems to have users.

tracefs isn't listed, and probably needs to become part of the
TRACING entry.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-25 15:46 Moving unmaintained filesystems to staging Matthew Wilcox
@ 2018-04-25 15:47 ` Christoph Hellwig
  2018-04-25 20:30 ` David Sterba
  2018-04-26  6:11 ` Pavel Machek
  2 siblings, 0 replies; 73+ messages in thread
From: Christoph Hellwig @ 2018-04-25 15:47 UTC (permalink / raw)
  To: Matthew Wilcox; +Cc: linux-fsdevel, linux-kernel

On Wed, Apr 25, 2018 at 08:46:02AM -0700, Matthew Wilcox wrote:
> hfsplus - Mac filesystem.

I don't think this is unmaintained, and it is pretty heavily used.

> minix

Still plenty of use.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-25 15:46 Moving unmaintained filesystems to staging Matthew Wilcox
  2018-04-25 15:47 ` Christoph Hellwig
@ 2018-04-25 20:30 ` David Sterba
  2018-04-26  2:57   ` Matthew Wilcox
  2018-04-26  4:58   ` Moving unmaintained filesystems to staging Nikolay Borisov
  2018-04-26  6:11 ` Pavel Machek
  2 siblings, 2 replies; 73+ messages in thread
From: David Sterba @ 2018-04-25 20:30 UTC (permalink / raw)
  To: Matthew Wilcox; +Cc: linux-fsdevel, linux-kernel

On Wed, Apr 25, 2018 at 08:46:02AM -0700, Matthew Wilcox wrote:
> Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
> developers report bugs in hfs, which they deem a security hole because
> Ubuntu attempts to automount an inserted USB device as hfs.
> 
> We have no maintainer for hfs, and no likely prospect of anyone stepping
> up soon to become hfs maintainer.  I think it's irresponsible of us
> to present unmaintained code on an equal basis with filesystems under
> active maintenance like ext2.
> 
> I therefore propose we move the following filesystems which are explicitly
> listed as Orphaned to staging:
> 
> affs - Amiga filesystem.
> efs - old SGI filesystem predating XFS, used on CDs for a while.
> hfs - Mac filesystem.
> hfsplus - Mac filesystem.
> 
> I further propose we move the following filesystems which have no entry
> in MAINTAINERS to staging:
> 
> adfs - Acorn filesystem from the 1990s.
> minix
> qnx6

I had similar toughts some time ago while browsing the fs/ directory.
Access to the filesystem images can be reimplemented in FUSE, but other
than that, I don't think the in-kernel code would be missed.

It's hard to know how many users are there. I was curious eg. about bfs,
befs, coda or feevxfs, looked at the last commits and searched around
web if there are any mentions or user community. But as long as there's
somebody listed in MAINTAINERS, the above are not candidates for moving
to staging or deletion.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-25 20:30 ` David Sterba
@ 2018-04-26  2:57   ` Matthew Wilcox
  2018-04-26 10:28     ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Martin Steigerwald
  2018-04-26  4:58   ` Moving unmaintained filesystems to staging Nikolay Borisov
  1 sibling, 1 reply; 73+ messages in thread
From: Matthew Wilcox @ 2018-04-26  2:57 UTC (permalink / raw)
  To: dsterba, linux-fsdevel, linux-kernel

On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
> I had similar toughts some time ago while browsing the fs/ directory.
> Access to the filesystem images can be reimplemented in FUSE, but other
> than that, I don't think the in-kernel code would be missed.
> 
> It's hard to know how many users are there. I was curious eg. about bfs,
> befs, coda or feevxfs, looked at the last commits and searched around
> web if there are any mentions or user community. But as long as there's
> somebody listed in MAINTAINERS, the above are not candidates for moving
> to staging or deletion.

Yeah, it's pretty sad how few commits some of these filesystems have
had in recent years.  One can argue that they're stable and don't need
to be fixed because they aren't broken, but I find it hard to believe
that any of them were better-implemented than ext2 which still sees
regular bugfixes.

As long as there's someone who can be pestered with bugs, I don't see
the need to push their baby out of the tree.  I'm a bit more nervous
about hfsplus; if it has users and no maintainer, those users are at risk.

Perhaps we could advertise on Craigslist or something ... maintainer
wanted for LTR.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-25 20:30 ` David Sterba
  2018-04-26  2:57   ` Matthew Wilcox
@ 2018-04-26  4:58   ` Nikolay Borisov
  2018-04-26  5:30     ` Willy Tarreau
  1 sibling, 1 reply; 73+ messages in thread
From: Nikolay Borisov @ 2018-04-26  4:58 UTC (permalink / raw)
  To: dsterba, Matthew Wilcox, linux-fsdevel, linux-kernel



On 25.04.2018 23:30, David Sterba wrote:
> On Wed, Apr 25, 2018 at 08:46:02AM -0700, Matthew Wilcox wrote:
>> Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
>> developers report bugs in hfs, which they deem a security hole because
>> Ubuntu attempts to automount an inserted USB device as hfs.
>>
>> We have no maintainer for hfs, and no likely prospect of anyone stepping
>> up soon to become hfs maintainer.  I think it's irresponsible of us
>> to present unmaintained code on an equal basis with filesystems under
>> active maintenance like ext2.
>>
>> I therefore propose we move the following filesystems which are explicitly
>> listed as Orphaned to staging:
>>
>> affs - Amiga filesystem.
>> efs - old SGI filesystem predating XFS, used on CDs for a while.
>> hfs - Mac filesystem.
>> hfsplus - Mac filesystem.
>>
>> I further propose we move the following filesystems which have no entry
>> in MAINTAINERS to staging:
>>
>> adfs - Acorn filesystem from the 1990s.
>> minix
>> qnx6
> 
> I had similar toughts some time ago while browsing the fs/ directory.
> Access to the filesystem images can be reimplemented in FUSE, but other
> than that, I don't think the in-kernel code would be missed.
> 
> It's hard to know how many users are there. I was curious eg. about bfs,
> befs, coda or feevxfs, looked at the last commits and searched around
> web if there are any mentions or user community. But as long as there's
> somebody listed in MAINTAINERS, the above are not candidates for moving
> to staging or deletion.
> 

I think the presence of maintainers entry is necessary but insufficient.
What if the maintainer has long lost interest but just didn't bother
updating the file. In the very least maintainers should be pinged and
asked if they are still maintaining the respective piece of code.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-26  4:58   ` Moving unmaintained filesystems to staging Nikolay Borisov
@ 2018-04-26  5:30     ` Willy Tarreau
  0 siblings, 0 replies; 73+ messages in thread
From: Willy Tarreau @ 2018-04-26  5:30 UTC (permalink / raw)
  To: Nikolay Borisov; +Cc: dsterba, Matthew Wilcox, linux-fsdevel, linux-kernel

On Thu, Apr 26, 2018 at 07:58:52AM +0300, Nikolay Borisov wrote:
> 
> 
> On 25.04.2018 23:30, David Sterba wrote:
> > On Wed, Apr 25, 2018 at 08:46:02AM -0700, Matthew Wilcox wrote:
> >> Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
> >> developers report bugs in hfs, which they deem a security hole because
> >> Ubuntu attempts to automount an inserted USB device as hfs.
> >>
> >> We have no maintainer for hfs, and no likely prospect of anyone stepping
> >> up soon to become hfs maintainer.  I think it's irresponsible of us
> >> to present unmaintained code on an equal basis with filesystems under
> >> active maintenance like ext2.
> >>
> >> I therefore propose we move the following filesystems which are explicitly
> >> listed as Orphaned to staging:
> >>
> >> affs - Amiga filesystem.
> >> efs - old SGI filesystem predating XFS, used on CDs for a while.
> >> hfs - Mac filesystem.
> >> hfsplus - Mac filesystem.
> >>
> >> I further propose we move the following filesystems which have no entry
> >> in MAINTAINERS to staging:
> >>
> >> adfs - Acorn filesystem from the 1990s.
> >> minix
> >> qnx6
> > 
> > I had similar toughts some time ago while browsing the fs/ directory.
> > Access to the filesystem images can be reimplemented in FUSE, but other
> > than that, I don't think the in-kernel code would be missed.
> > 
> > It's hard to know how many users are there. I was curious eg. about bfs,
> > befs, coda or feevxfs, looked at the last commits and searched around
> > web if there are any mentions or user community. But as long as there's
> > somebody listed in MAINTAINERS, the above are not candidates for moving
> > to staging or deletion.
> > 
> 
> I think the presence of maintainers entry is necessary but insufficient.
> What if the maintainer has long lost interest but just didn't bother
> updating the file. In the very least maintainers should be pinged and
> asked if they are still maintaining the respective piece of code.

That's a good point. However the age of the last commit must not be used
as an excuse for moving them, because if the few users are very happy with
the code, never meet corner cases and never have to report bugs, neither
them nor the maintainers should be punished. In my opinion the only two
reasons for deprecating or removing code are that it's not maintained
anymore or that it's using ancient infrastructure that needs to be
changed and there's no way to adapt the old code to the new one.

In fact I think that the problem with very silent maintainers goes way
beyond FS. Everyone in MAINTAINERS who didn't send a commit in one year
should probably be asked if they're still doing the work, if they'd be
interested in someone else taking over, or if they think the whole code
has no reason to continue to exist.

Willy

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-25 15:46 Moving unmaintained filesystems to staging Matthew Wilcox
  2018-04-25 15:47 ` Christoph Hellwig
  2018-04-25 20:30 ` David Sterba
@ 2018-04-26  6:11 ` Pavel Machek
  2018-04-26 10:36   ` Martin Steigerwald
                     ` (2 more replies)
  2 siblings, 3 replies; 73+ messages in thread
From: Pavel Machek @ 2018-04-26  6:11 UTC (permalink / raw)
  To: Matthew Wilcox; +Cc: linux-fsdevel, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
> developers report bugs in hfs, which they deem a security hole because
> Ubuntu attempts to automount an inserted USB device as hfs.

We promise "no-regressions" for code in main repository, no such
promise for staging. We have quite a lot of code without maintainer.

Moving code to staging means it will get broken -- staging was not
designed for this. I believe moving anything there is bad idea.

Staging is for ugly code, not for code that needs new maintainter.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-04-26  2:57   ` Matthew Wilcox
@ 2018-04-26 10:28     ` Martin Steigerwald
  2018-04-26 10:45       ` moving affs + RDB partition support to staging? John Paul Adrian Glaubitz
                         ` (2 more replies)
  0 siblings, 3 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-04-26 10:28 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: dsterba, linux-fsdevel, linux-kernel, Jens Axboe, linux-m68k

Hi Matthew.

You probably put your stick into a cave with ancient sleeping dragons :)

Added in linux-m68k mailing list, as they likely have an opinion on how 
to treat affs + RDB partition support. Also added in Jens Axboe about 
patching that RDB support broken with 2 TB or larger harddisks issue 
which had been in Linux kernel for 6 years while a patch exists that to 
my testing back then solves the issue.

Matthew Wilcox - 26.04.18, 04:57:
> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
> > I had similar toughts some time ago while browsing the fs/
> > directory.
> > Access to the filesystem images can be reimplemented in FUSE, but
> > other than that, I don't think the in-kernel code would be missed.
> > 
> > It's hard to know how many users are there. I was curious eg. about
> > bfs, befs, coda or feevxfs, looked at the last commits and searched
> > around web if there are any mentions or user community. But as long
> > as there's somebody listed in MAINTAINERS, the above are not
> > candidates for moving to staging or deletion.
> 
> Yeah, it's pretty sad how few commits some of these filesystems have
> had in recent years.  One can argue that they're stable and don't need
> to be fixed because they aren't broken, but I find it hard to believe
> that any of them were better-implemented than ext2 which still sees
> regular bugfixes.

Regarding affs there is a severe issue which is not in affs itself but 
in the handling of Rigid Disk Block (RDB) partitions, the Amiga 
partitioning standard, which is far more advanced than MBR: It overruns 
for 2 TB or larger drives and then wraps over to the beginning of the 
drive – I bet you can imagine what happens if you write to an area 
larger than 2 TB. I learned this with an external 2TB RDB partitioned 
harddisk back then, which I used for Sam440ep (a kind of successor for 
old, classic Amiga hardware) backup + some Linux related stuff in 
another partition.

Joanne Dow, a developer who developed hdwrench.library which HDToolBox 
uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then, 
but never officially put it officially through upstreaming as I offered 
to make a good description and upstream it through Jens Axboe.

I may take this as a reason to… actually follow through this time, 
hopefully remembering all the details in order to provide a meaningful 
patch description – but I think mostly I can do just careful copy and 
paste. Even tough I believe Joanne Dow´s fix only fixed my bug report 
43511, but not 43511 which is more about a safeguarding issue in case of 
future overflows, I still think it would be good to go in in case affs + 
RDB stays in their current places.

However, in case you move affs to staging, I may be less motivated to do 
so, but then I suggest you also move RDB partitioning support to 
staging, cause this is the one that is known to be dangerously badly for 
2 TB or larger disks. And yeah, I admit I did not follow through with 
having that patch upstreamed. Probably I did not want to be responsible 
in case my description would not have been absolutely accurate or the 
patch breaks something else. I do not have that 2 TB drive anymore and 
don´t feel like setting one up in a suitable way in order to go about 
this patch, but my testing back then was quite elaborate and I still 
feel pretty confident about it.

I totally get your motivation, but I would find it somewhat sad to see 
the filesystems you mentioned go into staging. However, as I just shown 
clearly, for the user it may be better, cause there may be unfixed 
dangerous bugs. FUSE might be an interesting approach, but I bet it will 
not solve the maintenance issue. If there is no one maintaining it in 
the kernel, I think its unlikely to find someone adapting it to be a 
FUSE filesystem and maintaining it. And then I am not aware of FUSE 
based partitioning support. (And I think think ideally we´d had a 
microkernel and run all filesystems in userspace processes with a 
defined set of privileges, but that is simply not Linux as it is.)

Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in 
AmigaOS 4.1
https://lkml.org/lkml/2012/6/17/6

Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size 
instead of refusing to use it
https://bugzilla.kernel.org/show_bug.cgi?id=43521

Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big, 
while OK in AmigaOS 4.1
https://bugzilla.kernel.org/show_bug.cgi?id=43511

I forward the relevant mail of Joanne, in

https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7

I even have the patch in diff format. And I just checked, the issue is 
still unpatched as of 4.16.3.

----------  Weitergeleitete Nachricht  ----------

Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big, 
while OK in AmigaOS 4.1
Datum: Montag, 18. Juni 2012, 03:28:48 CEST
Von: jdow <jdow@earthlink.net>
An: Martin Steigerwald <Martin@lichtvoll.de>
Kopie: Geert Uytterhoeven <geert@linux-m68k.org>, linux-
kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>, linux-
m68k@vger.kernel.org

On 2012/06/17 14:20, Martin Steigerwald wrote:
> Am Sonntag, 17. Juni 2012 schrieb jdow:
>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
> <Martin@lichtvoll.de> wrote:
>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> | JXFS 64 bit file system
>>>> |
>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>> | system, which means that it reduces data loss if data writes to
>>>> | the disk are interrupted. It is the fastest and most reliable
>>>> | file system ever created for AmigaOS.
>>>>
>>>> http://www.amigaos.net/content/1/features
>>>>
>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets 
see
>>>> what they say about 2 TB limits.
>>>
>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>> 4096?
>>>
>>> block/partitions/amiga.c reads the block size from
>>> RigidDiskBlock.rdb_BlockBytes,
>>> but after conversion to 512-byte blocks, all further calculations 
are
>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>
>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>> bytes per block,
>>> so I'll have to get a deeper look into your RDB first...
> […]
>> Note that the work I did on the Linux RDB code eons ago took full
>> cognizance of the physical and virtual block sizes.
> […]
>> I've asked Martin for a digital copy of his RDBs and what he thinks 
the
>> partition(s) should look like. I should also be told whether the disk
>> is supposed to be solely Amiga OSs or not. I gather it's not.
>
> Its all in the bug report. profile-binary.img should be it.
>
> Its small so I attach it.
>
> Meanwhile I try to get the currently supported maximum disk size out 
from
> the OS 4 developers. Maybe JXFS is playing tricks that other 
filesystems do
> not play or simply has a different implementation.
>
> Thanks,

I've sent Jens a proposed fix changing these lines in amiga.c.
===8<--- was
         struct PartitionBlock *pb;
         int start_sect, nr_sects, blk, part, res = 0;
         int blksize = 1;        /* Multiplier for disk block size */
===8<--- becomes changing line 338 into lines 338-339
         struct PartitionBlock *pb;
         sector_t start_sect, nr_sects;
         int blk, part, res = 0;
         int blksize = 1;        /* Multiplier for disk block size */
===8<---

(I'm working without proper tools at the moment or I'd make a real diff.
Sorry.)

This fix may not be complete. The RDBs contain provisions for describing
the physical disk block size and how many physical blocks are used in a
file system's logical blocks. This MAY not work quite right large 
physical
block sizes.

{^_^}   Joanne Dow
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-------------------------------------------------------------
[…]

> As long as there's someone who can be pestered with bugs, I don't see
> the need to push their baby out of the tree.  I'm a bit more nervous
> about hfsplus; if it has users and no maintainer, those users are at
> risk.
> 
> Perhaps we could advertise on Craigslist or something ... maintainer
> wanted for LTR.

Thanks,
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-26  6:11 ` Pavel Machek
@ 2018-04-26 10:36   ` Martin Steigerwald
  2018-05-03  9:18     ` Pavel Machek
  2018-04-27  1:10   ` Luis R. Rodriguez
  2018-04-29 12:07   ` Greg KH
  2 siblings, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-04-26 10:36 UTC (permalink / raw)
  To: Pavel Machek; +Cc: Matthew Wilcox, linux-fsdevel, linux-kernel

Pavel Machek - 26.04.18, 08:11:
> On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > Recently ncpfs got moved to staging.  Also recently, we had some
> > fuzzer developers report bugs in hfs, which they deem a security
> > hole because Ubuntu attempts to automount an inserted USB device as
> > hfs.
> 
> We promise "no-regressions" for code in main repository, no such
> promise for staging. We have quite a lot of code without maintainer.
> 
> Moving code to staging means it will get broken -- staging was not
> designed for this. I believe moving anything there is bad idea.
> 
> Staging is for ugly code, not for code that needs new maintainter.

Good point.

Moving things in and out to some "unmaintained" directory… I am not sure 
about that either. I tend to think that moving code around does not 
solve the underlying issue.

Which, according to what I got from Matthew, was that distributors 
enable just about every filesystem they can enable which lead to hfs 
being used for automounting an USB stick formatted with it.

In the end what may be beneficial would be hinting distributors and 
people who compile their own kernels at what features are considered 
stable, secure and well tested and what features are not, but how to 
determine that? The hint could be some kernel option flag that would be 
displayed by make *config. But then it probably needs also a 
justification or at least a link to more information.

Actually did ever someone audit the whole kernel source?

Thanks,
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-04-26 10:28     ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Martin Steigerwald
@ 2018-04-26 10:45       ` John Paul Adrian Glaubitz
  2018-04-26 10:59         ` David Sterba
  2018-05-06  0:59         ` Al Viro
  2018-04-26 11:00       ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Christoph Hellwig
  2018-04-26 11:08       ` Geert Uytterhoeven
  2 siblings, 2 replies; 73+ messages in thread
From: John Paul Adrian Glaubitz @ 2018-04-26 10:45 UTC (permalink / raw)
  To: Martin Steigerwald, Matthew Wilcox
  Cc: dsterba, linux-fsdevel, linux-kernel, Jens Axboe, linux-m68k,
	Debian m68k

(adding debian-68k)

Hi Matthew!

On 04/26/2018 12:28 PM, Martin Steigerwald wrote:
> You probably put your stick into a cave with ancient sleeping dragons :)

Indeed.

> Added in linux-m68k mailing list, as they likely have an opinion on how
> to treat affs + RDB partition support. Also added in Jens Axboe about
> patching that RDB support broken with 2 TB or larger harddisks issue
> which had been in Linux kernel for 6 years while a patch exists that to
> my testing back then solves the issue.

The answer is that we are still very much actively using RDB and AFFS
supoort in the Linux kernel and if you were to remove it, you would
directly hit users.

I know it may sound crazy, but the Linux/m68k port (Atari, Mac, Amiga etc)
is a very actively used and maintained port which just recently received
three new drivers:

> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.18/scsi-queue&id=3109e5ae0311e937d49a5325134e50b742ac5f4a
> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=861928f4e60e826cd8871c0c37f4b3d825b8d81d
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/ata/pata_gayle.c?id=9ab27d1d35fda0c5fce624083e92546a8545e7e5

The community around the m68k CPU is constantly developing new hardware
(new accelerator boards, networking cards, IDE controllers etc for the
Amiga and so on). So, the community and the port are anything but dead.

>> Yeah, it's pretty sad how few commits some of these filesystems have
>> had in recent years.  One can argue that they're stable and don't need
>> to be fixed because they aren't broken, but I find it hard to believe
>> that any of them were better-implemented than ext2 which still sees
>> regular bugfixes.

Exactly. It works fine as is:

root@elgar:~> uname -a
Linux elgar 4.16.0-rc2-amiga-16784-ga8917fc #650 Mon Mar 5 15:32:52 NZDT 2018 m68k GNU/Linux
root@elgar:~> mount /dev/sda1 /mnt -taffs
root@elgar:~> ls -l /mnt | head
total 0
drwx------ 1 root root      0 Mar 30  2001 Alt
-rw------- 1 root root   1352 Mar 27  1997 Alt.info
drwx------ 1 root root      0 Nov 16 14:39 C
drwx------ 1 root root      0 Mar 27  1997 CS_Fonts
drwx------ 1 root root      0 Mar 27  1997 Classes
-rwx------ 1 root root   1132 Aug 14  1996 Classes.info
drwx------ 1 root root      0 Feb 10  2004 Commodities
-rw------- 1 root root    628 Jan 14  2002 Commodities.info
drwx------ 1 root root      0 Apr 10  1999 CyberTools
root@elgar:~> mount |grep affs
/dev/sda1 on /mnt type affs (rw,relatime,bs=512,volume=:)
root@elgar:~>

There is nothing at the moment that needs fixing.

> Regarding affs there is a severe issue which is not in affs itself but
> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
> partitioning standard, which is far more advanced than MBR: It overruns
> for 2 TB or larger drives and then wraps over to the beginning of the
> drive – I bet you can imagine what happens if you write to an area
> larger than 2 TB. I learned this with an external 2TB RDB partitioned
> harddisk back then, which I used for Sam440ep (a kind of successor for
> old, classic Amiga hardware) backup + some Linux related stuff in
> another partition.

The usecase for RDB-partitioned disks larger than 2 TiB is rather
obscure, so I don't really consider this a problem. Amigas running
Linux can use GPT for the other disks.

> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
> but never officially put it officially through upstreaming as I offered
> to make a good description and upstream it through Jens Axboe.

Could be an idea to do that.

> I may take this as a reason to… actually follow through this time,
> hopefully remembering all the details in order to provide a meaningful
> patch description – but I think mostly I can do just careful copy and
> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
> 43511, but not 43511 which is more about a safeguarding issue in case of
> future overflows, I still think it would be good to go in in case affs +
> RDB stays in their current places.

That would be cool. Let me know whether you need real Amiga hardware
for testing. We have plenty available.

> However, in case you move affs to staging, I may be less motivated to do
> so, but then I suggest you also move RDB partitioning support to
> staging, cause this is the one that is known to be dangerously badly for
> 2 TB or larger disks. And yeah, I admit I did not follow through with
> having that patch upstreamed. Probably I did not want to be responsible
> in case my description would not have been absolutely accurate or the
> patch breaks something else. I do not have that 2 TB drive anymore and
> don´t feel like setting one up in a suitable way in order to go about
> this patch, but my testing back then was quite elaborate and I still
> feel pretty confident about it.

I wholeheartedly object to move RDB and AFFS to staging and I guess
the Linux/m68k and Debian/m68k community agrees.

> I totally get your motivation, but I would find it somewhat sad to see
> the filesystems you mentioned go into staging. However, as I just shown
> clearly, for the user it may be better, cause there may be unfixed
> dangerous bugs.

No, it's not better for the user if you take something away which
works for 99% of us just fine.

> FUSE might be an interesting approach, but I bet it will
> not solve the maintenance issue. If there is no one maintaining it in
> the kernel, I think its unlikely to find someone adapting it to be a
> FUSE filesystem and maintaining it. And then I am not aware of FUSE
> based partitioning support. (And I think think ideally we´d had a
> microkernel and run all filesystems in userspace processes with a
> defined set of privileges, but that is simply not Linux as it is.)
> 
> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
> AmigaOS 4.1
> https://lkml.org/lkml/2012/6/17/6
> 
> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
> instead of refusing to use it
> https://bugzilla.kernel.org/show_bug.cgi?id=43521
> 
> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
> while OK in AmigaOS 4.1
> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> 
> I forward the relevant mail of Joanne, in
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
> 
> I even have the patch in diff format. And I just checked, the issue is
> still unpatched as of 4.16.3.

Adrian

-- 
  .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-04-26 10:45       ` moving affs + RDB partition support to staging? John Paul Adrian Glaubitz
@ 2018-04-26 10:59         ` David Sterba
  2018-04-26 11:06           ` John Paul Adrian Glaubitz
  2018-05-06  0:59         ` Al Viro
  1 sibling, 1 reply; 73+ messages in thread
From: David Sterba @ 2018-04-26 10:59 UTC (permalink / raw)
  To: John Paul Adrian Glaubitz
  Cc: Martin Steigerwald, Matthew Wilcox, dsterba, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

On Thu, Apr 26, 2018 at 12:45:41PM +0200, John Paul Adrian Glaubitz wrote:
> (adding debian-68k)
> 
> Hi Matthew!
> 
> On 04/26/2018 12:28 PM, Martin Steigerwald wrote:
> > You probably put your stick into a cave with ancient sleeping dragons :)
> 
> Indeed.
> 
> > Added in linux-m68k mailing list, as they likely have an opinion on how
> > to treat affs + RDB partition support. Also added in Jens Axboe about
> > patching that RDB support broken with 2 TB or larger harddisks issue
> > which had been in Linux kernel for 6 years while a patch exists that to
> > my testing back then solves the issue.
> 
> The answer is that we are still very much actively using RDB and AFFS
> supoort in the Linux kernel and if you were to remove it, you would
> directly hit users.

Based on that I think removing affs will not happen, but the upstream
maintenance status should be updated accordingly.

> I know it may sound crazy, but the Linux/m68k port (Atari, Mac, Amiga etc)
> is a very actively used and maintained port which just recently received
> three new drivers:
> 
> > https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.18/scsi-queue&id=3109e5ae0311e937d49a5325134e50b742ac5f4a
> > https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=861928f4e60e826cd8871c0c37f4b3d825b8d81d
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/ata/pata_gayle.c?id=9ab27d1d35fda0c5fce624083e92546a8545e7e5
> 
> The community around the m68k CPU is constantly developing new hardware
> (new accelerator boards, networking cards, IDE controllers etc for the
> Amiga and so on). So, the community and the port are anything but dead.
> 
> > > Yeah, it's pretty sad how few commits some of these filesystems have
> > > had in recent years.  One can argue that they're stable and don't need
> > > to be fixed because they aren't broken, but I find it hard to believe
> > > that any of them were better-implemented than ext2 which still sees
> > > regular bugfixes.
> 
> Exactly. It works fine as is:
...
> There is nothing at the moment that needs fixing.

So, I'm willing to act as upstream maintainer for affs, send pull
requests with fixes if you ever need that (unless you find someone
else).

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-04-26 10:28     ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Martin Steigerwald
  2018-04-26 10:45       ` moving affs + RDB partition support to staging? John Paul Adrian Glaubitz
@ 2018-04-26 11:00       ` Christoph Hellwig
  2018-04-26 11:08       ` Geert Uytterhoeven
  2 siblings, 0 replies; 73+ messages in thread
From: Christoph Hellwig @ 2018-04-26 11:00 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: Matthew Wilcox, dsterba, linux-fsdevel, linux-kernel, Jens Axboe,
	linux-m68k

On Thu, Apr 26, 2018 at 12:28:40PM +0200, Martin Steigerwald wrote:
> Added in linux-m68k mailing list, as they likely have an opinion on how 
> to treat affs + RDB partition support. Also added in Jens Axboe about 
> patching that RDB support broken with 2 TB or larger harddisks issue 
> which had been in Linux kernel for 6 years while a patch exists that to 
> my testing back then solves the issue.

Jens can't do much unless you actually sent said patch to the
linux-block list.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-04-26 10:59         ` David Sterba
@ 2018-04-26 11:06           ` John Paul Adrian Glaubitz
  0 siblings, 0 replies; 73+ messages in thread
From: John Paul Adrian Glaubitz @ 2018-04-26 11:06 UTC (permalink / raw)
  To: dsterba, Martin Steigerwald, Matthew Wilcox, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

On 04/26/2018 12:59 PM, David Sterba wrote:
>> The answer is that we are still very much actively using RDB and AFFS
>> supoort in the Linux kernel and if you were to remove it, you would
>> directly hit users.
> 
> Based on that I think removing affs will not happen, but the upstream
> maintenance status should be updated accordingly.

As a fellow SUSE employee, I'm very happy to hear that someone
from SUSE is picking up the work <3. FWIW, Andreas Schwab from
SUSE maintains an internal port of openSUSE for m68k :).

>> Exactly. It works fine as is:
> ...
>> There is nothing at the moment that needs fixing.
> 
> So, I'm willing to act as upstream maintainer for affs, send pull
> requests with fixes if you ever need that (unless you find someone
> else).

Thanks a thousand times. If we happen to meet at a SUSE event,
I'll invite you to beverage of your choice ;-).

Thanks!
Adrian

-- 
  .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-04-26 10:28     ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Martin Steigerwald
  2018-04-26 10:45       ` moving affs + RDB partition support to staging? John Paul Adrian Glaubitz
  2018-04-26 11:00       ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Christoph Hellwig
@ 2018-04-26 11:08       ` Geert Uytterhoeven
  2018-04-26 23:56         ` Finn Thain
                           ` (3 more replies)
  2 siblings, 4 replies; 73+ messages in thread
From: Geert Uytterhoeven @ 2018-04-26 11:08 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow

Hi Martin,

CC jdow

On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
<martin@lichtvoll.de> wrote:
> You probably put your stick into a cave with ancient sleeping dragons :)
>
> Added in linux-m68k mailing list, as they likely have an opinion on how
> to treat affs + RDB partition support. Also added in Jens Axboe about
> patching that RDB support broken with 2 TB or larger harddisks issue
> which had been in Linux kernel for 6 years while a patch exists that to
> my testing back then solves the issue.

While non-native Linux filesystem support (e.g. affs/isofs/...) could be
handled by FUSE, moving RDB partition support to staging is not an option,
as it is the only partitioning scheme that Amigas can boot from.

If there are bugs in the RDB parser that people run into, they should
be fixed.
If there are limitations in the RDB format on large disks, that's still not
a reason to move it to staging (hi msdos partitioning!).

> Matthew Wilcox - 26.04.18, 04:57:
>> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
>> > I had similar toughts some time ago while browsing the fs/
>> > directory.
>> > Access to the filesystem images can be reimplemented in FUSE, but
>> > other than that, I don't think the in-kernel code would be missed.
>> >
>> > It's hard to know how many users are there. I was curious eg. about
>> > bfs, befs, coda or feevxfs, looked at the last commits and searched
>> > around web if there are any mentions or user community. But as long
>> > as there's somebody listed in MAINTAINERS, the above are not
>> > candidates for moving to staging or deletion.
>>
>> Yeah, it's pretty sad how few commits some of these filesystems have
>> had in recent years.  One can argue that they're stable and don't need
>> to be fixed because they aren't broken, but I find it hard to believe
>> that any of them were better-implemented than ext2 which still sees
>> regular bugfixes.
>
> Regarding affs there is a severe issue which is not in affs itself but
> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
> partitioning standard, which is far more advanced than MBR: It overruns
> for 2 TB or larger drives and then wraps over to the beginning of the
> drive – I bet you can imagine what happens if you write to an area
> larger than 2 TB. I learned this with an external 2TB RDB partitioned
> harddisk back then, which I used for Sam440ep (a kind of successor for
> old, classic Amiga hardware) backup + some Linux related stuff in
> another partition.
>
> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
> but never officially put it officially through upstreaming as I offered
> to make a good description and upstream it through Jens Axboe.
>
> I may take this as a reason to… actually follow through this time,
> hopefully remembering all the details in order to provide a meaningful
> patch description – but I think mostly I can do just careful copy and
> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
> 43511, but not 43511 which is more about a safeguarding issue in case of
> future overflows, I still think it would be good to go in in case affs +
> RDB stays in their current places.
>
> However, in case you move affs to staging, I may be less motivated to do
> so, but then I suggest you also move RDB partitioning support to
> staging, cause this is the one that is known to be dangerously badly for
> 2 TB or larger disks. And yeah, I admit I did not follow through with
> having that patch upstreamed. Probably I did not want to be responsible
> in case my description would not have been absolutely accurate or the
> patch breaks something else. I do not have that 2 TB drive anymore and
> don´t feel like setting one up in a suitable way in order to go about
> this patch, but my testing back then was quite elaborate and I still
> feel pretty confident about it.
>
> I totally get your motivation, but I would find it somewhat sad to see
> the filesystems you mentioned go into staging. However, as I just shown
> clearly, for the user it may be better, cause there may be unfixed
> dangerous bugs. FUSE might be an interesting approach, but I bet it will
> not solve the maintenance issue. If there is no one maintaining it in
> the kernel, I think its unlikely to find someone adapting it to be a
> FUSE filesystem and maintaining it. And then I am not aware of FUSE
> based partitioning support. (And I think think ideally we´d had a
> microkernel and run all filesystems in userspace processes with a
> defined set of privileges, but that is simply not Linux as it is.)
>
> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
> AmigaOS 4.1
> https://lkml.org/lkml/2012/6/17/6
>
> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
> instead of refusing to use it
> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>
> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
> while OK in AmigaOS 4.1
> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>
> I forward the relevant mail of Joanne, in
>
> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>
> I even have the patch in diff format. And I just checked, the issue is
> still unpatched as of 4.16.3.
>
> ----------  Weitergeleitete Nachricht  ----------
>
> Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
> while OK in AmigaOS 4.1
> Datum: Montag, 18. Juni 2012, 03:28:48 CEST
> Von: jdow <jdow@earthlink.net>
> An: Martin Steigerwald <Martin@lichtvoll.de>
> Kopie: Geert Uytterhoeven <geert@linux-m68k.org>, linux-
> kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>, linux-
> m68k@vger.kernel.org
>
> On 2012/06/17 14:20, Martin Steigerwald wrote:
>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>> <Martin@lichtvoll.de> wrote:
>>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>>> | JXFS 64 bit file system
>>>>> |
>>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>>> | system, which means that it reduces data loss if data writes to
>>>>> | the disk are interrupted. It is the fastest and most reliable
>>>>> | file system ever created for AmigaOS.
>>>>>
>>>>> http://www.amigaos.net/content/1/features
>>>>>
>>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
> see
>>>>> what they say about 2 TB limits.
>>>>
>>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>>> 4096?
>>>>
>>>> block/partitions/amiga.c reads the block size from
>>>> RigidDiskBlock.rdb_BlockBytes,
>>>> but after conversion to 512-byte blocks, all further calculations
> are
>>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>>
>>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>>> bytes per block,
>>>> so I'll have to get a deeper look into your RDB first...
>> […]
>>> Note that the work I did on the Linux RDB code eons ago took full
>>> cognizance of the physical and virtual block sizes.
>> […]
>>> I've asked Martin for a digital copy of his RDBs and what he thinks
> the
>>> partition(s) should look like. I should also be told whether the disk
>>> is supposed to be solely Amiga OSs or not. I gather it's not.
>>
>> Its all in the bug report. profile-binary.img should be it.
>>
>> Its small so I attach it.
>>
>> Meanwhile I try to get the currently supported maximum disk size out
> from
>> the OS 4 developers. Maybe JXFS is playing tricks that other
> filesystems do
>> not play or simply has a different implementation.
>>
>> Thanks,
>
> I've sent Jens a proposed fix changing these lines in amiga.c.
> ===8<--- was
>          struct PartitionBlock *pb;
>          int start_sect, nr_sects, blk, part, res = 0;
>          int blksize = 1;        /* Multiplier for disk block size */
> ===8<--- becomes changing line 338 into lines 338-339
>          struct PartitionBlock *pb;
>          sector_t start_sect, nr_sects;
>          int blk, part, res = 0;
>          int blksize = 1;        /* Multiplier for disk block size */
> ===8<---
>
> (I'm working without proper tools at the moment or I'd make a real diff.
> Sorry.)
>
> This fix may not be complete. The RDBs contain provisions for describing
> the physical disk block size and how many physical blocks are used in a
> file system's logical blocks. This MAY not work quite right large
> physical
> block sizes.
>
> {^_^}   Joanne Dow
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> -------------------------------------------------------------
> […]
>
>> As long as there's someone who can be pestered with bugs, I don't see
>> the need to push their baby out of the tree.  I'm a bit more nervous
>> about hfsplus; if it has users and no maintainer, those users are at
>> risk.
>>
>> Perhaps we could advertise on Craigslist or something ... maintainer
>> wanted for LTR.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-04-26 11:08       ` Geert Uytterhoeven
@ 2018-04-26 23:56         ` Finn Thain
  2018-04-27  1:43           ` moving affs + RDB partition support to staging? jdow
  2018-04-27  1:26         ` jdow
                           ` (2 subsequent siblings)
  3 siblings, 1 reply; 73+ messages in thread
From: Finn Thain @ 2018-04-26 23:56 UTC (permalink / raw)
  To: Geert Uytterhoeven
  Cc: Martin Steigerwald, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow

On Thu, 26 Apr 2018, Geert Uytterhoeven wrote:

> 
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be 
> handled by FUSE

Moving to FUSE is a great divide-and-conquer strategy for those who just 
want the code to die and don't care about any of the data in that format.

If there is a maintainence burden that can be shared then it should be 
shared -- until it can be established that there is no data of value in 
that format.

> moving RDB partition support to staging is not an option, as it is the 
> only partitioning scheme that Amigas can boot from.
> 

Whether or not the original hardware is in use is mostly irrelevant.

As long as the old format is accessible using current hardware, the data 
in that format remains accessible (to archivists, to curators, to your 
decendents, etc).

> If there are bugs in the RDB parser that people run into, they should be 
> fixed. If there are limitations in the RDB format on large disks, that's 
> still not a reason to move it to staging (hi msdos partitioning!).
> 

-- 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-26  6:11 ` Pavel Machek
  2018-04-26 10:36   ` Martin Steigerwald
@ 2018-04-27  1:10   ` Luis R. Rodriguez
  2018-04-29 12:07   ` Greg KH
  2 siblings, 0 replies; 73+ messages in thread
From: Luis R. Rodriguez @ 2018-04-27  1:10 UTC (permalink / raw)
  To: Pavel Machek; +Cc: Matthew Wilcox, Linux FS Devel, linux-kernel

On Wed, Apr 25, 2018 at 11:11 PM, Pavel Machek <pavel@ucw.cz> wrote:
> On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
>> Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
>> developers report bugs in hfs, which they deem a security hole because
>> Ubuntu attempts to automount an inserted USB device as hfs.
>
> We promise "no-regressions" for code in main repository, no such
> promise for staging. We have quite a lot of code without maintainer.
>
> Moving code to staging means it will get broken -- staging was not
> designed for this. I believe moving anything there is bad idea.
>
> Staging is for ugly code, not for code that needs new maintainter.

Staging is also a hospice for drivers on their way out. Can some of
these be eventually be axed?

  Luis

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-04-26 11:08       ` Geert Uytterhoeven
  2018-04-26 23:56         ` Finn Thain
@ 2018-04-27  1:26         ` jdow
  2018-05-06  8:52           ` John Paul Adrian Glaubitz
  2018-04-27  2:11         ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
  2018-04-27  8:01         ` Martin Steigerwald
  3 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-04-27  1:26 UTC (permalink / raw)
  To: Geert Uytterhoeven, Martin Steigerwald
  Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k

OK guys, I am working from th ememory of a 74 year old with a severe head cold. 
The filesystems were always limited to 2 TB since the seek command was signed 32 
bit integer number of BYTEs. So block size has nothing to do with this 2TB 
issue. That's something else in the descriptions.

The RDBs grew a block size and sector size concept in describing partitions. One 
had to be a power of two multiple of the other, as I recall. The relationship 
worked two ways. If you had a Fujitsu Magneto Optical drive with 2k physical 
sectors you could still allocate space in virtual sectors, block sizes, of 512 
bytes. Or you could use much larger blocks than sector size to cut down on the 
unallocated blocks list. Of course, all OSs do that these days. With a single 
unsigned 16 bit integer your address space would fill up a 32 GB with 512 byte 
blocks. We could see this would blow out so we decribed block size and disk size 
in blocks with 32 bit integers giving 4 TB of 1 BYTE blocks. Supposing the file 
sizes stored are such as to support an 8k block size that means describing a 
very large disk should be very easy, even if block size was a 16 bit unsigned 
integer.

To carry this forward more accurately I'll have to dig up some old archival data 
and see what it actually says. This isn't going to happen when I sit here 
sniffling. Maybe in a few days I can dredge up the actual numbers. I seem to 
remember at the time that without a 64 bit OS the disk addressing limit was 2 
TB, since seeks were signed integers. The disk descriptions could be 
considerably larger.

And before I forget there are two features of the RDBs that I heartily recommend 
never implementing on Linux. They were good ideas at the time; but, times 
changed. The RDBs are capable of storing a filesystem driver and some drive init 
code for the plugin disk driver card. That is giving malware authors entirely 
goo easy a shot at owning a machine. Martin S., I would strongly suggest that 
going forward those two capabilities be removed from the RDB readers in AmigaOS 
as well as Linux OS.

I also note that at one time the RDB parsing facility in Linux benefitted from 
my hands on the code and handled the block size/sector size issues very nicely. 
I had an 18 GB disk I wanted to be able to read on Linux. I seem to remember 
somebody at one time had a "this doesn't make any real world sense" event and 
had to be reminded that the real world was wider than he envisioned. I don't 
know what has happened since then.

{^_^}   Joanne Dow

On 20180426 04:08, Geert Uytterhoeven wrote:
> Hi Martin,
> 
> CC jdow
> 
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
> <martin@lichtvoll.de> wrote:
>> You probably put your stick into a cave with ancient sleeping dragons :)
>>
>> Added in linux-m68k mailing list, as they likely have an opinion on how
>> to treat affs + RDB partition support. Also added in Jens Axboe about
>> patching that RDB support broken with 2 TB or larger harddisks issue
>> which had been in Linux kernel for 6 years while a patch exists that to
>> my testing back then solves the issue.
> 
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
> handled by FUSE, moving RDB partition support to staging is not an option,
> as it is the only partitioning scheme that Amigas can boot from.
> 
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's still not
> a reason to move it to staging (hi msdos partitioning!).
> 
>> Matthew Wilcox - 26.04.18, 04:57:
>>> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
>>>> I had similar toughts some time ago while browsing the fs/
>>>> directory.
>>>> Access to the filesystem images can be reimplemented in FUSE, but
>>>> other than that, I don't think the in-kernel code would be missed.
>>>>
>>>> It's hard to know how many users are there. I was curious eg. about
>>>> bfs, befs, coda or feevxfs, looked at the last commits and searched
>>>> around web if there are any mentions or user community. But as long
>>>> as there's somebody listed in MAINTAINERS, the above are not
>>>> candidates for moving to staging or deletion.
>>>
>>> Yeah, it's pretty sad how few commits some of these filesystems have
>>> had in recent years.  One can argue that they're stable and don't need
>>> to be fixed because they aren't broken, but I find it hard to believe
>>> that any of them were better-implemented than ext2 which still sees
>>> regular bugfixes.
>>
>> Regarding affs there is a severe issue which is not in affs itself but
>> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
>> partitioning standard, which is far more advanced than MBR: It overruns
>> for 2 TB or larger drives and then wraps over to the beginning of the
>> drive – I bet you can imagine what happens if you write to an area
>> larger than 2 TB. I learned this with an external 2TB RDB partitioned
>> harddisk back then, which I used for Sam440ep (a kind of successor for
>> old, classic Amiga hardware) backup + some Linux related stuff in
>> another partition.
>>
>> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
>> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
>> but never officially put it officially through upstreaming as I offered
>> to make a good description and upstream it through Jens Axboe.
>>
>> I may take this as a reason to… actually follow through this time,
>> hopefully remembering all the details in order to provide a meaningful
>> patch description – but I think mostly I can do just careful copy and
>> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
>> 43511, but not 43511 which is more about a safeguarding issue in case of
>> future overflows, I still think it would be good to go in in case affs +
>> RDB stays in their current places.
>>
>> However, in case you move affs to staging, I may be less motivated to do
>> so, but then I suggest you also move RDB partitioning support to
>> staging, cause this is the one that is known to be dangerously badly for
>> 2 TB or larger disks. And yeah, I admit I did not follow through with
>> having that patch upstreamed. Probably I did not want to be responsible
>> in case my description would not have been absolutely accurate or the
>> patch breaks something else. I do not have that 2 TB drive anymore and
>> don´t feel like setting one up in a suitable way in order to go about
>> this patch, but my testing back then was quite elaborate and I still
>> feel pretty confident about it.
>>
>> I totally get your motivation, but I would find it somewhat sad to see
>> the filesystems you mentioned go into staging. However, as I just shown
>> clearly, for the user it may be better, cause there may be unfixed
>> dangerous bugs. FUSE might be an interesting approach, but I bet it will
>> not solve the maintenance issue. If there is no one maintaining it in
>> the kernel, I think its unlikely to find someone adapting it to be a
>> FUSE filesystem and maintaining it. And then I am not aware of FUSE
>> based partitioning support. (And I think think ideally we´d had a
>> microkernel and run all filesystems in userspace processes with a
>> defined set of privileges, but that is simply not Linux as it is.)
>>
>> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
>> AmigaOS 4.1
>> https://lkml.org/lkml/2012/6/17/6
>>
>> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
>> instead of refusing to use it
>> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>>
>> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>
>> I forward the relevant mail of Joanne, in
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>>
>> I even have the patch in diff format. And I just checked, the issue is
>> still unpatched as of 4.16.3.
>>
>> ----------  Weitergeleitete Nachricht  ----------
>>
>> Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> Datum: Montag, 18. Juni 2012, 03:28:48 CEST
>> Von: jdow <jdow@earthlink.net>
>> An: Martin Steigerwald <Martin@lichtvoll.de>
>> Kopie: Geert Uytterhoeven <geert@linux-m68k.org>, linux-
>> kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>, linux-
>> m68k@vger.kernel.org
>>
>> On 2012/06/17 14:20, Martin Steigerwald wrote:
>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>>> <Martin@lichtvoll.de> wrote:
>>>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>>>> | JXFS 64 bit file system
>>>>>> |
>>>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>>>> | system, which means that it reduces data loss if data writes to
>>>>>> | the disk are interrupted. It is the fastest and most reliable
>>>>>> | file system ever created for AmigaOS.
>>>>>>
>>>>>> http://www.amigaos.net/content/1/features
>>>>>>
>>>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
>> see
>>>>>> what they say about 2 TB limits.
>>>>>
>>>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>>>> 4096?
>>>>>
>>>>> block/partitions/amiga.c reads the block size from
>>>>> RigidDiskBlock.rdb_BlockBytes,
>>>>> but after conversion to 512-byte blocks, all further calculations
>> are
>>>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>>>
>>>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>>>> bytes per block,
>>>>> so I'll have to get a deeper look into your RDB first...
>>> […]
>>>> Note that the work I did on the Linux RDB code eons ago took full
>>>> cognizance of the physical and virtual block sizes.
>>> […]
>>>> I've asked Martin for a digital copy of his RDBs and what he thinks
>> the
>>>> partition(s) should look like. I should also be told whether the disk
>>>> is supposed to be solely Amiga OSs or not. I gather it's not.
>>>
>>> Its all in the bug report. profile-binary.img should be it.
>>>
>>> Its small so I attach it.
>>>
>>> Meanwhile I try to get the currently supported maximum disk size out
>> from
>>> the OS 4 developers. Maybe JXFS is playing tricks that other
>> filesystems do
>>> not play or simply has a different implementation.
>>>
>>> Thanks,
>>
>> I've sent Jens a proposed fix changing these lines in amiga.c.
>> ===8<--- was
>>           struct PartitionBlock *pb;
>>           int start_sect, nr_sects, blk, part, res = 0;
>>           int blksize = 1;        /* Multiplier for disk block size */
>> ===8<--- becomes changing line 338 into lines 338-339
>>           struct PartitionBlock *pb;
>>           sector_t start_sect, nr_sects;
>>           int blk, part, res = 0;
>>           int blksize = 1;        /* Multiplier for disk block size */
>> ===8<---
>>
>> (I'm working without proper tools at the moment or I'd make a real diff.
>> Sorry.)
>>
>> This fix may not be complete. The RDBs contain provisions for describing
>> the physical disk block size and how many physical blocks are used in a
>> file system's logical blocks. This MAY not work quite right large
>> physical
>> block sizes.
>>
>> {^_^}   Joanne Dow
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> -------------------------------------------------------------
>> […]
>>
>>> As long as there's someone who can be pestered with bugs, I don't see
>>> the need to push their baby out of the tree.  I'm a bit more nervous
>>> about hfsplus; if it has users and no maintainer, those users are at
>>> risk.
>>>
>>> Perhaps we could advertise on Craigslist or something ... maintainer
>>> wanted for LTR.
> 
> Gr{oetje,eeting}s,
> 
>                          Geert
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-04-26 23:56         ` Finn Thain
@ 2018-04-27  1:43           ` jdow
  0 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-04-27  1:43 UTC (permalink / raw)
  To: Finn Thain, Geert Uytterhoeven
  Cc: Martin Steigerwald, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k

On 20180426 16:56, Finn Thain wrote:
> On Thu, 26 Apr 2018, Geert Uytterhoeven wrote:
> 
>>
>> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
>> handled by FUSE
> 
> Moving to FUSE is a great divide-and-conquer strategy for those who just
> want the code to die and don't care about any of the data in that format.
> 
> If there is a maintainence burden that can be shared then it should be
> shared -- until it can be established that there is no data of value in
> that format.
> 
>> moving RDB partition support to staging is not an option, as it is the
>> only partitioning scheme that Amigas can boot from.
>>
> 
> Whether or not the original hardware is in use is mostly irrelevant.
> 
> As long as the old format is accessible using current hardware, the data
> in that format remains accessible (to archivists, to curators, to your
> decendents, etc).
> 
>> If there are bugs in the RDB parser that people run into, they should be
>> fixed. If there are limitations in the RDB format on large disks, that's
>> still not a reason to move it to staging (hi msdos partitioning!).
>>

This intrepid cyberunit is inclined to suggest that understanding the RDBs can 
go a long way towards defining if there is a bug somewhere and whether it is in 
the RDB description or its misuse.

There are some things RDBs can do that REALLY REALLY don't make sense until you 
run across the situation which called for it creation. There are two variables 
that suggest some blocks at the beginning and the end, respectively, of a 
partition are not accessible by the OS. I have used these facilities to 
"interleave" partitions and RDBs. I have built a disk which reserved about 128 
512 byte blocks for RDBs plus filesystem code (which probably should be 
abandoned) which embedded the RDBs describing the partition within the 
partition. Then I reserved space at the end of the partition and embedded a 
second partition in that space. As absurd as it sounds this had at one time a 
decent use case. Disk space was an expensive premium in those days so wasting 
space to get nice integer numbers in the disk description, which was phony for a 
hard disk in any case, we allowed any numbers and if that went past the end of 
the disk we reserved the necessary space so that it would never be used. The 
space at the beginning of a partition was needed in any case because a one block 
partition signature needed space at the start of the partition. It held the 
filesystem's signature, OFS, AFS, SFS, etc.

There is also a good reason for allowing the anchor for the RDBs to start in any 
of the first 16 blocks with a recommendation not to use block 0 as other FSs 
used that. And we wanted to accommodate at least two different partition 
description technologies to work on the disk. My code always placed the RDBs at 
block 3.

I hope passing along some of this history will mitigate some fo the feelings 
that RDBs are inherently flawed or full of bugs or whatnot. (Full pf security 
holes is another story. DriveInit code and filesystem code have worried me from 
day one.)

{^_^}   Joanne

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-04-26 11:08       ` Geert Uytterhoeven
  2018-04-26 23:56         ` Finn Thain
  2018-04-27  1:26         ` jdow
@ 2018-04-27  2:11         ` Michael Schmitz
  2018-06-24  9:06           ` Martin Steigerwald
  2018-04-27  8:01         ` Martin Steigerwald
  3 siblings, 1 reply; 73+ messages in thread
From: Michael Schmitz @ 2018-04-27  2:11 UTC (permalink / raw)
  To: Geert Uytterhoeven
  Cc: Martin Steigerwald, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow

Hi Geert,

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

(MSDOS partition support is not the only one with limitations - the
SGI partition parser also uses int types. Brings back memories of /me
hacking a large disk into many small partitions because Irix couldn't
digest it whole!)

Going to look into Atari partition format limitations now ...

Cheers,

  Michael


On Thu, Apr 26, 2018 at 11:08 PM, Geert Uytterhoeven
<geert@linux-m68k.org> wrote:
> Hi Martin,
>
> CC jdow
>
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
> <martin@lichtvoll.de> wrote:
>> You probably put your stick into a cave with ancient sleeping dragons :)
>>
>> Added in linux-m68k mailing list, as they likely have an opinion on how
>> to treat affs + RDB partition support. Also added in Jens Axboe about
>> patching that RDB support broken with 2 TB or larger harddisks issue
>> which had been in Linux kernel for 6 years while a patch exists that to
>> my testing back then solves the issue.
>
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
> handled by FUSE, moving RDB partition support to staging is not an option,
> as it is the only partitioning scheme that Amigas can boot from.
>
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's still not
> a reason to move it to staging (hi msdos partitioning!).
>
>> Matthew Wilcox - 26.04.18, 04:57:
>>> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
>>> > I had similar toughts some time ago while browsing the fs/
>>> > directory.
>>> > Access to the filesystem images can be reimplemented in FUSE, but
>>> > other than that, I don't think the in-kernel code would be missed.
>>> >
>>> > It's hard to know how many users are there. I was curious eg. about
>>> > bfs, befs, coda or feevxfs, looked at the last commits and searched
>>> > around web if there are any mentions or user community. But as long
>>> > as there's somebody listed in MAINTAINERS, the above are not
>>> > candidates for moving to staging or deletion.
>>>
>>> Yeah, it's pretty sad how few commits some of these filesystems have
>>> had in recent years.  One can argue that they're stable and don't need
>>> to be fixed because they aren't broken, but I find it hard to believe
>>> that any of them were better-implemented than ext2 which still sees
>>> regular bugfixes.
>>
>> Regarding affs there is a severe issue which is not in affs itself but
>> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
>> partitioning standard, which is far more advanced than MBR: It overruns
>> for 2 TB or larger drives and then wraps over to the beginning of the
>> drive – I bet you can imagine what happens if you write to an area
>> larger than 2 TB. I learned this with an external 2TB RDB partitioned
>> harddisk back then, which I used for Sam440ep (a kind of successor for
>> old, classic Amiga hardware) backup + some Linux related stuff in
>> another partition.
>>
>> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
>> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
>> but never officially put it officially through upstreaming as I offered
>> to make a good description and upstream it through Jens Axboe.
>>
>> I may take this as a reason to… actually follow through this time,
>> hopefully remembering all the details in order to provide a meaningful
>> patch description – but I think mostly I can do just careful copy and
>> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
>> 43511, but not 43511 which is more about a safeguarding issue in case of
>> future overflows, I still think it would be good to go in in case affs +
>> RDB stays in their current places.
>>
>> However, in case you move affs to staging, I may be less motivated to do
>> so, but then I suggest you also move RDB partitioning support to
>> staging, cause this is the one that is known to be dangerously badly for
>> 2 TB or larger disks. And yeah, I admit I did not follow through with
>> having that patch upstreamed. Probably I did not want to be responsible
>> in case my description would not have been absolutely accurate or the
>> patch breaks something else. I do not have that 2 TB drive anymore and
>> don´t feel like setting one up in a suitable way in order to go about
>> this patch, but my testing back then was quite elaborate and I still
>> feel pretty confident about it.
>>
>> I totally get your motivation, but I would find it somewhat sad to see
>> the filesystems you mentioned go into staging. However, as I just shown
>> clearly, for the user it may be better, cause there may be unfixed
>> dangerous bugs. FUSE might be an interesting approach, but I bet it will
>> not solve the maintenance issue. If there is no one maintaining it in
>> the kernel, I think its unlikely to find someone adapting it to be a
>> FUSE filesystem and maintaining it. And then I am not aware of FUSE
>> based partitioning support. (And I think think ideally we´d had a
>> microkernel and run all filesystems in userspace processes with a
>> defined set of privileges, but that is simply not Linux as it is.)
>>
>> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
>> AmigaOS 4.1
>> https://lkml.org/lkml/2012/6/17/6
>>
>> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
>> instead of refusing to use it
>> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>>
>> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>
>> I forward the relevant mail of Joanne, in
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>>
>> I even have the patch in diff format. And I just checked, the issue is
>> still unpatched as of 4.16.3.
>>
>> ----------  Weitergeleitete Nachricht  ----------
>>
>> Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> Datum: Montag, 18. Juni 2012, 03:28:48 CEST
>> Von: jdow <jdow@earthlink.net>
>> An: Martin Steigerwald <Martin@lichtvoll.de>
>> Kopie: Geert Uytterhoeven <geert@linux-m68k.org>, linux-
>> kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>, linux-
>> m68k@vger.kernel.org
>>
>> On 2012/06/17 14:20, Martin Steigerwald wrote:
>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>>> <Martin@lichtvoll.de> wrote:
>>>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>>>> | JXFS 64 bit file system
>>>>>> |
>>>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>>>> | system, which means that it reduces data loss if data writes to
>>>>>> | the disk are interrupted. It is the fastest and most reliable
>>>>>> | file system ever created for AmigaOS.
>>>>>>
>>>>>> http://www.amigaos.net/content/1/features
>>>>>>
>>>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
>> see
>>>>>> what they say about 2 TB limits.
>>>>>
>>>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>>>> 4096?
>>>>>
>>>>> block/partitions/amiga.c reads the block size from
>>>>> RigidDiskBlock.rdb_BlockBytes,
>>>>> but after conversion to 512-byte blocks, all further calculations
>> are
>>>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>>>
>>>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>>>> bytes per block,
>>>>> so I'll have to get a deeper look into your RDB first...
>>> […]
>>>> Note that the work I did on the Linux RDB code eons ago took full
>>>> cognizance of the physical and virtual block sizes.
>>> […]
>>>> I've asked Martin for a digital copy of his RDBs and what he thinks
>> the
>>>> partition(s) should look like. I should also be told whether the disk
>>>> is supposed to be solely Amiga OSs or not. I gather it's not.
>>>
>>> Its all in the bug report. profile-binary.img should be it.
>>>
>>> Its small so I attach it.
>>>
>>> Meanwhile I try to get the currently supported maximum disk size out
>> from
>>> the OS 4 developers. Maybe JXFS is playing tricks that other
>> filesystems do
>>> not play or simply has a different implementation.
>>>
>>> Thanks,
>>
>> I've sent Jens a proposed fix changing these lines in amiga.c.
>> ===8<--- was
>>          struct PartitionBlock *pb;
>>          int start_sect, nr_sects, blk, part, res = 0;
>>          int blksize = 1;        /* Multiplier for disk block size */
>> ===8<--- becomes changing line 338 into lines 338-339
>>          struct PartitionBlock *pb;
>>          sector_t start_sect, nr_sects;
>>          int blk, part, res = 0;
>>          int blksize = 1;        /* Multiplier for disk block size */
>> ===8<---
>>
>> (I'm working without proper tools at the moment or I'd make a real diff.
>> Sorry.)
>>
>> This fix may not be complete. The RDBs contain provisions for describing
>> the physical disk block size and how many physical blocks are used in a
>> file system's logical blocks. This MAY not work quite right large
>> physical
>> block sizes.
>>
>> {^_^}   Joanne Dow
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> -------------------------------------------------------------
>> […]
>>
>>> As long as there's someone who can be pestered with bugs, I don't see
>>> the need to push their baby out of the tree.  I'm a bit more nervous
>>> about hfsplus; if it has users and no maintainer, those users are at
>>> risk.
>>>
>>> Perhaps we could advertise on Craigslist or something ... maintainer
>>> wanted for LTR.
>
> Gr{oetje,eeting}s,
>
>                         Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-04-26 11:08       ` Geert Uytterhoeven
                           ` (2 preceding siblings ...)
  2018-04-27  2:11         ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
@ 2018-04-27  8:01         ` Martin Steigerwald
  3 siblings, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-04-27  8:01 UTC (permalink / raw)
  To: Geert Uytterhoeven
  Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow

Geert Uytterhoeven - 26.04.18, 13:08:
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
> 
> <martin@lichtvoll.de> wrote:
> > You probably put your stick into a cave with ancient sleeping
> > dragons 
> > 
> > Added in linux-m68k mailing list, as they likely have an opinion on
> > how to treat affs + RDB partition support. Also added in Jens Axboe
> > about patching that RDB support broken with 2 TB or larger
> > harddisks issue which had been in Linux kernel for 6 years while a
> > patch exists that to my testing back then solves the issue.
[…]
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's
> still not a reason to move it to staging (hi msdos partitioning!).

What I ran into was *not* a limitation in the RDB format, but a bug in 
the Linux implementation on it. After Joanne Dow´s change the 2 TB disk 
was detected and handled properly by Linux. Also AmigaOS 4.x handles 
those disks just well and I think also AmigaOS 3.1/3.5/3.9 supported 
them, but I am not sure on the details on that, it has been a long time 
since I last booted one of my Amiga systems.

Many classic Amiga users may not deal with such large disks, but AmigaOS 
4 users probably still, and some of them may run Linux on their PowerPC 
motherboards as well.

So I think the issue is worth fixing and am looking into submitting the 
fix, which looks pretty straight-forward to me upstream unless someone 
bets me to it.

Thanks,
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-26  6:11 ` Pavel Machek
  2018-04-26 10:36   ` Martin Steigerwald
  2018-04-27  1:10   ` Luis R. Rodriguez
@ 2018-04-29 12:07   ` Greg KH
  2018-04-29 20:07     ` Ondrej Zary
  2 siblings, 1 reply; 73+ messages in thread
From: Greg KH @ 2018-04-29 12:07 UTC (permalink / raw)
  To: Pavel Machek; +Cc: Matthew Wilcox, linux-fsdevel, linux-kernel

On Thu, Apr 26, 2018 at 08:11:08AM +0200, Pavel Machek wrote:
> On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
> > developers report bugs in hfs, which they deem a security hole because
> > Ubuntu attempts to automount an inserted USB device as hfs.
> 
> We promise "no-regressions" for code in main repository, no such
> promise for staging. We have quite a lot of code without maintainer.
> 
> Moving code to staging means it will get broken -- staging was not
> designed for this. I believe moving anything there is bad idea.
> 
> Staging is for ugly code, not for code that needs new maintainter.

Staging is used for getting code _out_ of the kernel tree as well as
_in_.  We use it all the time to move code there, see if anyone shows up
in 6-8 months to say "I will fix this!", and if not, we delete it.

Look at what just happened to IRDA in the 4.17-rc1 release as an example
of this.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-29 12:07   ` Greg KH
@ 2018-04-29 20:07     ` Ondrej Zary
  2018-04-29 23:37       ` Greg KH
  0 siblings, 1 reply; 73+ messages in thread
From: Ondrej Zary @ 2018-04-29 20:07 UTC (permalink / raw)
  To: Greg KH; +Cc: Pavel Machek, Matthew Wilcox, linux-fsdevel, linux-kernel

On Sunday 29 April 2018 14:07:05 Greg KH wrote:
> On Thu, Apr 26, 2018 at 08:11:08AM +0200, Pavel Machek wrote:
> > On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > > Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
> > > developers report bugs in hfs, which they deem a security hole because
> > > Ubuntu attempts to automount an inserted USB device as hfs.
> >
> > We promise "no-regressions" for code in main repository, no such
> > promise for staging. We have quite a lot of code without maintainer.
> >
> > Moving code to staging means it will get broken -- staging was not
> > designed for this. I believe moving anything there is bad idea.
> >
> > Staging is for ugly code, not for code that needs new maintainter.
>
> Staging is used for getting code _out_ of the kernel tree as well as
> _in_.  We use it all the time to move code there, see if anyone shows up
> in 6-8 months to say "I will fix this!", and if not, we delete it.
>
> Look at what just happened to IRDA in the 4.17-rc1 release as an example
> of this.

Really a "great" example of deleting working code :( 

-- 
Ondrej Zary

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-29 20:07     ` Ondrej Zary
@ 2018-04-29 23:37       ` Greg KH
  2018-05-01 10:14         ` Pavel Machek
  0 siblings, 1 reply; 73+ messages in thread
From: Greg KH @ 2018-04-29 23:37 UTC (permalink / raw)
  To: Ondrej Zary; +Cc: Pavel Machek, Matthew Wilcox, linux-fsdevel, linux-kernel

On Sun, Apr 29, 2018 at 10:07:26PM +0200, Ondrej Zary wrote:
> On Sunday 29 April 2018 14:07:05 Greg KH wrote:
> > On Thu, Apr 26, 2018 at 08:11:08AM +0200, Pavel Machek wrote:
> > > On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > > > Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
> > > > developers report bugs in hfs, which they deem a security hole because
> > > > Ubuntu attempts to automount an inserted USB device as hfs.
> > >
> > > We promise "no-regressions" for code in main repository, no such
> > > promise for staging. We have quite a lot of code without maintainer.
> > >
> > > Moving code to staging means it will get broken -- staging was not
> > > designed for this. I believe moving anything there is bad idea.
> > >
> > > Staging is for ugly code, not for code that needs new maintainter.
> >
> > Staging is used for getting code _out_ of the kernel tree as well as
> > _in_.  We use it all the time to move code there, see if anyone shows up
> > in 6-8 months to say "I will fix this!", and if not, we delete it.
> >
> > Look at what just happened to IRDA in the 4.17-rc1 release as an example
> > of this.
> 
> Really a "great" example of deleting working code :( 

What do you mean?  The irda code was broken and not working at all.
There were loads of bug reports about it for years, with no developers
or maintainers willing to do the work on it to get it to actually work
again.

If someone does want to step up and do it, great!  It's a simple revert
of two git commits and they are back in business.

Dropping code from the tree is not like it is gone for forever.  If
someone wants to pick it up, it is trivial to do so.  Git is good :)

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-29 23:37       ` Greg KH
@ 2018-05-01 10:14         ` Pavel Machek
  0 siblings, 0 replies; 73+ messages in thread
From: Pavel Machek @ 2018-05-01 10:14 UTC (permalink / raw)
  To: Greg KH; +Cc: Ondrej Zary, Matthew Wilcox, linux-fsdevel, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 2353 bytes --]

On Sun 2018-04-29 16:37:37, Greg KH wrote:
> On Sun, Apr 29, 2018 at 10:07:26PM +0200, Ondrej Zary wrote:
> > On Sunday 29 April 2018 14:07:05 Greg KH wrote:
> > > On Thu, Apr 26, 2018 at 08:11:08AM +0200, Pavel Machek wrote:
> > > > On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > > > > Recently ncpfs got moved to staging.  Also recently, we had some fuzzer
> > > > > developers report bugs in hfs, which they deem a security hole because
> > > > > Ubuntu attempts to automount an inserted USB device as hfs.
> > > >
> > > > We promise "no-regressions" for code in main repository, no such
> > > > promise for staging. We have quite a lot of code without maintainer.
> > > >
> > > > Moving code to staging means it will get broken -- staging was not
> > > > designed for this. I believe moving anything there is bad idea.
> > > >
> > > > Staging is for ugly code, not for code that needs new maintainter.
> > >
> > > Staging is used for getting code _out_ of the kernel tree as well as
> > > _in_.  We use it all the time to move code there, see if anyone shows up
> > > in 6-8 months to say "I will fix this!", and if not, we delete it.
> > >
> > > Look at what just happened to IRDA in the 4.17-rc1 release as an example
> > > of this.
> > 
> > Really a "great" example of deleting working code :( 
> 
> What do you mean?  The irda code was broken and not working at all.
> There were loads of bug reports about it for years, with no developers
> or maintainers willing to do the work on it to get it to actually work
> again.
> 
> If someone does want to step up and do it, great!  It's a simple revert
> of two git commits and they are back in business.

> Dropping code from the tree is not like it is gone for forever.  If
> someone wants to pick it up, it is trivial to do so.

That is a lie and you know it.

In particular, having code moved to staging means it is going to
bitrot, because it will not be updated with global changes.

Plus coding standards change over time, so if you simply revert,
you'll not be able to simply merge it back.

Plus that revert means bisection is no longer easy/possible to find
the real breakages.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Moving unmaintained filesystems to staging
  2018-04-26 10:36   ` Martin Steigerwald
@ 2018-05-03  9:18     ` Pavel Machek
  0 siblings, 0 replies; 73+ messages in thread
From: Pavel Machek @ 2018-05-03  9:18 UTC (permalink / raw)
  To: Martin Steigerwald; +Cc: Matthew Wilcox, linux-fsdevel, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

On Thu 2018-04-26 12:36:50, Martin Steigerwald wrote:
> Pavel Machek - 26.04.18, 08:11:
> > On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > > Recently ncpfs got moved to staging.  Also recently, we had some
> > > fuzzer developers report bugs in hfs, which they deem a security
> > > hole because Ubuntu attempts to automount an inserted USB device as
> > > hfs.
> > 
> > We promise "no-regressions" for code in main repository, no such
> > promise for staging. We have quite a lot of code without maintainer.
> > 
> > Moving code to staging means it will get broken -- staging was not
> > designed for this. I believe moving anything there is bad idea.
> > 
> > Staging is for ugly code, not for code that needs new maintainter.
> 
> Good point.
> 
> Moving things in and out to some "unmaintained" directory… I am not sure 
> about that either. I tend to think that moving code around does not 
> solve the underlying issue.
> 
> Which, according to what I got from Matthew, was that distributors 
> enable just about every filesystem they can enable which lead to hfs 
> being used for automounting an USB stick formatted with it.

If distro's are shipping it, they are also willing to fix it. So
... no need to drop the code just yet.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-04-26 10:45       ` moving affs + RDB partition support to staging? John Paul Adrian Glaubitz
  2018-04-26 10:59         ` David Sterba
@ 2018-05-06  0:59         ` Al Viro
  2018-05-06  7:40           ` Al Viro
                             ` (2 more replies)
  1 sibling, 3 replies; 73+ messages in thread
From: Al Viro @ 2018-05-06  0:59 UTC (permalink / raw)
  To: John Paul Adrian Glaubitz
  Cc: Martin Steigerwald, Matthew Wilcox, dsterba, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

On Thu, Apr 26, 2018 at 12:45:41PM +0200, John Paul Adrian Glaubitz wrote:

> Exactly. It works fine as is:
> 
> root@elgar:~> uname -a
> Linux elgar 4.16.0-rc2-amiga-16784-ga8917fc #650 Mon Mar 5 15:32:52 NZDT 2018 m68k GNU/Linux
> root@elgar:~> mount /dev/sda1 /mnt -taffs
> root@elgar:~> ls -l /mnt | head
> total 0
> drwx------ 1 root root      0 Mar 30  2001 Alt
> -rw------- 1 root root   1352 Mar 27  1997 Alt.info
> drwx------ 1 root root      0 Nov 16 14:39 C
> drwx------ 1 root root      0 Mar 27  1997 CS_Fonts
> drwx------ 1 root root      0 Mar 27  1997 Classes
> -rwx------ 1 root root   1132 Aug 14  1996 Classes.info
> drwx------ 1 root root      0 Feb 10  2004 Commodities
> -rw------- 1 root root    628 Jan 14  2002 Commodities.info
> drwx------ 1 root root      0 Apr 10  1999 CyberTools
> root@elgar:~> mount |grep affs
> /dev/sda1 on /mnt type affs (rw,relatime,bs=512,volume=:)
> root@elgar:~>
> 
> There is nothing at the moment that needs fixing.

Funny, that...  I'd been going through the damn thing for the
last week or so; open-by-fhandle/nfs export support is completely
buggered.  And as for the rest... the least said about the error
handling, the better - something like rename() hitting an IO
error (read one) can not only screw the on-disk data into the
ground, it can do seriously bad things to kernel data structures.

Is there anything resembling fsck for that thing, BTW?  Nevermind
the repairs, just the validity checks would be nice...

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06  0:59         ` Al Viro
@ 2018-05-06  7:40           ` Al Viro
  2018-05-06 20:46             ` Al Viro
  2018-05-06  8:40           ` John Paul Adrian Glaubitz
  2018-05-06 10:12           ` Martin Steigerwald
  2 siblings, 1 reply; 73+ messages in thread
From: Al Viro @ 2018-05-06  7:40 UTC (permalink / raw)
  To: John Paul Adrian Glaubitz
  Cc: Martin Steigerwald, Matthew Wilcox, dsterba, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

On Sun, May 06, 2018 at 01:59:51AM +0100, Al Viro wrote:

> > There is nothing at the moment that needs fixing.
> 
> Funny, that...  I'd been going through the damn thing for the
> last week or so; open-by-fhandle/nfs export support is completely
> buggered.  And as for the rest... the least said about the error
> handling, the better - something like rename() hitting an IO
> error (read one) can not only screw the on-disk data into the
> ground, it can do seriously bad things to kernel data structures.

... and while we are at it, consider the following:

mkdir a
mkdir b
touch a/x
ln a/x b

<umount and mount again, to get cold dcache, or just wait for memory
pressure to evict those suckers>

process A: unlink("a/x");
process B: open("b/x");

We had two entries - one in a, another in b; the one in a is ST_FILE one,
the one in b - ST_LINKFILE, with ->original refering to the ST_FILE one.

unlink() needs to kick the entry out of a; since it can't leave an
ST_FILE not included into any directory (and since the file should live on
due to b/x still being there) it ends up removing ST_LINKFILE entry from
b and moving ST_FILE one in its place.  That happens in affs_remove_link().

However, open() gets there just as this surgery is about to begin.
It gets to affs_lookup(), which finds the entry for b/x, reads it,
stashes block number of that entry into dentry->d_fsdata, notices
that it's ST_LINKFILE one, picks the block number of ST_FILE one
out of it and proceeds to look up the inode; that doesn't end up
doing any work (the same inode is in icache due to a/x having been
looked up by unlink(2)).

In the meanwhile, since the hash table of b has been unlocked once
we'd done hash lookup, affs_remove_link() can proceed with the
surgery.  It *does* try to catch dentries with ->d_fsdata containing
the block number of sacrificed ST_LINKFILE and reset that to block
number of ST_FILE that gets moved in its place.  However, it does
so by
static void
affs_fix_dcache(struct inode *inode, u32 entry_ino)
{
        struct dentry *dentry;
        spin_lock(&inode->i_lock);
        hlist_for_each_entry(dentry, &inode->i_dentry, d_u.d_alias) {
                if (entry_ino == (u32)(long)dentry->d_fsdata) {
                        dentry->d_fsdata = (void *)inode->i_ino;
                        break;
                }
        }
        spin_unlock(&inode->i_lock);
}
i.e. looping through dentries that point to our inode.  Except that
*our* dentry isn't attached to inode yet, so we are SOL - it's
left with ->d_fsdata pointing to (destroyed) ST_LINKFILE.

After a while process B closes the file and unlinks it.  Take a look
at affs_remove_header() and you'll see how much fun we are in for -
it uses ->d_fsdata to pick the entry to find and remove...

That's an fs corruptor, plain and simple.  As far as I had been able
to reconstruct the history, leftover after Roman's locking changes
17 years ago.  AFAICS, I'd missed it back then - the races I'd spotted
had been dealt with about a year later (when we started to lock the
victim in vfs_unlink/vfs_rmdir/vfs_rename), but this one went unnoticed...

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06  0:59         ` Al Viro
  2018-05-06  7:40           ` Al Viro
@ 2018-05-06  8:40           ` John Paul Adrian Glaubitz
  2018-05-06 10:12           ` Martin Steigerwald
  2 siblings, 0 replies; 73+ messages in thread
From: John Paul Adrian Glaubitz @ 2018-05-06  8:40 UTC (permalink / raw)
  To: Al Viro
  Cc: Martin Steigerwald, Matthew Wilcox, dsterba, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

On 05/06/2018 02:59 AM, Al Viro wrote:
>> There is nothing at the moment that needs fixing.
> 
> Funny, that...  I'd been going through the damn thing for the
> last week or so; open-by-fhandle/nfs export support is completely
> buggered.  And as for the rest... the least said about the error
> handling, the better - something like rename() hitting an IO
> error (read one) can not only screw the on-disk data into the
> ground, it can do seriously bad things to kernel data structures.

Well, yes, there are certainly things in the affs code that can cause
unexpected behavior. But that wasn't my point. My point was that the
purpose for what people are using the affs driver in the kernel, it's
doing its job perfectly fine.

The standard use case is either to run Debian m68k on an Amiga and
access the Amiga partitions through Linux for simple file transfers
between the two operating systems, e.g. copying a new kernel plus
newly generated initrd to the affs boot partition of AmigaOS.

Or, just reading data off an old Amiga harddisk or floppy disk by
either attaching the disk to the Linux machine directly or reading
the contents of a floppy disk with an advanced floppy disk controller
like the Catweasel or KryoFlux.

For both cases, I haven't run into any issues with the affs driver
and I also haven't heard of complaints from anyone in communities
like EAB Amiga, amiga.org, a1k.org etc. That doesn't mean that people
haven't run into issues, of course. I'm just saying I haven't heard
of any.

> Is there anything resembling fsck for that thing, BTW?  Nevermind
> the repairs, just the validity checks would be nice...

Not that I know of.

Adrian

PS: Even Matthew Garrett seems to be a Linux/m68k fan:

    > https://twitter.com/mjg59/status/945779704329629697

    :)

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-04-27  1:26         ` jdow
@ 2018-05-06  8:52           ` John Paul Adrian Glaubitz
  2018-05-06 10:10             ` Martin Steigerwald
  2018-05-07  4:54             ` jdow
  0 siblings, 2 replies; 73+ messages in thread
From: John Paul Adrian Glaubitz @ 2018-05-06  8:52 UTC (permalink / raw)
  To: jdow, Geert Uytterhoeven, Martin Steigerwald
  Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k

On 04/27/2018 03:26 AM, jdow wrote:
> And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times
> changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely
> goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS
> as well as Linux OS.

I assume removing the feature for AmigaOS isn't really possible since we don't have
the source code for that, do we?

Also, if I remember correctly, Mac partitions can store filesystem drivers as well
and its actually a feature being used in MacOS. parted received a patch some time
ago to fix the correct handling for storing the filesystem driver in the partition
table.

I would be generally against removing these features as I don't think the security
risk is relevant for the majority of users. The Amiga is a hobbyist machine these
days and AmigaOS has certainly way more on than way to be compromised through
vulnerabilities.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06  8:52           ` John Paul Adrian Glaubitz
@ 2018-05-06 10:10             ` Martin Steigerwald
  2018-05-07  4:54             ` jdow
  1 sibling, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-05-06 10:10 UTC (permalink / raw)
  To: John Paul Adrian Glaubitz
  Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
	Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
	linux-m68k

John Paul Adrian Glaubitz - 06.05.18, 10:52:
> On 04/27/2018 03:26 AM, jdow wrote:
> > And before I forget there are two features of the RDBs that I
> > heartily recommend never implementing on Linux. They were good
> > ideas at the time; but, times changed. The RDBs are capable of
> > storing a filesystem driver and some drive init code for the plugin
> > disk driver card. That is giving malware authors entirely goo easy
> > a shot at owning a machine. Martin S., I would strongly suggest
> > that going forward those two capabilities be removed from the RDB
> > readers in AmigaOS as well as Linux OS.
> 
> I assume removing the feature for AmigaOS isn't really possible since
> we don't have the source code for that, do we?

AmigaOS 4.x does not support loading filesystems from RDB anymore as far 
as I know. Meanwhile I am not involved with the AmigaOS development 
anymore, but that is the last state I know. Similarily to Linux 
filesystems drivers are loaded as "modules" into the kernel. Its also 
still possible to load a filesystem as a file from disk, but that does 
not work for the filesystem the kernel boots from. The AmigaOS kernel 
still decides which of the kernel filesystem to use according to the 
DOSType of the partition.

Thanks,
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06  0:59         ` Al Viro
  2018-05-06  7:40           ` Al Viro
  2018-05-06  8:40           ` John Paul Adrian Glaubitz
@ 2018-05-06 10:12           ` Martin Steigerwald
  2 siblings, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-05-06 10:12 UTC (permalink / raw)
  To: Al Viro
  Cc: John Paul Adrian Glaubitz, Matthew Wilcox, dsterba,
	linux-fsdevel, linux-kernel, Jens Axboe, linux-m68k, Debian m68k

Al Viro - 06.05.18, 02:59:
> On Thu, Apr 26, 2018 at 12:45:41PM +0200, John Paul Adrian Glaubitz 
wrote:
> > Exactly. It works fine as is:
> > 
> > root@elgar:~> uname -a
> > Linux elgar 4.16.0-rc2-amiga-16784-ga8917fc #650 Mon Mar 5 15:32:52
> > NZDT 2018 m68k GNU/Linux root@elgar:~> mount /dev/sda1 /mnt -taffs
> > root@elgar:~> ls -l /mnt | head
> > total 0
> > drwx------ 1 root root      0 Mar 30  2001 Alt
> > -rw------- 1 root root   1352 Mar 27  1997 Alt.info
> > drwx------ 1 root root      0 Nov 16 14:39 C
> > drwx------ 1 root root      0 Mar 27  1997 CS_Fonts
> > drwx------ 1 root root      0 Mar 27  1997 Classes
> > -rwx------ 1 root root   1132 Aug 14  1996 Classes.info
> > drwx------ 1 root root      0 Feb 10  2004 Commodities
> > -rw------- 1 root root    628 Jan 14  2002 Commodities.info
> > drwx------ 1 root root      0 Apr 10  1999 CyberTools
> > root@elgar:~> mount |grep affs
> > /dev/sda1 on /mnt type affs (rw,relatime,bs=512,volume=:)
> > root@elgar:~>
> > 
> > There is nothing at the moment that needs fixing.
> 
> Funny, that...  I'd been going through the damn thing for the
> last week or so; open-by-fhandle/nfs export support is completely
> buggered.  And as for the rest... the least said about the error
> handling, the better - something like rename() hitting an IO
> error (read one) can not only screw the on-disk data into the
> ground, it can do seriously bad things to kernel data structures.
> 
> Is there anything resembling fsck for that thing, BTW?  Nevermind
> the repairs, just the validity checks would be nice...

I am not aware of the fsck command for affs on Linux. There is a 
partitioning tool called amiga-fdisk, but for checking a filesystem you 
would need to use a tool under AmigaOS.

-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06  7:40           ` Al Viro
@ 2018-05-06 20:46             ` Al Viro
  2018-05-06 20:49               ` John Paul Adrian Glaubitz
  2018-05-06 21:32               ` Al Viro
  0 siblings, 2 replies; 73+ messages in thread
From: Al Viro @ 2018-05-06 20:46 UTC (permalink / raw)
  To: John Paul Adrian Glaubitz
  Cc: Martin Steigerwald, Matthew Wilcox, dsterba, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

On Sun, May 06, 2018 at 08:39:55AM +0100, Al Viro wrote:
> On Sun, May 06, 2018 at 01:59:51AM +0100, Al Viro wrote:
> 
> > > There is nothing at the moment that needs fixing.
> > 
> > Funny, that...  I'd been going through the damn thing for the
> > last week or so; open-by-fhandle/nfs export support is completely
> > buggered.  And as for the rest... the least said about the error
> > handling, the better - something like rename() hitting an IO
> > error (read one) can not only screw the on-disk data into the
> > ground, it can do seriously bad things to kernel data structures.
> 
> ... and while we are at it, consider the following:

[snip]

Another piece of fun: in
affs_add_entry() we have

        retval = affs_insert_hash(dir, bh);
        mark_buffer_dirty_inode(bh, inode);
        affs_unlock_dir(dir);
        affs_unlock_link(inode);

	d_instantiate(dentry, inode);
done:
        affs_brelse(inode_bh);
        affs_brelse(bh);
        return retval;

and in its callers - things like
        error = affs_add_entry(dir, inode, dentry, ST_USERDIR);
        if (error) {
                clear_nlink(inode);
                mark_inode_dirty(inode);
                iput(inode);
                return error;
        }

	Guess what happens if we hit affs_insert_hash() failure?
d_instantiate() doesn't do anything to in-core inode refcount -
that's caller's responsibility.  affs_new_inode() has allocated
an inode with ->i_count equal to 1; d_instantiate() transfers that
reference into dentry (as ->d_inode).  And then, since we'd got
a non-zero error we do hit iput() (same as we would've if an error
happened early enough in affs_add_entry() to bypass d_instantiate()).
That drives ->i_count to 0, getting inode freed (zero link count
when the last in-core reference is dropped means that there's no
point retaining it in icache).
	So in that case we end up with struct dentry (still hashed
and available for lookups to pick) that has ->d_inode pointing
to freed memory.  Welcome to memory corruption...

	I'm fixing that pile of crap (along with the NFS exports
one and, hopefully, rename mess as well).  HOWEVER, I am not going
to take over the damn thing - David has violated the 11th
commandment (Thou Shalt Never Volunteer), so he gets to joy of
learning that codebase and taking care of it from now on.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06 20:46             ` Al Viro
@ 2018-05-06 20:49               ` John Paul Adrian Glaubitz
  2018-05-06 21:32               ` Al Viro
  1 sibling, 0 replies; 73+ messages in thread
From: John Paul Adrian Glaubitz @ 2018-05-06 20:49 UTC (permalink / raw)
  To: Al Viro
  Cc: Martin Steigerwald, Matthew Wilcox, dsterba, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

Hi Al!

On 05/06/2018 10:46 PM, Al Viro wrote:
> 	I'm fixing that pile of crap (along with the NFS exports
> one and, hopefully, rename mess as well).  HOWEVER, I am not going
> to take over the damn thing - David has violated the 11th
> commandment (Thou Shalt Never Volunteer), so he gets to joy of
> learning that codebase and taking care of it from now on.

Thanks a lot for your stab at this and fixing the worst issues.

Since we're using that code regularly in Debian, your help is highly
appreciated :-). I'm very glad this discussion has lead to the code
being improved!

Again, thank you!
Adrian

-- 
  .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06 20:46             ` Al Viro
  2018-05-06 20:49               ` John Paul Adrian Glaubitz
@ 2018-05-06 21:32               ` Al Viro
  2018-05-07  2:15                 ` Al Viro
  1 sibling, 1 reply; 73+ messages in thread
From: Al Viro @ 2018-05-06 21:32 UTC (permalink / raw)
  To: John Paul Adrian Glaubitz
  Cc: Martin Steigerwald, Matthew Wilcox, dsterba, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

On Sun, May 06, 2018 at 09:46:23PM +0100, Al Viro wrote:

> 	I'm fixing that pile of crap (along with the NFS exports
> one and, hopefully, rename mess as well).  HOWEVER, I am not going
> to take over the damn thing - David has violated the 11th
> commandment (Thou Shalt Never Volunteer), so he gets to joy of
> learning that codebase and taking care of it from now on.

	Same scenario on link(2) ends up with
* ST_LINKFILE created, inserted into the link chain and left around,
without being present in any hash chain
* target becoming positive hashed dentry, as if link(2) has succeeded,
so dcache lookups will be finding it (for a while)
* unlink(2) on source will have very interesting effects, what with
the attempts to move ST_FILE entry into the directory presumed to
contain ST_LINKFILE one, removing ST_LINKFILE from it at the same time.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06 21:32               ` Al Viro
@ 2018-05-07  2:15                 ` Al Viro
  2018-05-07  2:40                   ` Michael Schmitz
  0 siblings, 1 reply; 73+ messages in thread
From: Al Viro @ 2018-05-07  2:15 UTC (permalink / raw)
  To: John Paul Adrian Glaubitz
  Cc: Martin Steigerwald, Matthew Wilcox, dsterba, linux-fsdevel,
	linux-kernel, Jens Axboe, linux-m68k, Debian m68k

On Sun, May 06, 2018 at 10:32:47PM +0100, Al Viro wrote:
> On Sun, May 06, 2018 at 09:46:23PM +0100, Al Viro wrote:
> 
> > 	I'm fixing that pile of crap (along with the NFS exports
> > one and, hopefully, rename mess as well).  HOWEVER, I am not going
> > to take over the damn thing - David has violated the 11th
> > commandment (Thou Shalt Never Volunteer), so he gets to joy of
> > learning that codebase and taking care of it from now on.
> 
> 	Same scenario on link(2) ends up with
> * ST_LINKFILE created, inserted into the link chain and left around,
> without being present in any hash chain
> * target becoming positive hashed dentry, as if link(2) has succeeded,
> so dcache lookups will be finding it (for a while)
> * unlink(2) on source will have very interesting effects, what with
> the attempts to move ST_FILE entry into the directory presumed to
> contain ST_LINKFILE one, removing ST_LINKFILE from it at the same time.

Oh, lovely...  Looks like that thing wants the hash chains sorted by
block number.  affs_insert_hash() doesn't give a toss - it always
adds to the very end of chain.

Maintaining that requirement (and I can understand the rationale -
they don't want too many back-and-forth seeks on directory lookups)
is going to be great fun on rename(), especially for the case when
the target of rename happens to be a primary name for a file with
additional hardlinks.

I think I see how to deal with that sanely, but... ouch.

Incidentally, we'd better verify that hash chains are not looped - as it
is, there's no checks whatsoever, and it *will* happily loop if you
feed it an image with such braindamage.  I really hope that no fan of
desktop experience has set the things up for e.g. USB sticks with
that on them being recognized and automounted on insertion...

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-07  2:15                 ` Al Viro
@ 2018-05-07  2:40                   ` Michael Schmitz
  2018-05-07  7:08                     ` Martin Steigerwald
  0 siblings, 1 reply; 73+ messages in thread
From: Michael Schmitz @ 2018-05-07  2:40 UTC (permalink / raw)
  To: Al Viro
  Cc: John Paul Adrian Glaubitz, Martin Steigerwald, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Development,
	Jens Axboe, Linux/m68k, Debian m68k

Al,

I don't think there is USB sticks with affs on them as yet. There
isn't even USB host controller support for Amiga hardware (yet).

Last I tried USB on m68k (Atari, 060 accelerator) the desktop
experience was such that I'd rather not repeat that in a hurry (and
that was a simple FAT USB stick).

I see your point regarding the immense practical joke value on any
desktop PC ... my work desktop has the affs module present. Happy to
try this out if someone can provide a sample disk image suitable for
USB flash media.

Cheers,

  Michael


On Mon, May 7, 2018 at 2:15 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Sun, May 06, 2018 at 10:32:47PM +0100, Al Viro wrote:
>> On Sun, May 06, 2018 at 09:46:23PM +0100, Al Viro wrote:
>>
>> >     I'm fixing that pile of crap (along with the NFS exports
>> > one and, hopefully, rename mess as well).  HOWEVER, I am not going
>> > to take over the damn thing - David has violated the 11th
>> > commandment (Thou Shalt Never Volunteer), so he gets to joy of
>> > learning that codebase and taking care of it from now on.
>>
>>       Same scenario on link(2) ends up with
>> * ST_LINKFILE created, inserted into the link chain and left around,
>> without being present in any hash chain
>> * target becoming positive hashed dentry, as if link(2) has succeeded,
>> so dcache lookups will be finding it (for a while)
>> * unlink(2) on source will have very interesting effects, what with
>> the attempts to move ST_FILE entry into the directory presumed to
>> contain ST_LINKFILE one, removing ST_LINKFILE from it at the same time.
>
> Oh, lovely...  Looks like that thing wants the hash chains sorted by
> block number.  affs_insert_hash() doesn't give a toss - it always
> adds to the very end of chain.
>
> Maintaining that requirement (and I can understand the rationale -
> they don't want too many back-and-forth seeks on directory lookups)
> is going to be great fun on rename(), especially for the case when
> the target of rename happens to be a primary name for a file with
> additional hardlinks.
>
> I think I see how to deal with that sanely, but... ouch.
>
> Incidentally, we'd better verify that hash chains are not looped - as it
> is, there's no checks whatsoever, and it *will* happily loop if you
> feed it an image with such braindamage.  I really hope that no fan of
> desktop experience has set the things up for e.g. USB sticks with
> that on them being recognized and automounted on insertion...
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-06  8:52           ` John Paul Adrian Glaubitz
  2018-05-06 10:10             ` Martin Steigerwald
@ 2018-05-07  4:54             ` jdow
  1 sibling, 0 replies; 73+ messages in thread
From: jdow @ 2018-05-07  4:54 UTC (permalink / raw)
  To: John Paul Adrian Glaubitz, Geert Uytterhoeven, Martin Steigerwald
  Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k

On 20180506 01:52, John Paul Adrian Glaubitz wrote:
> On 04/27/2018 03:26 AM, jdow wrote:
>> And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times
>> changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely
>> goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS
>> as well as Linux OS.
> 
> I assume removing the feature for AmigaOS isn't really possible since we don't have
> the source code for that, do we?
> 
> Also, if I remember correctly, Mac partitions can store filesystem drivers as well
> and its actually a feature being used in MacOS. parted received a patch some time
> ago to fix the correct handling for storing the filesystem driver in the partition
> table.
> 
> I would be generally against removing these features as I don't think the security
> risk is relevant for the majority of users. The Amiga is a hobbyist machine these
> days and AmigaOS has certainly way more on than way to be compromised through
> vulnerabilities.
> 
> Adrian

You do not necessarily have the source for the device drivers. However the 
DriveInit code and the filesystem code get executed by the OS initialization 
code. The objection I have to the concept is that it's invisible to the user. 
The Linux filesystem code is either compiled into the kernel or is available in 
the libraries where it can be monitored at several levels from source code on 
up. Within AmigaDOS it can be monitored fairly easily by an AV tool - in theory. 
Alas, this is trying to lock the barn door after the barn has burned to the 
ground with a clever enough piece of malware. At least AmigaDOS AV tools should 
be expected to examine DriveInit and filesystem images on disk in the RDBs for 
malware modifications to those blocks. This is a burden Linux should not be 
forced to bear. So loading filesystems from RDBs instead of the more usual and 
accepted Linux practices should be disabled. And at least a portion of this 
discussion is Linux related. That's why I mentioned disabling the feature. While 
I cannot see much money in AmigaDOS related malware I can see it in Linux 
malware. And there's no real "glory" in launching malware on AmigaDOS. It's too 
easy a problem last I knew. "Whoopie, you have just proven you can ride a 
tricycle. Can't you do better?" (One could argue that AmigaDOS 1.0 was 
self-inflicted malware foisted on marvelous hardware for the era.)

{^_^}   Joanne Dow

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-07  2:40                   ` Michael Schmitz
@ 2018-05-07  7:08                     ` Martin Steigerwald
  2018-05-07 20:50                       ` Michael Schmitz
  0 siblings, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-05-07  7:08 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Al Viro, John Paul Adrian Glaubitz, Matthew Wilcox, David Sterba,
	Linux FS Devel, Linux Kernel Development, Jens Axboe, Linux/m68k,
	Debian m68k

Michael Schmitz - 07.05.18, 04:40:
> Al,
> 
> I don't think there is USB sticks with affs on them as yet. There
> isn't even USB host controller support for Amiga hardware (yet).
> 
> Last I tried USB on m68k (Atari, 060 accelerator) the desktop
> experience was such that I'd rather not repeat that in a hurry (and
> that was a simple FAT USB stick).

There is USB support available on Amiga since a long time.

On "Classic" Amigas AmigaOS 3.x with Poseidon USB stack + some USB card.

On AmigaOS 4.x built-in. AmigaOS 4.x hardware like Sam boards from Acube 
Systems have USB controllers that work out of the bux.

And I am pretty sure, you can also tell it to use Amiga Fast Filesystem 
(on Linux affs) on an USB stick. Also you can plug in an external 
harddisk with RDB partitions and whatever filesystems you wish.

Thanks,
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-07  7:08                     ` Martin Steigerwald
@ 2018-05-07 20:50                       ` Michael Schmitz
  2018-05-07 20:56                         ` Ingo Jürgensmann
  0 siblings, 1 reply; 73+ messages in thread
From: Michael Schmitz @ 2018-05-07 20:50 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: Al Viro, John Paul Adrian Glaubitz, Matthew Wilcox, David Sterba,
	Linux FS Devel, Linux Kernel Development, Jens Axboe, Linux/m68k,
	Debian m68k

Martin,

On Mon, May 7, 2018 at 7:08 PM, Martin Steigerwald <martin@lichtvoll.de> wrote:
> Michael Schmitz - 07.05.18, 04:40:
>> Al,
>>
>> I don't think there is USB sticks with affs on them as yet. There
>> isn't even USB host controller support for Amiga hardware (yet).
>>
>> Last I tried USB on m68k (Atari, 060 accelerator) the desktop
>> experience was such that I'd rather not repeat that in a hurry (and
>> that was a simple FAT USB stick).
>
> There is USB support available on Amiga since a long time.

Good to hear that. I stand corrected.

> On "Classic" Amigas AmigaOS 3.x with Poseidon USB stack + some USB card.

Haven't seen a Linux driver for that 'some USB card' yet.

> On AmigaOS 4.x built-in. AmigaOS 4.x hardware like Sam boards from Acube
> Systems have USB controllers that work out of the bux.

Forgot about the new (non-m68k) hardware. My focus is somewhat narrow,
on m68k and Linux.

> And I am pretty sure, you can also tell it to use Amiga Fast Filesystem
> (on Linux affs) on an USB stick. Also you can plug in an external
> harddisk with RDB partitions and whatever filesystems you wish.

I already conceded that's possible.

So our problem with the bug Al spotted, and AFFS on USB media are twtofold:

AmigaOS:
Exploitable: yes (unless the AmigaOS AFFS driver detects and mitigates this).
Likelihood: low (as Joanne said there are easier ways to do harm to
these systems)

Linux:
Exploitable: yes, except on hardware that doesn't have USB hardware support.
Likelihood: high

Can we blacklist affs from being autoloaded through udev on USB
storage media discovery?

Cheers,

  Michael

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-07 20:50                       ` Michael Schmitz
@ 2018-05-07 20:56                         ` Ingo Jürgensmann
  2018-05-07 20:58                           ` John Paul Adrian Glaubitz
  0 siblings, 1 reply; 73+ messages in thread
From: Ingo Jürgensmann @ 2018-05-07 20:56 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Martin Steigerwald, Al Viro, John Paul Adrian Glaubitz,
	Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Development, Jens Axboe, Linux/m68k, Debian m68k

Am 07.05.2018 um 22:50 schrieb Michael Schmitz <schmitzmic@gmail.com>:

>>> I don't think there is USB sticks with affs on them as yet. There
>>> isn't even USB host controller support for Amiga hardware (yet).
>>> 
>>> Last I tried USB on m68k (Atari, 060 accelerator) the desktop
>>> experience was such that I'd rather not repeat that in a hurry (and
>>> that was a simple FAT USB stick).
>> 
>> There is USB support available on Amiga since a long time.
> 
> Good to hear that. I stand corrected.
> 
>> On "Classic" Amigas AmigaOS 3.x with Poseidon USB stack + some USB card.
> 
> Haven't seen a Linux driver for that 'some USB card' yet.

There is for example RapidRoad (<http://wiki.icomp.de/wiki/RapidRoad>) which is an addon for the X-Surf 100 NIC (<http://wiki.icomp.de/wiki/X-Surf-100>). Adrian already has this addon module, I will buy one this year as well. 

-- 
Ciao...          //        http://blog.windfluechter.net
      Ingo     \X/     XMPP: ij@jabber.windfluechter.net
	
gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-05-07 20:56                         ` Ingo Jürgensmann
@ 2018-05-07 20:58                           ` John Paul Adrian Glaubitz
  0 siblings, 0 replies; 73+ messages in thread
From: John Paul Adrian Glaubitz @ 2018-05-07 20:58 UTC (permalink / raw)
  To: Ingo Jürgensmann, Michael Schmitz
  Cc: Martin Steigerwald, Al Viro, Matthew Wilcox, David Sterba,
	Linux FS Devel, Linux Kernel Development, Jens Axboe, Linux/m68k,
	Debian m68k

On 05/07/2018 10:56 PM, Ingo Jürgensmann wrote:
>> Haven't seen a Linux driver for that 'some USB card' yet.
> 
> There is for example RapidRoad (<http://wiki.icomp.de/wiki/RapidRoad>) which
> is an addon for the X-Surf 100 NIC (<http://wiki.icomp.de/wiki/X-Surf-100>).
> Adrian already has this addon module, I will buy one this year as well. 

Michael is the one who has started working on the driver, so I think
he knows :-).

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-04-27  2:11         ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
@ 2018-06-24  9:06           ` Martin Steigerwald
  2018-06-24 11:33             ` moving affs + RDB partition support to staging? jdow
                               ` (2 more replies)
  0 siblings, 3 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-24  9:06 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow

Hi.

Michael Schmitz - 27.04.18, 04:11:
> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> indicate the RDB parser bug is fixed by the patch given there, so if
> Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing 
the a patch from Joanne Dow�s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which 
I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless 
someone else does.

Thanks.
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-24  9:06           ` Martin Steigerwald
@ 2018-06-24 11:33             ` jdow
  2018-06-24 11:40             ` jdow
  2018-06-25  7:53             ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
  2 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-24 11:33 UTC (permalink / raw)
  To: Martin Steigerwald, Michael Schmitz
  Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, jdow

It's more likely the filesystem unless I didn't get the RDBs up to 3.1 status. I 
am pretty damn sure I did as I had an 18 gigabyte disk I was using at the time. 
And I made sure it would mount with Linux. Files greater than 2 gigs were sure 
to get messed up. But that was not RDBs. The disk had 9 gigs worth of AFS 
partitions and 9 gigs worth of SFS partitions. The big SFS partition was most of 
the 9 gigabytes. (The hdf file is 8,932,818,944 bytes.) There was a tiny system 
partition on it. It used some of Bill Hawes' goodies to overlay some useful paths.

I'm not sure the machine still boots. The disk in question no longer spins up.

{^_^}

On 20180624 02:06, Martin Steigerwald wrote:
> Hi.
> 
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
> 
> Ok, better be honest than having anyone waiting for it:
> 
> I do not care enough about this, in order to motivate myself preparing
> the a patch from Joanne Dow´s fix.
> 
> I am not even using my Amiga boxes anymore, not even the Sam440ep which
> I still have in my apartment.
> 
> So RDB support in Linux it remains broken for disks larger 2 TB, unless
> someone else does.
> 
> Thanks.
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-24  9:06           ` Martin Steigerwald
  2018-06-24 11:33             ` moving affs + RDB partition support to staging? jdow
@ 2018-06-24 11:40             ` jdow
  2018-06-26  2:23               ` Michael Schmitz
  2018-06-25  7:53             ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
  2 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-24 11:40 UTC (permalink / raw)
  To: Martin Steigerwald, Michael Schmitz
  Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn dool.

If memory serves the RDBs think in blocks rather than bytes so it should work up 
to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. 
But that wastes just a WHOLE LOT of disk in block maps. Go up to 4096 or 8192. 
The latter is 35 TB.

{^_^}
On 20180624 02:06, Martin Steigerwald wrote:
> Hi.
> 
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
> 
> Ok, better be honest than having anyone waiting for it:
> 
> I do not care enough about this, in order to motivate myself preparing
> the a patch from Joanne Dow´s fix.
> 
> I am not even using my Amiga boxes anymore, not even the Sam440ep which
> I still have in my apartment.
> 
> So RDB support in Linux it remains broken for disks larger 2 TB, unless
> someone else does.
> 
> Thanks.
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-06-24  9:06           ` Martin Steigerwald
  2018-06-24 11:33             ` moving affs + RDB partition support to staging? jdow
  2018-06-24 11:40             ` jdow
@ 2018-06-25  7:53             ` Michael Schmitz
  2018-06-25  8:26               ` Martin Steigerwald
  2018-06-25  8:40               ` Geert Uytterhoeven
  2 siblings, 2 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-25  7:53 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow,
	John Paul Adrian Glaubitz

Hi Martin,

OK, I'll prepare a patch and submit it to linux-block for review. I'll
have to refer to your testing back in 2012 since all I can test is
whether the patch still allows partition tables on small disks to be
recognized at this time (unless Adrian has a 2 TB disk and a SATA-SCSI
bridge to test this properly on).

Cheers,

    Michael



Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> Hi.
>
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
> Ok, better be honest than having anyone waiting for it:
>
> I do not care enough about this, in order to motivate myself preparing 
> the a patch from Joanne Dow´s fix.
>
> I am not even using my Amiga boxes anymore, not even the Sam440ep which 
> I still have in my apartment.
>
> So RDB support in Linux it remains broken for disks larger 2 TB, unless 
> someone else does.
>
> Thanks.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-06-25  7:53             ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
@ 2018-06-25  8:26               ` Martin Steigerwald
  2018-06-25  8:40               ` Geert Uytterhoeven
  1 sibling, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-25  8:26 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow,
	John Paul Adrian Glaubitz

Hi Michael.

Michael Schmitz - 25.06.18, 09:53:
> OK, I'll prepare a patch and submit it to linux-block for review. I'll
> have to refer to your testing back in 2012 since all I can test is
> whether the patch still allows partition tables on small disks to be
> recognized at this time (unless Adrian has a 2 TB disk and a
> SATA-SCSI bridge to test this properly on).

Thank you very much.

I believe the testing I did is still valid.

Feel free to include any parts of the description of the test I made 
back then into your patch description as you see it as relevant for it.

Also feel free to include my Tested-By: (probably with a hint the test 
was in 2012).

I am not sure I am ready to permanently let go of (some of) the hardware 
I still have collected, but at some time I may. I do have SATA-SCSI 
bridges. That A4000T I have here probably could be a nice Debian m68k 
build host.

Thanks,
Martin

> 
> Cheers,
> 
>     Michael
> 
> Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> > Hi.
> > 
> > Michael Schmitz - 27.04.18, 04:11:
> >> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >> indicate the RDB parser bug is fixed by the patch given there, so
> >> if
> >> Martin now submits the patch, all should be well?
> > 
> > Ok, better be honest than having anyone waiting for it:
> > 
> > I do not care enough about this, in order to motivate myself
> > preparing the a patch from Joanne Dow�s fix.
> > 
> > I am not even using my Amiga boxes anymore, not even the Sam440ep
> > which I still have in my apartment.
> > 
> > So RDB support in Linux it remains broken for disks larger 2 TB,
> > unless someone else does.
> > 
> > Thanks.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
  2018-06-25  7:53             ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
  2018-06-25  8:26               ` Martin Steigerwald
@ 2018-06-25  8:40               ` Geert Uytterhoeven
  1 sibling, 0 replies; 73+ messages in thread
From: Geert Uytterhoeven @ 2018-06-25  8:40 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Martin Steigerwald, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k, jdow,
	John Paul Adrian Glaubitz

Hi Michael,

On Mon, Jun 25, 2018 at 9:53 AM Michael Schmitz <schmitzmic@gmail.com> wrote:
> OK, I'll prepare a patch and submit it to linux-block for review. I'll

Thanks a lot!

> have to refer to your testing back in 2012 since all I can test is
> whether the patch still allows partition tables on small disks to be
> recognized at this time (unless Adrian has a 2 TB disk and a SATA-SCSI
> bridge to test this properly on).

Sparse file and loopback mounting with losetup --partscan?

> Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> > Michael Schmitz - 27.04.18, 04:11:
> >> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >> indicate the RDB parser bug is fixed by the patch given there, so if
> >> Martin now submits the patch, all should be well?
> > Ok, better be honest than having anyone waiting for it:
> >
> > I do not care enough about this, in order to motivate myself preparing
> > the a patch from Joanne Dow´s fix.
> >
> > I am not even using my Amiga boxes anymore, not even the Sam440ep which
> > I still have in my apartment.
> >
> > So RDB support in Linux it remains broken for disks larger 2 TB, unless
> > someone else does.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-24 11:40             ` jdow
@ 2018-06-26  2:23               ` Michael Schmitz
  2018-06-26  5:17                 ` jdow
  2018-06-26  8:02                 ` Martin Steigerwald
  0 siblings, 2 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-26  2:23 UTC (permalink / raw)
  To: jdow
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

  Michael


On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
> BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
> dool.
>
> If memory serves the RDBs think in blocks rather than bytes so it should
> work up to 2 gigablocks whatever your block size is. 512 blocks is
> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
> Go up to 4096 or 8192. The latter is 35 TB.
>
> {^_^}
> On 20180624 02:06, Martin Steigerwald wrote:
>>
>> Hi.
>>
>> Michael Schmitz - 27.04.18, 04:11:
>>>
>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>> Martin now submits the patch, all should be well?
>>
>>
>> Ok, better be honest than having anyone waiting for it:
>>
>> I do not care enough about this, in order to motivate myself preparing
>> the a patch from Joanne Dow´s fix.
>>
>> I am not even using my Amiga boxes anymore, not even the Sam440ep which
>> I still have in my apartment.
>>
>> So RDB support in Linux it remains broken for disks larger 2 TB, unless
>> someone else does.
>>
>> Thanks.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  2:23               ` Michael Schmitz
@ 2018-06-26  5:17                 ` jdow
  2018-06-26  8:12                   ` Martin Steigerwald
  2018-06-26  8:31                   ` Michael Schmitz
  2018-06-26  8:02                 ` Martin Steigerwald
  1 sibling, 2 replies; 73+ messages in thread
From: jdow @ 2018-06-26  5:17 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

As long as it preserves compatibility it should be OK, I suppose. Personally I'd 
make any partitioning tool front end gently force the block size towards 8k as 
the disk size gets larger. The file systems may also run into 2TB issues that 
are not obvious. An unused blocks list will have to go beyond a uint32_t size, 
for example. But a block list (OFS for sure, don't remember for the newer AFS) 
uses a tad under 1% of the disk all by itself. A block bitmap is not quite so 
bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I 
remember thinking about this for awhile and then determining I REALLY did not 
want to think about it as my brain was getting tied into a gordian knot.

{^_^}

On 20180625 19:23, Michael Schmitz wrote:
> Joanne,
> 
> Martin's boot log (including your patch) says:
> 
> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
> 4)
> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> Attached SCSI disk
> 
> so it's indeed a case of self inflicted damage (RDSK (512) means 512
> byte blocks) and can be worked around by using a different block size.
> 
> Your memory serves right indeed - blocksize is in 512 bytes units.
> I'll still submit a patch to Jens anyway as this may bite others yet.
> 
> Cheers,
> 
>    Michael
> 
> 
> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>> BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
>> dool.
>>
>> If memory serves the RDBs think in blocks rather than bytes so it should
>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
>> Go up to 4096 or 8192. The latter is 35 TB.
>>
>> {^_^}
>> On 20180624 02:06, Martin Steigerwald wrote:
>>>
>>> Hi.
>>>
>>> Michael Schmitz - 27.04.18, 04:11:
>>>>
>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>> Martin now submits the patch, all should be well?
>>>
>>>
>>> Ok, better be honest than having anyone waiting for it:
>>>
>>> I do not care enough about this, in order to motivate myself preparing
>>> the a patch from Joanne Dow´s fix.
>>>
>>> I am not even using my Amiga boxes anymore, not even the Sam440ep which
>>> I still have in my apartment.
>>>
>>> So RDB support in Linux it remains broken for disks larger 2 TB, unless
>>> someone else does.
>>>
>>> Thanks.
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  2:23               ` Michael Schmitz
  2018-06-26  5:17                 ` jdow
@ 2018-06-26  8:02                 ` Martin Steigerwald
  2018-06-26  8:40                   ` Michael Schmitz
  2018-06-26  9:31                   ` jdow
  1 sibling, 2 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-26  8:02 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
	Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
	linux-m68k

Michael.

Michael Schmitz - 26.06.18, 04:23:
> Joanne,
> 
> Martin's boot log (including your patch) says:
> 
> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
> 4)
> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> Attached SCSI disk
> 
> so it's indeed a case of self inflicted damage (RDSK (512) means 512
> byte blocks) and can be worked around by using a different block size.

Well I pretty much believe that this was the standard block size of the 
tool I used in order to create the RDB. I think it has  been Media 
Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has 
been deprecated by AmigaOS upstream.

Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would 
have set it up differently.

Anyway: It works like this in AmigaOS 4 without any issues. With 512 
Byte Blocks. I think it is good that Linux does not create data 
corruption when writing to disks that work just fine in AmigaOS. 
Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or 
Acube Sam machines may dual boot AmigaOS and Linux on their machines.

Thanks again for putting together a patch.

Thanks,
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  5:17                 ` jdow
@ 2018-06-26  8:12                   ` Martin Steigerwald
  2018-06-26  9:46                     ` jdow
  2018-06-26  8:31                   ` Michael Schmitz
  1 sibling, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-26  8:12 UTC (permalink / raw)
  To: jdow
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Joanne.

jdow - 26.06.18, 07:17:
> As long as it preserves compatibility it should be OK, I suppose.
> Personally I'd make any partitioning tool front end gently force the
> block size towards 8k as the disk size gets larger. The file systems
> may also run into 2TB issues that are not obvious. An unused blocks
> list will have to go beyond a uint32_t size, for example. But a block
> list (OFS for sure, don't remember for the newer AFS) uses a tad
> under 1% of the disk all by itself. A block bitmap is not quite so
> bad. {^_-}
> 
> Just be sure you are aware of all the ramifications when you make a
> change. I remember thinking about this for awhile and then
> determining I REALLY did not want to think about it as my brain was
> getting tied into a gordian knot.

Heh… :)

Well, all I can say it that it just worked on AmigaOS 4. I did not see 
any data corruption in any of the filesystems. Well as far as I have 
been able to check. There has been no repair tool for JXFS I think. But 
as I migrated the data on it to SFS, I was able to copy everything.

Famous last words.

Well especially the disk size was detected properly and there was no 
overflow like on Linux. So I´d say rather have on error less than one 
error more.

Of course, it could also be an option to outright *refuse* to detect 
such disks with a big fat warning in kernel log that it may unsafe. But 
overflowing and thus on writes overwriting existing data is not safe.

I do think it is safe enough, but what I do know about the internals 
about RDB (more than the average user certainly, but not as much as you 
or some other AmigaOS developer who digged  deeper into that).

So in case you´d rather see Linux to refuse to handle disks like that, 
that would also be fine with me. Just do not handle them in the broken 
way that they are handled in Linux now. I.e. do not deliberately corrupt 
things as in "Its dangerous, so let´s overwrite data straight away, so 
the user gets it." :)

Anyway, in my opinion RDB still just so much more advanced than MBR and 
in some parts even on par with the much later GPT. With some limitations 
it is a quite brilliant partition format, if you ask me.

> {^_^}
> 
> On 20180625 19:23, Michael Schmitz wrote:
> > Joanne,
> > 
> > Martin's boot log (including your patch) says:
> > 
> > Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
> > (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
> > spb
> > 4)
> > Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> > Attached SCSI disk
> > 
> > so it's indeed a case of self inflicted damage (RDSK (512) means 512
> > byte blocks) and can be worked around by using a different block
> > size.
> > 
> > Your memory serves right indeed - blocksize is in 512 bytes units.
> > I'll still submit a patch to Jens anyway as this may bite others
> > yet.
> > 
> > Cheers,
> > 
> >    Michael
> > 
> > On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
> >> BTW - anybody who uses 512 byte blocks with an Amiga file system is
> >> a famn dool.
> >> 
> >> If memory serves the RDBs think in blocks rather than bytes so it
> >> should work up to 2 gigablocks whatever your block size is. 512
> >> blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >> disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.
> >> 
> >> {^_^}
> >> 
> >> On 20180624 02:06, Martin Steigerwald wrote:
> >>> Hi.
> >>> 
> >>> Michael Schmitz - 27.04.18, 04:11:
> >>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>> indicate the RDB parser bug is fixed by the patch given there, so
> >>>> if
> >>>> Martin now submits the patch, all should be well?
> >>> 
> >>> Ok, better be honest than having anyone waiting for it:
> >>> 
> >>> I do not care enough about this, in order to motivate myself
> >>> preparing the a patch from Joanne Dow´s fix.
> >>> 
> >>> I am not even using my Amiga boxes anymore, not even the Sam440ep
> >>> which I still have in my apartment.
> >>> 
> >>> So RDB support in Linux it remains broken for disks larger 2 TB,
> >>> unless someone else does.
> >>> 
> >>> Thanks.
[…]
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  5:17                 ` jdow
  2018-06-26  8:12                   ` Martin Steigerwald
@ 2018-06-26  8:31                   ` Michael Schmitz
  2018-06-26  9:45                     ` jdow
  1 sibling, 1 reply; 73+ messages in thread
From: Michael Schmitz @ 2018-06-26  8:31 UTC (permalink / raw)
  To: jdow
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing the
issues Martin reported. Your patch addresses that by using the correct
data type for the calculations (as do other partition parsers that may
have to deal with large disks) and fixes Martin's bug, so appears to be
the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations
don't currently overflow is not expected to cause new issues, other than
enabling use of disk and partitions larger than 2 TB (which may have
ramifications with filesystems on these partitions). So comptibility is
preserved. 

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size stored
in the RDB would enforce use of the same block size in filesystems.
We'll have to rely on the filesystem tools to get that right, too. Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should
allow partitions larger than 2 TB to work already (but I suspect Al Viro
may have found a few issues when he looked at the AFFS code so I won't
say more). Anyway partitioning tools and filesystems are unrelated to
the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on
partitions larger than 2 TB, say so and I'll have the kernel print a
warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a
RDB image from a disk known to currently work under Linux (thanks Geert
for the losetup hint). Can't do much more without procuring a working
Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
for tests is a long way away from my home indeed.

Cheers,

    Michael

Am 26.06.18 um 17:17 schrieb jdow:
> As long as it preserves compatibility it should be OK, I suppose.
> Personally I'd make any partitioning tool front end gently force the
> block size towards 8k as the disk size gets larger. The file systems
> may also run into 2TB issues that are not obvious. An unused blocks
> list will have to go beyond a uint32_t size, for example. But a block
> list (OFS for sure, don't remember for the newer AFS) uses a tad under
> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>
> Just be sure you are aware of all the ramifications when you make a
> change. I remember thinking about this for awhile and then determining
> I REALLY did not want to think about it as my brain was getting tied
> into a gordian knot.
>
> {^_^}
>
> On 20180625 19:23, Michael Schmitz wrote:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
>>
>> Your memory serves right indeed - blocksize is in 512 bytes units.
>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>
>> Cheers,
>>
>>    Michael
>>
>>
>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>> a famn
>>> dool.
>>>
>>> If memory serves the RDBs think in blocks rather than bytes so it
>>> should
>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>> block maps.
>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>
>>> {^_^}
>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>
>>>> Hi.
>>>>
>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>
>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>> Martin now submits the patch, all should be well?
>>>>
>>>>
>>>> Ok, better be honest than having anyone waiting for it:
>>>>
>>>> I do not care enough about this, in order to motivate myself preparing
>>>> the a patch from Joanne Dow´s fix.
>>>>
>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>> which
>>>> I still have in my apartment.
>>>>
>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>> unless
>>>> someone else does.
>>>>
>>>> Thanks.
>>>>
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-m68k" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  8:02                 ` Martin Steigerwald
@ 2018-06-26  8:40                   ` Michael Schmitz
  2018-06-26  9:31                   ` jdow
  1 sibling, 0 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-26  8:40 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
	Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
	linux-m68k

Hi Martin,


Am 26.06.18 um 20:02 schrieb Martin Steigerwald:
> Michael.
>
> Michael Schmitz - 26.06.18, 04:23:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
> Well I pretty much believe that this was the standard block size of the 
> tool I used in order to create the RDB. I think it has  been Media 

No offense meant - 512 bytes per block would certainly have been the
default and it certainly wasn't obvious that this needed changing.

> Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
> Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has 
> been deprecated by AmigaOS upstream.
>
> Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would 
> have set it up differently.
>
> Anyway: It works like this in AmigaOS 4 without any issues. With 512 
> Byte Blocks. I think it is good that Linux does not create data 
> corruption when writing to disks that work just fine in AmigaOS. 
> Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or 
> Acube Sam machines may dual boot AmigaOS and Linux on their machines.

That's the goal.

> Thanks again for putting together a patch.

The hard work had been done by Joanne and you six years ago, so no
matter at all. If Jens and the crowd at linux-block point out errors in
review, I take that as a learning experience...

Cheers,

     Michael


>
> Thanks,

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  8:02                 ` Martin Steigerwald
  2018-06-26  8:40                   ` Michael Schmitz
@ 2018-06-26  9:31                   ` jdow
  1 sibling, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-26  9:31 UTC (permalink / raw)
  To: Martin Steigerwald, Michael Schmitz
  Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k

HDToolBox is not mine. That is where the intelligence is needed.

{^_^}

On 20180626 01:02, Martin Steigerwald wrote:
> Michael.
> 
> Michael Schmitz - 26.06.18, 04:23:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
> 
> Well I pretty much believe that this was the standard block size of the
> tool I used in order to create the RDB. I think it has  been Media
> Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
> Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has
> been deprecated by AmigaOS upstream.
> 
> Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would
> have set it up differently.
> 
> Anyway: It works like this in AmigaOS 4 without any issues. With 512
> Byte Blocks. I think it is good that Linux does not create data
> corruption when writing to disks that work just fine in AmigaOS.
> Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or
> Acube Sam machines may dual boot AmigaOS and Linux on their machines.
> 
> Thanks again for putting together a patch.
> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  8:31                   ` Michael Schmitz
@ 2018-06-26  9:45                     ` jdow
  2018-06-27  1:07                       ` Michael Schmitz
  0 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-26  9:45 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

If it is not backwards compatible I for one would refuse to use it. And if it 
still mattered that much to me I'd also generate a reasonable alternative. 
Modifying RDBs nay not be even an approximation of a good idea. You'd discover 
that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it 
is for your personal use then you're entirely free to reject my advice and are 
probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for 
legacy disks only.

{^_^}

On 20180626 01:31, Michael Schmitz wrote:
> Joanne,
> 
> I think we all agree that doing 32 bit calculations on 512-byte block
> addresses that overflow on disks 2 TB and larger is a bug, causing the
> issues Martin reported. Your patch addresses that by using the correct
> data type for the calculations (as do other partition parsers that may
> have to deal with large disks) and fixes Martin's bug, so appears to be
> the right thing to do.
> 
> Using 64 bit data types for disks smaller than 2 TB where calculations
> don't currently overflow is not expected to cause new issues, other than
> enabling use of disk and partitions larger than 2 TB (which may have
> ramifications with filesystems on these partitions). So comptibility is
> preserved.
> 
> Forcing larger block sizes might be a good strategy to avoid overflow
> issues in filesystems as well, but I can't see how the block size stored
> in the RDB would enforce use of the same block size in filesystems.
> We'll have to rely on the filesystem tools to get that right, too. Linux
> AFFS does allow block sizes up to 4k (VFS limitation) so this should
> allow partitions larger than 2 TB to work already (but I suspect Al Viro
> may have found a few issues when he looked at the AFFS code so I won't
> say more). Anyway partitioning tools and filesystems are unrelated to
> the Linux partition parser code which is all we aim to fix in this patch.
> 
> If you feel strongly about unknown ramifications of any filesystems on
> partitions larger than 2 TB, say so and I'll have the kernel print a
> warning about these partitions.
> 
> I'll get this patch tested on Martin's test case image as well as on a
> RDB image from a disk known to currently work under Linux (thanks Geert
> for the losetup hint). Can't do much more without procuring a working
> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
> for tests is a long way away from my home indeed.
> 
> Cheers,
> 
>      Michael
> 
> Am 26.06.18 um 17:17 schrieb jdow:
>> As long as it preserves compatibility it should be OK, I suppose.
>> Personally I'd make any partitioning tool front end gently force the
>> block size towards 8k as the disk size gets larger. The file systems
>> may also run into 2TB issues that are not obvious. An unused blocks
>> list will have to go beyond a uint32_t size, for example. But a block
>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>
>> Just be sure you are aware of all the ramifications when you make a
>> change. I remember thinking about this for awhile and then determining
>> I REALLY did not want to think about it as my brain was getting tied
>> into a gordian knot.
>>
>> {^_^}
>>
>> On 20180625 19:23, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> Martin's boot log (including your patch) says:
>>>
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>> 4)
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>> Attached SCSI disk
>>>
>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>> byte blocks) and can be worked around by using a different block size.
>>>
>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>
>>> Cheers,
>>>
>>>     Michael
>>>
>>>
>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>> a famn
>>>> dool.
>>>>
>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>> should
>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>> block maps.
>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>
>>>> {^_^}
>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>
>>>>> Hi.
>>>>>
>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>
>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>> Martin now submits the patch, all should be well?
>>>>>
>>>>>
>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>
>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>> the a patch from Joanne Dow´s fix.
>>>>>
>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>> which
>>>>> I still have in my apartment.
>>>>>
>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>> unless
>>>>> someone else does.
>>>>>
>>>>> Thanks.
>>>>>
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-m68k" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  8:12                   ` Martin Steigerwald
@ 2018-06-26  9:46                     ` jdow
  0 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-26  9:46 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

(Get everybody)
Speak to the high level driver folks about that. The low level stuff is 
basically dumb. It tells you what it finds. It tells the rest of the OS (the 
file systems) what it found. The amiga-makefs (or whatever it is called) needs 
to grow some added intelligence just as HDToolBox needs to grow more 
intelligence. The step from RDB uint32_t to RDB uint64_t probably needs a great 
deal of thought. Personally I'd be tempted to adopt GUID based partitioning 
system. It has already considered this stuff.

{^_^}

On 20180626 01:12, Martin Steigerwald wrote:
> Joanne.
> 
> jdow - 26.06.18, 07:17:
>> As long as it preserves compatibility it should be OK, I suppose.
>> Personally I'd make any partitioning tool front end gently force the
>> block size towards 8k as the disk size gets larger. The file systems
>> may also run into 2TB issues that are not obvious. An unused blocks
>> list will have to go beyond a uint32_t size, for example. But a block
>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>> under 1% of the disk all by itself. A block bitmap is not quite so
>> bad. {^_-}
>>
>> Just be sure you are aware of all the ramifications when you make a
>> change. I remember thinking about this for awhile and then
>> determining I REALLY did not want to think about it as my brain was
>> getting tied into a gordian knot.
> 
> Heh… :)
> 
> Well, all I can say it that it just worked on AmigaOS 4. I did not see
> any data corruption in any of the filesystems. Well as far as I have
> been able to check. There has been no repair tool for JXFS I think. But
> as I migrated the data on it to SFS, I was able to copy everything.
> 
> Famous last words.
> 
> Well especially the disk size was detected properly and there was no
> overflow like on Linux. So I´d say rather have on error less than one
> error more.
> 
> Of course, it could also be an option to outright *refuse* to detect
> such disks with a big fat warning in kernel log that it may unsafe. But
> overflowing and thus on writes overwriting existing data is not safe.
> 
> I do think it is safe enough, but what I do know about the internals
> about RDB (more than the average user certainly, but not as much as you
> or some other AmigaOS developer who digged  deeper into that).
> 
> So in case you´d rather see Linux to refuse to handle disks like that,
> that would also be fine with me. Just do not handle them in the broken
> way that they are handled in Linux now. I.e. do not deliberately corrupt
> things as in "Its dangerous, so let´s overwrite data straight away, so
> the user gets it." :)
> 
> Anyway, in my opinion RDB still just so much more advanced than MBR and
> in some parts even on par with the much later GPT. With some limitations
> it is a quite brilliant partition format, if you ask me.
> 
>> {^_^}
>>
>> On 20180625 19:23, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> Martin's boot log (including your patch) says:
>>>
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
>>> spb
>>> 4)
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>> Attached SCSI disk
>>>
>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>> byte blocks) and can be worked around by using a different block
>>> size.
>>>
>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>> I'll still submit a patch to Jens anyway as this may bite others
>>> yet.
>>>
>>> Cheers,
>>>
>>>     Michael
>>>
>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>> a famn dool.
>>>>
>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>> should work up to 2 gigablocks whatever your block size is. 512
>>>> blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of
>>>> disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>> Hi.
>>>>>
>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>> indicate the RDB parser bug is fixed by the patch given there, so
>>>>>> if
>>>>>> Martin now submits the patch, all should be well?
>>>>>
>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>
>>>>> I do not care enough about this, in order to motivate myself
>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>
>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>> which I still have in my apartment.
>>>>>
>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>> unless someone else does.
>>>>>
>>>>> Thanks.
> […]
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-26  9:45                     ` jdow
@ 2018-06-27  1:07                       ` Michael Schmitz
  2018-06-27  6:24                         ` jdow
  2018-06-27  7:57                         ` moving affs + RDB partition support to staging? Martin Steigerwald
  0 siblings, 2 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-27  1:07 UTC (permalink / raw)
  To: jdow
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Joanne,

As far as I have been able to test, the change is backwards compatible
(RDB partitions from an old disk 80 GB disk are still recognized OK).
That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have
failed the current Linux RDB parser on legacy 32 bit systems is well
taken though. Maybe Martin can clarify that for me - was the 2 TB disk
in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get
away with printing a big fat warning during boot, advising the user
that the oversize RDB partition(s) scanned are not compatible with
legacy 32 bit AmigaOS. With the proposed fix they will work under both
AmigaOS 4.1 and Linux instead of truncating the first overflowing
partition at disk end and trashing valid partitions that overlap,
which is what Martin was after.

If that still seems too risky, we can make the default behaviour to
bail out once a potential overflow is detected, and allow the user to
override that through a boot parameter. I'd leave that decision up for
the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition
start and size calculations, so a change of the RDB blocksize to
reduce the block counts stored in the RDB would still result in the
same overflow. And amiga-fdisk is indeed utterly broken and needs
updating (along with probably most legacy m68k partitioners). Adrian
has advertised parted as replacement for the old tools - maybe this
would make a nice test case for parted?

Cheers,

  Michael



On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> If it is not backwards compatible I for one would refuse to use it. And if
> it still mattered that much to me I'd also generate a reasonable
> alternative. Modifying RDBs nay not be even an approximation of a good idea.
> You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
> only system. If it is for your personal use then you're entirely free to
> reject my advice and are probably smart enough to keep it working for
> yourself.
>
> GPT is probably the right way to go. Preserve the ability to read RDBs for
> legacy disks only.
>
> {^_^}
>
>
> On 20180626 01:31, Michael Schmitz wrote:
>>
>> Joanne,
>>
>> I think we all agree that doing 32 bit calculations on 512-byte block
>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>> issues Martin reported. Your patch addresses that by using the correct
>> data type for the calculations (as do other partition parsers that may
>> have to deal with large disks) and fixes Martin's bug, so appears to be
>> the right thing to do.
>>
>> Using 64 bit data types for disks smaller than 2 TB where calculations
>> don't currently overflow is not expected to cause new issues, other than
>> enabling use of disk and partitions larger than 2 TB (which may have
>> ramifications with filesystems on these partitions). So comptibility is
>> preserved.
>>
>> Forcing larger block sizes might be a good strategy to avoid overflow
>> issues in filesystems as well, but I can't see how the block size stored
>> in the RDB would enforce use of the same block size in filesystems.
>> We'll have to rely on the filesystem tools to get that right, too. Linux
>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>> allow partitions larger than 2 TB to work already (but I suspect Al Viro
>> may have found a few issues when he looked at the AFFS code so I won't
>> say more). Anyway partitioning tools and filesystems are unrelated to
>> the Linux partition parser code which is all we aim to fix in this patch.
>>
>> If you feel strongly about unknown ramifications of any filesystems on
>> partitions larger than 2 TB, say so and I'll have the kernel print a
>> warning about these partitions.
>>
>> I'll get this patch tested on Martin's test case image as well as on a
>> RDB image from a disk known to currently work under Linux (thanks Geert
>> for the losetup hint). Can't do much more without procuring a working
>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
>> for tests is a long way away from my home indeed.
>>
>> Cheers,
>>
>>      Michael
>>
>> Am 26.06.18 um 17:17 schrieb jdow:
>>>
>>> As long as it preserves compatibility it should be OK, I suppose.
>>> Personally I'd make any partitioning tool front end gently force the
>>> block size towards 8k as the disk size gets larger. The file systems
>>> may also run into 2TB issues that are not obvious. An unused blocks
>>> list will have to go beyond a uint32_t size, for example. But a block
>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>>
>>> Just be sure you are aware of all the ramifications when you make a
>>> change. I remember thinking about this for awhile and then determining
>>> I REALLY did not want to think about it as my brain was getting tied
>>> into a gordian knot.
>>>
>>> {^_^}
>>>
>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>
>>>> Joanne,
>>>>
>>>> Martin's boot log (including your patch) says:
>>>>
>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>> 4)
>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>> Attached SCSI disk
>>>>
>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>> byte blocks) and can be worked around by using a different block size.
>>>>
>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>
>>>> Cheers,
>>>>
>>>>     Michael
>>>>
>>>>
>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>
>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>> a famn
>>>>> dool.
>>>>>
>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>> should
>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>> block maps.
>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>
>>>>> {^_^}
>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>
>>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>
>>>>>>>
>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>>> Martin now submits the patch, all should be well?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>
>>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>
>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>> which
>>>>>> I still have in my apartment.
>>>>>>
>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>> unless
>>>>>> someone else does.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-27  1:07                       ` Michael Schmitz
@ 2018-06-27  6:24                         ` jdow
  2018-06-27  8:03                           ` Martin Steigerwald
  2018-06-27  9:00                           ` moving affs + RDB partition support to staging? Michael Schmitz
  2018-06-27  7:57                         ` moving affs + RDB partition support to staging? Martin Steigerwald
  1 sibling, 2 replies; 73+ messages in thread
From: jdow @ 2018-06-27  6:24 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

You allergic to using a GPT solution? It will get away from some of the evils 
that RDB has inherent in it because they are also features? (Loading a 
filesystem or DriveInit code from RDBs is just asking for a nearly impossible to 
remove malware infection.) Furthermore, any 32 bit system that sees an RDSK 
block is going to try to translate it. If you add a new RDB format you are going 
to get bizarre and probably quite destructive results from the mistake. Fail 
safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system 
seems utterly outlandish to me. You cited another overflow potential. There are 
at least three we've identified, I believe. Are you 100% sure there are no more? 
The specific one you mention of translating RDB to Linux has a proper solution 
in the RDB reader. It should recover such overflow errors in the RDB as it can 
with due care and polish. It should flag any other overflow error it detects 
within the RDBs and return an error such as to leave the disk unmounted or 
mounted read-only if you feel like messing up a poor sod's backups. The simple 
solution is to read each of the variables with the nominal RDB size and convert 
it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the Amiga 
Elephant so to speak. I am not trying to assert any control. Do as you wish; 
but, I would plead with you to avoid ANY chance you can for the user to make a 
bonehead stupid move and lose all his treasured disk archives. Doing otherwise 
is very poor form.

{o.o}   Joanne "Said enough, she has" Dow

On 20180626 18:07, Michael Schmitz wrote:
> Joanne,
> 
> As far as I have been able to test, the change is backwards compatible
> (RDB partitions from an old disk 80 GB disk are still recognized OK).
> That"s only been done on an emulator though.
> 
> Your advice about the dangers of using RDB disks that would have
> failed the current Linux RDB parser on legacy 32 bit systems is well
> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
> in question ever used on a 32 bit Amiga system?
> 
> RDB disk format is meant for legacy use only, so I think we can get
> away with printing a big fat warning during boot, advising the user
> that the oversize RDB partition(s) scanned are not compatible with
> legacy 32 bit AmigaOS. With the proposed fix they will work under both
> AmigaOS 4.1 and Linux instead of truncating the first overflowing
> partition at disk end and trashing valid partitions that overlap,
> which is what Martin was after.
> 
> If that still seems too risky, we can make the default behaviour to
> bail out once a potential overflow is detected, and allow the user to
> override that through a boot parameter. I'd leave that decision up for
> the code review on linux-block.
> 
> Two more comments: Linux uses 512 byte block sizes for the partition
> start and size calculations, so a change of the RDB blocksize to
> reduce the block counts stored in the RDB would still result in the
> same overflow. And amiga-fdisk is indeed utterly broken and needs
> updating (along with probably most legacy m68k partitioners). Adrian
> has advertised parted as replacement for the old tools - maybe this
> would make a nice test case for parted?
> 
> Cheers,
> 
>    Michael
> 
> 
> 
> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>> If it is not backwards compatible I for one would refuse to use it. And if
>> it still mattered that much to me I'd also generate a reasonable
>> alternative. Modifying RDBs nay not be even an approximation of a good idea.
>> You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
>> only system. If it is for your personal use then you're entirely free to
>> reject my advice and are probably smart enough to keep it working for
>> yourself.
>>
>> GPT is probably the right way to go. Preserve the ability to read RDBs for
>> legacy disks only.
>>
>> {^_^}
>>
>>
>> On 20180626 01:31, Michael Schmitz wrote:
>>>
>>> Joanne,
>>>
>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>> issues Martin reported. Your patch addresses that by using the correct
>>> data type for the calculations (as do other partition parsers that may
>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>> the right thing to do.
>>>
>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>> don't currently overflow is not expected to cause new issues, other than
>>> enabling use of disk and partitions larger than 2 TB (which may have
>>> ramifications with filesystems on these partitions). So comptibility is
>>> preserved.
>>>
>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>> issues in filesystems as well, but I can't see how the block size stored
>>> in the RDB would enforce use of the same block size in filesystems.
>>> We'll have to rely on the filesystem tools to get that right, too. Linux
>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>> allow partitions larger than 2 TB to work already (but I suspect Al Viro
>>> may have found a few issues when he looked at the AFFS code so I won't
>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>> the Linux partition parser code which is all we aim to fix in this patch.
>>>
>>> If you feel strongly about unknown ramifications of any filesystems on
>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>> warning about these partitions.
>>>
>>> I'll get this patch tested on Martin's test case image as well as on a
>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>> for the losetup hint). Can't do much more without procuring a working
>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
>>> for tests is a long way away from my home indeed.
>>>
>>> Cheers,
>>>
>>>       Michael
>>>
>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>
>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>> Personally I'd make any partitioning tool front end gently force the
>>>> block size towards 8k as the disk size gets larger. The file systems
>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>>>
>>>> Just be sure you are aware of all the ramifications when you make a
>>>> change. I remember thinking about this for awhile and then determining
>>>> I REALLY did not want to think about it as my brain was getting tied
>>>> into a gordian knot.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>
>>>>> Joanne,
>>>>>
>>>>> Martin's boot log (including your patch) says:
>>>>>
>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>> 4)
>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>> Attached SCSI disk
>>>>>
>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>> byte blocks) and can be worked around by using a different block size.
>>>>>
>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>
>>>>> Cheers,
>>>>>
>>>>>      Michael
>>>>>
>>>>>
>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>
>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>> a famn
>>>>>> dool.
>>>>>>
>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>> should
>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>> block maps.
>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>
>>>>>> {^_^}
>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>
>>>>>>>>
>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>
>>>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>
>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>> which
>>>>>>> I still have in my apartment.
>>>>>>>
>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>> unless
>>>>>>> someone else does.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-27  1:07                       ` Michael Schmitz
  2018-06-27  6:24                         ` jdow
@ 2018-06-27  7:57                         ` Martin Steigerwald
  2018-06-28  2:56                           ` jdow
  1 sibling, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-27  7:57 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
	Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
	linux-m68k

Michael Schmitz - 27.06.18, 03:07:
> Joanne,
> 
> As far as I have been able to test, the change is backwards compatible
> (RDB partitions from an old disk 80 GB disk are still recognized OK).
> That"s only been done on an emulator though.
> 
> Your advice about the dangers of using RDB disks that would have
> failed the current Linux RDB parser on legacy 32 bit systems is well
> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
> in question ever used on a 32 bit Amiga system?

Sure! Are there actually *any* 64 bit Amiga systems? I am not completely 
sure architecture-wise, but the operating system is still just 32-Bit.

However, on AmigaOS officially since at least 3.5/3.9 there are various 
mechanisms to handle 64-bit block numbers for disk devices, called 
NSD64, TD64 and scsi direct¹. One of them, NSD 64 in AmigaOS 3.5/3.9, is 
part of the operating system. First via SetPatch based update to 
Kickstart ROM, but with AmigaOS 4.x its just included.

[1] http://www.amigawiki.de/doku.php?id=de:system:filesystems_limits 

(I do not agree with maximum filesystem size limits stated there for 
Amiga Fast Filesystem (FFS), but well, for early FFS versions they could 
have been true, I am pretty sure that at least with higher block sizes 
you can have FFS > 2 GiB in size. And I remember having had JFXS and I 
think also SFS2 partitions in AmigaOS 4 with more than 100 GiB.)

> RDB disk format is meant for legacy use only, so I think we can get
> away with printing a big fat warning during boot, advising the user
> that the oversize RDB partition(s) scanned are not compatible with
> legacy 32 bit AmigaOS. With the proposed fix they will work under both
> AmigaOS 4.1 and Linux instead of truncating the first overflowing
> partition at disk end and trashing valid partitions that overlap,
> which is what Martin was after.

They are compatible. For AmigaOS 4.x I am completely sure, cause I 
installed and used the disk there. That was the original motivation for 
my Linux bug report back then: I used the disk in AmigaOS 4 *just fine* 
and it broke in Linux. I used Media Toolbox in order to create the RDB I 
posted back then.

Its difficult to find proof for my claims online, but here is an 
official one:

http://wiki.amigaos.net/wiki/RDB

Limits of filesystem sizes may of course be different, but here we are 
talking about RDB.

Also this confirms that RDB only uses 512 byte block size on AmigaOS 4.

Hmmm, I even created the page back then. Interestingly at about the same 
time I made the bug report about the Linux issue. The calculations in 
there are not really from me. I bet I copied them over from what other 
AmigaOS developers wrote on development mailing lists, at that time. 
With their agreement. I think I started my research back then due by my 
own question on: Will such a large disk actually work in AmigaOS?

It is all a long time ago…

> If that still seems too risky, we can make the default behaviour to
> bail out once a potential overflow is detected, and allow the user to
> override that through a boot parameter. I'd leave that decision up for
> the code review on linux-block.
> 
> Two more comments: Linux uses 512 byte block sizes for the partition
> start and size calculations, so a change of the RDB blocksize to
> reduce the block counts stored in the RDB would still result in the
> same overflow. And amiga-fdisk is indeed utterly broken and needs
> updating (along with probably most legacy m68k partitioners). Adrian
> has advertised parted as replacement for the old tools - maybe this
> would make a nice test case for parted?

I always used AmigaOS based tools to partition in RDB format, cause I 
never trusted amiga-fdisk :).

> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> > If it is not backwards compatible I for one would refuse to use it.
> > And if it still mattered that much to me I'd also generate a
> > reasonable alternative. Modifying RDBs nay not be even an
> > approximation of a good idea. You'd discover that as soon as an RDB
> > uint64_t disk is tasted by a uint32_t only system. If it is for
> > your personal use then you're entirely free to reject my advice and
> > are probably smart enough to keep it working for yourself.
> > 
> > GPT is probably the right way to go. Preserve the ability to read
> > RDBs for legacy disks only.
> > 
> > {^_^}
> > 
> > On 20180626 01:31, Michael Schmitz wrote:
> >> Joanne,
> >> 
> >> I think we all agree that doing 32 bit calculations on 512-byte
> >> block
> >> addresses that overflow on disks 2 TB and larger is a bug, causing
> >> the issues Martin reported. Your patch addresses that by using the
> >> correct data type for the calculations (as do other partition
> >> parsers that may have to deal with large disks) and fixes Martin's
> >> bug, so appears to be the right thing to do.
> >> 
> >> Using 64 bit data types for disks smaller than 2 TB where
> >> calculations don't currently overflow is not expected to cause new
> >> issues, other than enabling use of disk and partitions larger than
> >> 2 TB (which may have ramifications with filesystems on these
> >> partitions). So comptibility is preserved.
> >> 
> >> Forcing larger block sizes might be a good strategy to avoid
> >> overflow
> >> issues in filesystems as well, but I can't see how the block size
> >> stored in the RDB would enforce use of the same block size in
> >> filesystems. We'll have to rely on the filesystem tools to get
> >> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >> limitation) so this should allow partitions larger than 2 TB to
> >> work already (but I suspect Al Viro may have found a few issues
> >> when he looked at the AFFS code so I won't say more). Anyway
> >> partitioning tools and filesystems are unrelated to the Linux
> >> partition parser code which is all we aim to fix in this patch.
> >> 
> >> If you feel strongly about unknown ramifications of any filesystems
> >> on partitions larger than 2 TB, say so and I'll have the kernel
> >> print a warning about these partitions.
> >> 
> >> I'll get this patch tested on Martin's test case image as well as
> >> on a RDB image from a disk known to currently work under Linux
> >> (thanks Geert for the losetup hint). Can't do much more without
> >> procuring a working Amiga disk image to use with an emulator,
> >> sorry. The Amiga I plan to use for tests is a long way away from
> >> my home indeed.
> >> 
> >> Cheers,
> >> 
> >>      Michael
> >> 
> >> Am 26.06.18 um 17:17 schrieb jdow:
> >>> As long as it preserves compatibility it should be OK, I suppose.
> >>> Personally I'd make any partitioning tool front end gently force
> >>> the
> >>> block size towards 8k as the disk size gets larger. The file
> >>> systems
> >>> may also run into 2TB issues that are not obvious. An unused
> >>> blocks
> >>> list will have to go beyond a uint32_t size, for example. But a
> >>> block
> >>> list (OFS for sure, don't remember for the newer AFS) uses a tad
> >>> under 1% of the disk all by itself. A block bitmap is not quite
> >>> so bad. {^_-}
> >>> 
> >>> Just be sure you are aware of all the ramifications when you make
> >>> a
> >>> change. I remember thinking about this for awhile and then
> >>> determining I REALLY did not want to think about it as my brain
> >>> was getting tied into a gordian knot.
> >>> 
> >>> {^_^}
> >>> 
> >>> On 20180625 19:23, Michael Schmitz wrote:
> >>>> Joanne,
> >>>> 
> >>>> Martin's boot log (including your patch) says:
> >>>> 
> >>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)
> >>>> sdb1
> >>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
> >>>> spb
> >>>> 4)
> >>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> >>>> Attached SCSI disk
> >>>> 
> >>>> so it's indeed a case of self inflicted damage (RDSK (512) means
> >>>> 512
> >>>> byte blocks) and can be worked around by using a different block
> >>>> size.
> >>>> 
> >>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>> units.
> >>>> I'll still submit a patch to Jens anyway as this may bite others
> >>>> yet.
> >>>> 
> >>>> Cheers,
> >>>> 
> >>>>     Michael
> >>>> 
> >>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> 
wrote:
> >>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system
> >>>>> is
> >>>>> a famn
> >>>>> dool.
> >>>>> 
> >>>>> If memory serves the RDBs think in blocks rather than bytes so
> >>>>> it
> >>>>> should
> >>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
> >>>>> is
> >>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
> >>>>> block maps.
> >>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>> 
> >>>>> {^_^}
> >>>>> 
> >>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>> Hi.
> >>>>>> 
> >>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>> test results at
> >>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>> indicate the RDB parser bug is fixed by the patch given there,
> >>>>>>> so if
> >>>>>>> Martin now submits the patch, all should be well?
> >>>>>> 
> >>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>> 
> >>>>>> I do not care enough about this, in order to motivate myself
> >>>>>> preparing the a patch from Joanne Dow´s fix.
> >>>>>> 
> >>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>> Sam440ep
> >>>>>> which
> >>>>>> I still have in my apartment.
> >>>>>> 
> >>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>> TB,
> >>>>>> unless
> >>>>>> someone else does.
> >>>>>> 
> >>>>>> Thanks.
> >>>>> 
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>> linux-m68k" in
> >>>>> the body of a message to majordomo@vger.kernel.org
> >>>>> More majordomo info at 
> >>>>> http://vger.kernel.org/majordomo-info.html
> >>> 
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe
> >>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> > linux-m68k" in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-27  6:24                         ` jdow
@ 2018-06-27  8:03                           ` Martin Steigerwald
  2018-06-28  2:57                             ` jdow
  2018-06-27  9:00                           ` moving affs + RDB partition support to staging? Michael Schmitz
  1 sibling, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-27  8:03 UTC (permalink / raw)
  To: jdow
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Dear Joanne.

jdow - 27.06.18, 08:24:
> You allergic to using a GPT solution? It will get away from some of
> the evils that RDB has inherent in it because they are also features?
> (Loading a filesystem or DriveInit code from RDBs is just asking for
> a nearly impossible to remove malware infection.) Furthermore, any 32
> bit system that sees an RDSK block is going to try to translate it.
> If you add a new RDB format you are going to get bizarre and probably
> quite destructive results from the mistake. Fail safe is a rather
> good notion, methinks.
> 
> Personally I figure this is all rather surreal. 2TG of junk on an
> Amiga system seems utterly outlandish to me. You cited another
> overflow potential. There are at least three we've identified, I
> believe. Are you 100% sure there are no more? The specific one you
> mention of translating RDB to Linux has a proper solution in the RDB
> reader. It should recover such overflow errors in the RDB as it can
> with due care and polish. It should flag any other overflow error it
> detects within the RDBs and return an error such as to leave the disk
> unmounted or mounted read-only if you feel like messing up a poor
> sod's backups. The simple solution is to read each of the variables
> with the nominal RDB size and convert it to uint64_t before
> calculating byte indices.
> 
> However, consider my inputs as advice from an adult who has seen the
> Amiga Elephant so to speak. I am not trying to assert any control. Do
> as you wish; but, I would plead with you to avoid ANY chance you can
> for the user to make a bonehead stupid move and lose all his
> treasured disk archives. Doing otherwise is very poor form.

I am pretty confident that larger than 2 TiB disks are fully supported 
within AmigaOS 4, as I outlined in my other mail.

So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in 
2012 already *just* fine. I even found I had the same questions back 
then, and researched it. Which lead to this official article back then:

http://wiki.amigaos.net/wiki/RDB

I am also pretty sure that AmigaOS still uses RDB as partitioning 
format. They support MBR. I don�t think AmigaOS 4.1 supports GPT. 
Whether to implement that of course is the decision of AmigaOS 4 
development team. I am no longer a member of it since some time.

Linux m68k should already be able to use disks in GPT format, but you 
likely won�t be able to read them in AmigaOS, unless there is some third 
party support for it meanwhile.

Thanks,
Martin

> 
> {o.o}   Joanne "Said enough, she has" Dow
> 
> On 20180626 18:07, Michael Schmitz wrote:
> > Joanne,
> > 
> > As far as I have been able to test, the change is backwards
> > compatible (RDB partitions from an old disk 80 GB disk are still
> > recognized OK). That"s only been done on an emulator though.
> > 
> > Your advice about the dangers of using RDB disks that would have
> > failed the current Linux RDB parser on legacy 32 bit systems is well
> > taken though. Maybe Martin can clarify that for me - was the 2 TB
> > disk in question ever used on a 32 bit Amiga system?
> > 
> > RDB disk format is meant for legacy use only, so I think we can get
> > away with printing a big fat warning during boot, advising the user
> > that the oversize RDB partition(s) scanned are not compatible with
> > legacy 32 bit AmigaOS. With the proposed fix they will work under
> > both AmigaOS 4.1 and Linux instead of truncating the first
> > overflowing partition at disk end and trashing valid partitions
> > that overlap, which is what Martin was after.
> > 
> > If that still seems too risky, we can make the default behaviour to
> > bail out once a potential overflow is detected, and allow the user
> > to
> > override that through a boot parameter. I'd leave that decision up
> > for the code review on linux-block.
> > 
> > Two more comments: Linux uses 512 byte block sizes for the partition
> > start and size calculations, so a change of the RDB blocksize to
> > reduce the block counts stored in the RDB would still result in the
> > same overflow. And amiga-fdisk is indeed utterly broken and needs
> > updating (along with probably most legacy m68k partitioners). Adrian
> > has advertised parted as replacement for the old tools - maybe this
> > would make a nice test case for parted?
> > 
> > Cheers,
> > 
> >    Michael
> > 
> > On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> >> If it is not backwards compatible I for one would refuse to use it.
> >> And if it still mattered that much to me I'd also generate a
> >> reasonable alternative. Modifying RDBs nay not be even an
> >> approximation of a good idea. You'd discover that as soon as an
> >> RDB uint64_t disk is tasted by a uint32_t only system. If it is
> >> for your personal use then you're entirely free to reject my
> >> advice and are probably smart enough to keep it working for
> >> yourself.
> >> 
> >> GPT is probably the right way to go. Preserve the ability to read
> >> RDBs for legacy disks only.
> >> 
> >> {^_^}
> >> 
> >> On 20180626 01:31, Michael Schmitz wrote:
> >>> Joanne,
> >>> 
> >>> I think we all agree that doing 32 bit calculations on 512-byte
> >>> block
> >>> addresses that overflow on disks 2 TB and larger is a bug, causing
> >>> the issues Martin reported. Your patch addresses that by using
> >>> the correct data type for the calculations (as do other partition
> >>> parsers that may have to deal with large disks) and fixes
> >>> Martin's bug, so appears to be the right thing to do.
> >>> 
> >>> Using 64 bit data types for disks smaller than 2 TB where
> >>> calculations don't currently overflow is not expected to cause
> >>> new issues, other than enabling use of disk and partitions larger
> >>> than 2 TB (which may have ramifications with filesystems on these
> >>> partitions). So comptibility is preserved.
> >>> 
> >>> Forcing larger block sizes might be a good strategy to avoid
> >>> overflow
> >>> issues in filesystems as well, but I can't see how the block size
> >>> stored in the RDB would enforce use of the same block size in
> >>> filesystems. We'll have to rely on the filesystem tools to get
> >>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >>> limitation) so this should allow partitions larger than 2 TB to
> >>> work already (but I suspect Al Viro may have found a few issues
> >>> when he looked at the AFFS code so I won't say more). Anyway
> >>> partitioning tools and filesystems are unrelated to the Linux
> >>> partition parser code which is all we aim to fix in this patch.
> >>> 
> >>> If you feel strongly about unknown ramifications of any
> >>> filesystems on partitions larger than 2 TB, say so and I'll have
> >>> the kernel print a warning about these partitions.
> >>> 
> >>> I'll get this patch tested on Martin's test case image as well as
> >>> on a RDB image from a disk known to currently work under Linux
> >>> (thanks Geert for the losetup hint). Can't do much more without
> >>> procuring a working Amiga disk image to use with an emulator,
> >>> sorry. The Amiga I plan to use for tests is a long way away from
> >>> my home indeed.
> >>> 
> >>> Cheers,
> >>> 
> >>>       Michael
> >>> 
> >>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>> As long as it preserves compatibility it should be OK, I suppose.
> >>>> Personally I'd make any partitioning tool front end gently force
> >>>> the
> >>>> block size towards 8k as the disk size gets larger. The file
> >>>> systems
> >>>> may also run into 2TB issues that are not obvious. An unused
> >>>> blocks
> >>>> list will have to go beyond a uint32_t size, for example. But a
> >>>> block
> >>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
> >>>> under 1% of the disk all by itself. A block bitmap is not quite
> >>>> so bad. {^_-}
> >>>> 
> >>>> Just be sure you are aware of all the ramifications when you make
> >>>> a
> >>>> change. I remember thinking about this for awhile and then
> >>>> determining I REALLY did not want to think about it as my brain
> >>>> was getting tied into a gordian knot.
> >>>> 
> >>>> {^_^}
> >>>> 
> >>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>> 
> >>>>> Martin's boot log (including your patch) says:
> >>>>> 
> >>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)
> >>>>> sdb1
> >>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
> >>>>> 2 spb
> >>>>> 4)
> >>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> >>>>> Attached SCSI disk
> >>>>> 
> >>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
> >>>>> 512
> >>>>> byte blocks) and can be worked around by using a different block
> >>>>> size.
> >>>>> 
> >>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>> units.
> >>>>> I'll still submit a patch to Jens anyway as this may bite others
> >>>>> yet.
> >>>>> 
> >>>>> Cheers,
> >>>>> 
> >>>>>      Michael
> >>>>> 
> >>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> 
wrote:
> >>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>> system is
> >>>>>> a famn
> >>>>>> dool.
> >>>>>> 
> >>>>>> If memory serves the RDBs think in blocks rather than bytes so
> >>>>>> it
> >>>>>> should
> >>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
> >>>>>> is
> >>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
> >>>>>> in
> >>>>>> block maps.
> >>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>> 
> >>>>>> {^_^}
> >>>>>> 
> >>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>> Hi.
> >>>>>>> 
> >>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>> test results at
> >>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>> there, so if
> >>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>> 
> >>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>> 
> >>>>>>> I do not care enough about this, in order to motivate myself
> >>>>>>> preparing the a patch from Joanne Dow�s fix.
> >>>>>>> 
> >>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>> Sam440ep
> >>>>>>> which
> >>>>>>> I still have in my apartment.
> >>>>>>> 
> >>>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>>> TB,
> >>>>>>> unless
> >>>>>>> someone else does.
> >>>>>>> 
> >>>>>>> Thanks.
> >>>>>> 
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>> linux-m68k" in
> >>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>> More majordomo info at 
> >>>>>> http://vger.kernel.org/majordomo-info.html
> >>>> 
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe
> >>>> linux-m68k" in the body of a message to
> >>>> majordomo@vger.kernel.org
> >>>> More majordomo info at 
> >>>> http://vger.kernel.org/majordomo-info.html
> >> 
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe
> >> linux-m68k" in the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-27  6:24                         ` jdow
  2018-06-27  8:03                           ` Martin Steigerwald
@ 2018-06-27  9:00                           ` Michael Schmitz
  2018-06-28  3:44                             ` jdow
  1 sibling, 1 reply; 73+ messages in thread
From: Michael Schmitz @ 2018-06-27  9:00 UTC (permalink / raw)
  To: jdow
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Joanne,

I'm not at all allergic to avoiding RDB at all cost for new disks. If 
AmigaOS 4.1 supports more recent partition formats, all the better. This 
is all about supporting use of legacy RDB disks on Linux (though 2 TB 
does stretch the definition of 'legacy' a little). My interest in this 
is to ensure we can continue to use RDB format disks on m68k Amiga 
computers which have no other way to boot Linux from disk.

Not proposing to change the RDB format at all, either. Just trying to 
make sure we translate RDB info into Linux 512-byte block offset and 
size numbers correctly. The kernel won't modify the RDB at all 
(intentionally, that is - with the correct choice of partition sizes, 
Martin might well have wiped out his RDB with the current version of the 
parser).

The choice of refusing to mount a disk (or mounting read-only) rests 
with the VFS drivers alone - AFFS in that case. Not touching any of 
that. At partition scan time, we only have the option of making the 
partition available (with a warning printed), or refusing to make it 
available to the kernel. Once it's made available, all bets are off.

 From what Martin writes, his test case RDB was valid and worked as 
expected on 32 bit AmigaOS (4.1). Apparently, that version has the 
necessary extensions to handle the large offsets resulting from 2 TB 
disks. Not sure what safeguards are in place when connecting such a disk 
to older versions of AmigaOS, but that is a different matter entirely.

The overflows in partition offset and size are the only ones I can see 
in the partition parser - there is no other overflow I've identified. I 
just stated that in order to place a partition towards the end of a 2 TB 
disk, the offset calculation will overflow regardless of what 
combination of rdb->rdb_BlockBytes and sector addresses stored in the 
RDB (in units of 512 byte blocks) we use:

         blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;


                 nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
                             be32_to_cpu(pb->pb_Environment[9])) *
                            be32_to_cpu(pb->pb_Environment[3]) *
                            be32_to_cpu(pb->pb_Environment[5]) *
                            blksize;
                 if (!nr_sects)
                         continue;
                 start_sect = be32_to_cpu(pb->pb_Environment[9]) *
                              be32_to_cpu(pb->pb_Environment[3]) *
                              be32_to_cpu(pb->pb_Environment[5]) *
                              blksize;

But in the interest of avoiding any accidental use of a RDB partition 
where calculations currently overflow, I'll make the default behaviour 
to bail out (instead of using wrong offset or size as we currently do). 
Given the 'eat_my_RDB_disk=1' boot option, the user may proceed at their 
own risk (though I still can't see what harm should result from now 
translating a well formed v4.1 2 TB disk RDB correctly for the first time).

Whether or not Linux correctly handles AFFS filesystems larger than 1 TB 
is a matter for VFS experts. Bailing out on nr_sects overflowing ought 
to prevent accidental use of AFFS filesystems on RDB disks which I 
suppose is the majority of use cases.

Bugs in partitioning tools on Linux are entirely out of scope - the 
partitioning tools bypass the partition structure discovered by the 
kernel, and work straight on the raw device. No protecting against that.

If you can point out a way to cause data loss with these precautions, 
for a disk 2 TB or larger that was partitioned and used on a recent 
version or AmigaOS supporting such large disks, I'd consider omitting 
the 'eat_my_RDB_disk' boot option, and just bail out as the only safe 
option.

Cheers,

	Michael


Am 27.06.2018 um 18:24 schrieb jdow:
> You allergic to using a GPT solution? It will get away from some of the
> evils that RDB has inherent in it because they are also features?
> (Loading a filesystem or DriveInit code from RDBs is just asking for a
> nearly impossible to remove malware infection.) Furthermore, any 32 bit
> system that sees an RDSK block is going to try to translate it. If you
> add a new RDB format you are going to get bizarre and probably quite
> destructive results from the mistake. Fail safe is a rather good notion,
> methinks.
>
> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
> system seems utterly outlandish to me. You cited another overflow
> potential. There are at least three we've identified, I believe. Are you
> 100% sure there are no more? The specific one you mention of translating
> RDB to Linux has a proper solution in the RDB reader. It should recover
> such overflow errors in the RDB as it can with due care and polish. It
> should flag any other overflow error it detects within the RDBs and
> return an error such as to leave the disk unmounted or mounted read-only
> if you feel like messing up a poor sod's backups. The simple solution is
> to read each of the variables with the nominal RDB size and convert it
> to uint64_t before calculating byte indices.
>
> However, consider my inputs as advice from an adult who has seen the
> Amiga Elephant so to speak. I am not trying to assert any control. Do as
> you wish; but, I would plead with you to avoid ANY chance you can for
> the user to make a bonehead stupid move and lose all his treasured disk
> archives. Doing otherwise is very poor form.
>
> {o.o}   Joanne "Said enough, she has" Dow
>
> On 20180626 18:07, Michael Schmitz wrote:
>> Joanne,
>>
>> As far as I have been able to test, the change is backwards compatible
>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>> That"s only been done on an emulator though.
>>
>> Your advice about the dangers of using RDB disks that would have
>> failed the current Linux RDB parser on legacy 32 bit systems is well
>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>> in question ever used on a 32 bit Amiga system?
>>
>> RDB disk format is meant for legacy use only, so I think we can get
>> away with printing a big fat warning during boot, advising the user
>> that the oversize RDB partition(s) scanned are not compatible with
>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>> partition at disk end and trashing valid partitions that overlap,
>> which is what Martin was after.
>>
>> If that still seems too risky, we can make the default behaviour to
>> bail out once a potential overflow is detected, and allow the user to
>> override that through a boot parameter. I'd leave that decision up for
>> the code review on linux-block.
>>
>> Two more comments: Linux uses 512 byte block sizes for the partition
>> start and size calculations, so a change of the RDB blocksize to
>> reduce the block counts stored in the RDB would still result in the
>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>> updating (along with probably most legacy m68k partitioners). Adrian
>> has advertised parted as replacement for the old tools - maybe this
>> would make a nice test case for parted?
>>
>> Cheers,
>>
>>    Michael
>>
>>
>>
>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>> If it is not backwards compatible I for one would refuse to use it.
>>> And if
>>> it still mattered that much to me I'd also generate a reasonable
>>> alternative. Modifying RDBs nay not be even an approximation of a
>>> good idea.
>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>> uint32_t
>>> only system. If it is for your personal use then you're entirely free to
>>> reject my advice and are probably smart enough to keep it working for
>>> yourself.
>>>
>>> GPT is probably the right way to go. Preserve the ability to read
>>> RDBs for
>>> legacy disks only.
>>>
>>> {^_^}
>>>
>>>
>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>
>>>> Joanne,
>>>>
>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>>> issues Martin reported. Your patch addresses that by using the correct
>>>> data type for the calculations (as do other partition parsers that may
>>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>>> the right thing to do.
>>>>
>>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>>> don't currently overflow is not expected to cause new issues, other
>>>> than
>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>> ramifications with filesystems on these partitions). So comptibility is
>>>> preserved.
>>>>
>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>> issues in filesystems as well, but I can't see how the block size
>>>> stored
>>>> in the RDB would enforce use of the same block size in filesystems.
>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>> Linux
>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>> Viro
>>>> may have found a few issues when he looked at the AFFS code so I won't
>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>> the Linux partition parser code which is all we aim to fix in this
>>>> patch.
>>>>
>>>> If you feel strongly about unknown ramifications of any filesystems on
>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>> warning about these partitions.
>>>>
>>>> I'll get this patch tested on Martin's test case image as well as on a
>>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>>> for the losetup hint). Can't do much more without procuring a working
>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>> use
>>>> for tests is a long way away from my home indeed.
>>>>
>>>> Cheers,
>>>>
>>>>       Michael
>>>>
>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>
>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>> {^_-}
>>>>>
>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>> change. I remember thinking about this for awhile and then determining
>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>> into a gordian knot.
>>>>>
>>>>> {^_^}
>>>>>
>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>
>>>>>> Joanne,
>>>>>>
>>>>>> Martin's boot log (including your patch) says:
>>>>>>
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>>> 4)
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>> Attached SCSI disk
>>>>>>
>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>>> byte blocks) and can be worked around by using a different block
>>>>>> size.
>>>>>>
>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>      Michael
>>>>>>
>>>>>>
>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>>
>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>>> a famn
>>>>>>> dool.
>>>>>>>
>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>> should
>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>> block maps.
>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>
>>>>>>> {^_^}
>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>> so if
>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>
>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>> preparing
>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>
>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>> which
>>>>>>>> I still have in my apartment.
>>>>>>>>
>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>> unless
>>>>>>>> someone else does.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-27  7:57                         ` moving affs + RDB partition support to staging? Martin Steigerwald
@ 2018-06-28  2:56                           ` jdow
  0 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-28  2:56 UTC (permalink / raw)
  To: Martin Steigerwald, Michael Schmitz
  Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
	Linux Kernel Mailing List, Jens Axboe, linux-m68k

FFS is limited to 2 GHz file size if you don't want any corruption using 
fseek(). Otherwise it can go to 4 GHz sort of safely.
{^_^}

On 20180627 00:57, Martin Steigerwald wrote:
> Michael Schmitz - 27.06.18, 03:07:
>> Joanne,
>>
>> As far as I have been able to test, the change is backwards compatible
>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>> That"s only been done on an emulator though.
>>
>> Your advice about the dangers of using RDB disks that would have
>> failed the current Linux RDB parser on legacy 32 bit systems is well
>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>> in question ever used on a 32 bit Amiga system?
> 
> Sure! Are there actually *any* 64 bit Amiga systems? I am not completely
> sure architecture-wise, but the operating system is still just 32-Bit.
> 
> However, on AmigaOS officially since at least 3.5/3.9 there are various
> mechanisms to handle 64-bit block numbers for disk devices, called
> NSD64, TD64 and scsi direct¹. One of them, NSD 64 in AmigaOS 3.5/3.9, is
> part of the operating system. First via SetPatch based update to
> Kickstart ROM, but with AmigaOS 4.x its just included.
> 
> [1] http://www.amigawiki.de/doku.php?id=de:system:filesystems_limits
> 
> (I do not agree with maximum filesystem size limits stated there for
> Amiga Fast Filesystem (FFS), but well, for early FFS versions they could
> have been true, I am pretty sure that at least with higher block sizes
> you can have FFS > 2 GiB in size. And I remember having had JFXS and I
> think also SFS2 partitions in AmigaOS 4 with more than 100 GiB.)
> 
>> RDB disk format is meant for legacy use only, so I think we can get
>> away with printing a big fat warning during boot, advising the user
>> that the oversize RDB partition(s) scanned are not compatible with
>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>> partition at disk end and trashing valid partitions that overlap,
>> which is what Martin was after.
> 
> They are compatible. For AmigaOS 4.x I am completely sure, cause I
> installed and used the disk there. That was the original motivation for
> my Linux bug report back then: I used the disk in AmigaOS 4 *just fine*
> and it broke in Linux. I used Media Toolbox in order to create the RDB I
> posted back then.
> 
> Its difficult to find proof for my claims online, but here is an
> official one:
> 
> http://wiki.amigaos.net/wiki/RDB
> 
> Limits of filesystem sizes may of course be different, but here we are
> talking about RDB.
> 
> Also this confirms that RDB only uses 512 byte block size on AmigaOS 4.
> 
> Hmmm, I even created the page back then. Interestingly at about the same
> time I made the bug report about the Linux issue. The calculations in
> there are not really from me. I bet I copied them over from what other
> AmigaOS developers wrote on development mailing lists, at that time.
> With their agreement. I think I started my research back then due by my
> own question on: Will such a large disk actually work in AmigaOS?
> 
> It is all a long time ago…
> 
>> If that still seems too risky, we can make the default behaviour to
>> bail out once a potential overflow is detected, and allow the user to
>> override that through a boot parameter. I'd leave that decision up for
>> the code review on linux-block.
>>
>> Two more comments: Linux uses 512 byte block sizes for the partition
>> start and size calculations, so a change of the RDB blocksize to
>> reduce the block counts stored in the RDB would still result in the
>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>> updating (along with probably most legacy m68k partitioners). Adrian
>> has advertised parted as replacement for the old tools - maybe this
>> would make a nice test case for parted?
> 
> I always used AmigaOS based tools to partition in RDB format, cause I
> never trusted amiga-fdisk :).
> 
>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>> If it is not backwards compatible I for one would refuse to use it.
>>> And if it still mattered that much to me I'd also generate a
>>> reasonable alternative. Modifying RDBs nay not be even an
>>> approximation of a good idea. You'd discover that as soon as an RDB
>>> uint64_t disk is tasted by a uint32_t only system. If it is for
>>> your personal use then you're entirely free to reject my advice and
>>> are probably smart enough to keep it working for yourself.
>>>
>>> GPT is probably the right way to go. Preserve the ability to read
>>> RDBs for legacy disks only.
>>>
>>> {^_^}
>>>
>>> On 20180626 01:31, Michael Schmitz wrote:
>>>> Joanne,
>>>>
>>>> I think we all agree that doing 32 bit calculations on 512-byte
>>>> block
>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>> the issues Martin reported. Your patch addresses that by using the
>>>> correct data type for the calculations (as do other partition
>>>> parsers that may have to deal with large disks) and fixes Martin's
>>>> bug, so appears to be the right thing to do.
>>>>
>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>> calculations don't currently overflow is not expected to cause new
>>>> issues, other than enabling use of disk and partitions larger than
>>>> 2 TB (which may have ramifications with filesystems on these
>>>> partitions). So comptibility is preserved.
>>>>
>>>> Forcing larger block sizes might be a good strategy to avoid
>>>> overflow
>>>> issues in filesystems as well, but I can't see how the block size
>>>> stored in the RDB would enforce use of the same block size in
>>>> filesystems. We'll have to rely on the filesystem tools to get
>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
>>>> limitation) so this should allow partitions larger than 2 TB to
>>>> work already (but I suspect Al Viro may have found a few issues
>>>> when he looked at the AFFS code so I won't say more). Anyway
>>>> partitioning tools and filesystems are unrelated to the Linux
>>>> partition parser code which is all we aim to fix in this patch.
>>>>
>>>> If you feel strongly about unknown ramifications of any filesystems
>>>> on partitions larger than 2 TB, say so and I'll have the kernel
>>>> print a warning about these partitions.
>>>>
>>>> I'll get this patch tested on Martin's test case image as well as
>>>> on a RDB image from a disk known to currently work under Linux
>>>> (thanks Geert for the losetup hint). Can't do much more without
>>>> procuring a working Amiga disk image to use with an emulator,
>>>> sorry. The Amiga I plan to use for tests is a long way away from
>>>> my home indeed.
>>>>
>>>> Cheers,
>>>>
>>>>       Michael
>>>>
>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>> Personally I'd make any partitioning tool front end gently force
>>>>> the
>>>>> block size towards 8k as the disk size gets larger. The file
>>>>> systems
>>>>> may also run into 2TB issues that are not obvious. An unused
>>>>> blocks
>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>> block
>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>> under 1% of the disk all by itself. A block bitmap is not quite
>>>>> so bad. {^_-}
>>>>>
>>>>> Just be sure you are aware of all the ramifications when you make
>>>>> a
>>>>> change. I remember thinking about this for awhile and then
>>>>> determining I REALLY did not want to think about it as my brain
>>>>> was getting tied into a gordian knot.
>>>>>
>>>>> {^_^}
>>>>>
>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>> Joanne,
>>>>>>
>>>>>> Martin's boot log (including your patch) says:
>>>>>>
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)
>>>>>> sdb1
>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
>>>>>> spb
>>>>>> 4)
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>> Attached SCSI disk
>>>>>>
>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>> 512
>>>>>> byte blocks) and can be worked around by using a different block
>>>>>> size.
>>>>>>
>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
>>>>>> units.
>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>> yet.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>      Michael
>>>>>>
>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
> wrote:
>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system
>>>>>>> is
>>>>>>> a famn
>>>>>>> dool.
>>>>>>>
>>>>>>> If memory serves the RDBs think in blocks rather than bytes so
>>>>>>> it
>>>>>>> should
>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
>>>>>>> is
>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>> block maps.
>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>
>>>>>>> {^_^}
>>>>>>>
>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>> test results at
>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>> so if
>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>
>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>
>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>>>>
>>>>>>>> I am not even using my Amiga boxes anymore, not even the
>>>>>>>> Sam440ep
>>>>>>>> which
>>>>>>>> I still have in my apartment.
>>>>>>>>
>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
>>>>>>>> TB,
>>>>>>>> unless
>>>>>>>> someone else does.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at
>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-27  8:03                           ` Martin Steigerwald
@ 2018-06-28  2:57                             ` jdow
  2018-06-28  7:40                               ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
  0 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-28  2:57 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

The issue is what happens when one of those disks appears on a 3.1 system.
{^_^}

On 20180627 01:03, Martin Steigerwald wrote:
> Dear Joanne.
> 
> jdow - 27.06.18, 08:24:
>> You allergic to using a GPT solution? It will get away from some of
>> the evils that RDB has inherent in it because they are also features?
>> (Loading a filesystem or DriveInit code from RDBs is just asking for
>> a nearly impossible to remove malware infection.) Furthermore, any 32
>> bit system that sees an RDSK block is going to try to translate it.
>> If you add a new RDB format you are going to get bizarre and probably
>> quite destructive results from the mistake. Fail safe is a rather
>> good notion, methinks.
>>
>> Personally I figure this is all rather surreal. 2TG of junk on an
>> Amiga system seems utterly outlandish to me. You cited another
>> overflow potential. There are at least three we've identified, I
>> believe. Are you 100% sure there are no more? The specific one you
>> mention of translating RDB to Linux has a proper solution in the RDB
>> reader. It should recover such overflow errors in the RDB as it can
>> with due care and polish. It should flag any other overflow error it
>> detects within the RDBs and return an error such as to leave the disk
>> unmounted or mounted read-only if you feel like messing up a poor
>> sod's backups. The simple solution is to read each of the variables
>> with the nominal RDB size and convert it to uint64_t before
>> calculating byte indices.
>>
>> However, consider my inputs as advice from an adult who has seen the
>> Amiga Elephant so to speak. I am not trying to assert any control. Do
>> as you wish; but, I would plead with you to avoid ANY chance you can
>> for the user to make a bonehead stupid move and lose all his
>> treasured disk archives. Doing otherwise is very poor form.
> 
> I am pretty confident that larger than 2 TiB disks are fully supported
> within AmigaOS 4, as I outlined in my other mail.
> 
> So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in
> 2012 already *just* fine. I even found I had the same questions back
> then, and researched it. Which lead to this official article back then:
> 
> http://wiki.amigaos.net/wiki/RDB
> 
> I am also pretty sure that AmigaOS still uses RDB as partitioning
> format. They support MBR. I don´t think AmigaOS 4.1 supports GPT.
> Whether to implement that of course is the decision of AmigaOS 4
> development team. I am no longer a member of it since some time.
> 
> Linux m68k should already be able to use disks in GPT format, but you
> likely won´t be able to read them in AmigaOS, unless there is some third
> party support for it meanwhile.
> 
> Thanks,
> Martin
> 
>>
>> {o.o}   Joanne "Said enough, she has" Dow
>>
>> On 20180626 18:07, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> As far as I have been able to test, the change is backwards
>>> compatible (RDB partitions from an old disk 80 GB disk are still
>>> recognized OK). That"s only been done on an emulator though.
>>>
>>> Your advice about the dangers of using RDB disks that would have
>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>> taken though. Maybe Martin can clarify that for me - was the 2 TB
>>> disk in question ever used on a 32 bit Amiga system?
>>>
>>> RDB disk format is meant for legacy use only, so I think we can get
>>> away with printing a big fat warning during boot, advising the user
>>> that the oversize RDB partition(s) scanned are not compatible with
>>> legacy 32 bit AmigaOS. With the proposed fix they will work under
>>> both AmigaOS 4.1 and Linux instead of truncating the first
>>> overflowing partition at disk end and trashing valid partitions
>>> that overlap, which is what Martin was after.
>>>
>>> If that still seems too risky, we can make the default behaviour to
>>> bail out once a potential overflow is detected, and allow the user
>>> to
>>> override that through a boot parameter. I'd leave that decision up
>>> for the code review on linux-block.
>>>
>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>> start and size calculations, so a change of the RDB blocksize to
>>> reduce the block counts stored in the RDB would still result in the
>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>> updating (along with probably most legacy m68k partitioners). Adrian
>>> has advertised parted as replacement for the old tools - maybe this
>>> would make a nice test case for parted?
>>>
>>> Cheers,
>>>
>>>     Michael
>>>
>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>>> If it is not backwards compatible I for one would refuse to use it.
>>>> And if it still mattered that much to me I'd also generate a
>>>> reasonable alternative. Modifying RDBs nay not be even an
>>>> approximation of a good idea. You'd discover that as soon as an
>>>> RDB uint64_t disk is tasted by a uint32_t only system. If it is
>>>> for your personal use then you're entirely free to reject my
>>>> advice and are probably smart enough to keep it working for
>>>> yourself.
>>>>
>>>> GPT is probably the right way to go. Preserve the ability to read
>>>> RDBs for legacy disks only.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>> Joanne,
>>>>>
>>>>> I think we all agree that doing 32 bit calculations on 512-byte
>>>>> block
>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>> the issues Martin reported. Your patch addresses that by using
>>>>> the correct data type for the calculations (as do other partition
>>>>> parsers that may have to deal with large disks) and fixes
>>>>> Martin's bug, so appears to be the right thing to do.
>>>>>
>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>> calculations don't currently overflow is not expected to cause
>>>>> new issues, other than enabling use of disk and partitions larger
>>>>> than 2 TB (which may have ramifications with filesystems on these
>>>>> partitions). So comptibility is preserved.
>>>>>
>>>>> Forcing larger block sizes might be a good strategy to avoid
>>>>> overflow
>>>>> issues in filesystems as well, but I can't see how the block size
>>>>> stored in the RDB would enforce use of the same block size in
>>>>> filesystems. We'll have to rely on the filesystem tools to get
>>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
>>>>> limitation) so this should allow partitions larger than 2 TB to
>>>>> work already (but I suspect Al Viro may have found a few issues
>>>>> when he looked at the AFFS code so I won't say more). Anyway
>>>>> partitioning tools and filesystems are unrelated to the Linux
>>>>> partition parser code which is all we aim to fix in this patch.
>>>>>
>>>>> If you feel strongly about unknown ramifications of any
>>>>> filesystems on partitions larger than 2 TB, say so and I'll have
>>>>> the kernel print a warning about these partitions.
>>>>>
>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>> on a RDB image from a disk known to currently work under Linux
>>>>> (thanks Geert for the losetup hint). Can't do much more without
>>>>> procuring a working Amiga disk image to use with an emulator,
>>>>> sorry. The Amiga I plan to use for tests is a long way away from
>>>>> my home indeed.
>>>>>
>>>>> Cheers,
>>>>>
>>>>>        Michael
>>>>>
>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>> Personally I'd make any partitioning tool front end gently force
>>>>>> the
>>>>>> block size towards 8k as the disk size gets larger. The file
>>>>>> systems
>>>>>> may also run into 2TB issues that are not obvious. An unused
>>>>>> blocks
>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>> block
>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>> under 1% of the disk all by itself. A block bitmap is not quite
>>>>>> so bad. {^_-}
>>>>>>
>>>>>> Just be sure you are aware of all the ramifications when you make
>>>>>> a
>>>>>> change. I remember thinking about this for awhile and then
>>>>>> determining I REALLY did not want to think about it as my brain
>>>>>> was getting tied into a gordian knot.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>> Joanne,
>>>>>>>
>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)
>>>>>>> sdb1
>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>> 2 spb
>>>>>>> 4)
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>> Attached SCSI disk
>>>>>>>
>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>> 512
>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>> size.
>>>>>>>
>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
>>>>>>> units.
>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>> yet.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>>       Michael
>>>>>>>
>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
> wrote:
>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>> system is
>>>>>>>> a famn
>>>>>>>> dool.
>>>>>>>>
>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so
>>>>>>>> it
>>>>>>>> should
>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
>>>>>>>> is
>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
>>>>>>>> in
>>>>>>>> block maps.
>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>>
>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>> test results at
>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
>>>>>>>>>> there, so if
>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>
>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>
>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>>>>>
>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
>>>>>>>>> Sam440ep
>>>>>>>>> which
>>>>>>>>> I still have in my apartment.
>>>>>>>>>
>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
>>>>>>>>> TB,
>>>>>>>>> unless
>>>>>>>>> someone else does.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at
>>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in the body of a message to
>>>>>> majordomo@vger.kernel.org
>>>>>> More majordomo info at
>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-27  9:00                           ` moving affs + RDB partition support to staging? Michael Schmitz
@ 2018-06-28  3:44                             ` jdow
  2018-06-28  5:43                               ` Michael Schmitz
  0 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-28  3:44 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Michael, as long as m68k only supports int fseek( int ) 4 GB * block size is 
your HARD limit. Versions that support __int64 fseek64( __int64 ) can work with 
larger disks. RDBs could include RDSK and a new set of other blocks that replace 
the last two characters with "64" and use __int64 where needed in the various 
values. That way a clever disk partitioner could give allow normal (32 bit) RDB 
definitions where possible. Then at least SOME of the disk could be supported 
AND a very clever filesystem that abstracts very large disks properly could give 
access to the whole disk. (Read the RDBs first 32 bits. Then if a filesystem or 
driveinit was loaded re-read the RDBs to see if new 64 bit partitions are revealed.

I could be wrong but I do not think RDBs could be safely modified any other way 
to work. And, trust me as I bet this is still true, you will need a SERIOUSLY 
good bomb shelter on the Moon if you change RDBs. Suppose Joe Amigoid uses it, 
and then Joe Amigoid loads Amigados 2.4 because he wants to run a game that 
crashes on anything newer. Then Joe got far enough something writes to the disk 
and data is corrupted. Note further that Amigoids do NOT, repeat NOT, listen to 
facts in such cases. Hell, some of them never listened to facts about an 
incident at Jerry Pournelle's place when a 1.1 DPaint session with Kelly Freas 
hung and we lost a delightful drawing. Jerry reported it. Amigoids screamed. I 
tried to tell them I was there, it was my machine, and 1.1 was, indeed, crap.

{o.o}

On 20180627 02:00, Michael Schmitz wrote:
> Joanne,
> 
> I'm not at all allergic to avoiding RDB at all cost for new disks. If AmigaOS 
> 4.1 supports more recent partition formats, all the better. This is all about 
> supporting use of legacy RDB disks on Linux (though 2 TB does stretch the 
> definition of 'legacy' a little). My interest in this is to ensure we can 
> continue to use RDB format disks on m68k Amiga computers which have no other way 
> to boot Linux from disk.
> 
> Not proposing to change the RDB format at all, either. Just trying to make sure 
> we translate RDB info into Linux 512-byte block offset and size numbers 
> correctly. The kernel won't modify the RDB at all (intentionally, that is - with 
> the correct choice of partition sizes, Martin might well have wiped out his RDB 
> with the current version of the parser).
> 
> The choice of refusing to mount a disk (or mounting read-only) rests with the 
> VFS drivers alone - AFFS in that case. Not touching any of that. At partition 
> scan time, we only have the option of making the partition available (with a 
> warning printed), or refusing to make it available to the kernel. Once it's made 
> available, all bets are off.
> 
>  From what Martin writes, his test case RDB was valid and worked as expected on 
> 32 bit AmigaOS (4.1). Apparently, that version has the necessary extensions to 
> handle the large offsets resulting from 2 TB disks. Not sure what safeguards are 
> in place when connecting such a disk to older versions of AmigaOS, but that is a 
> different matter entirely.
> 
> The overflows in partition offset and size are the only ones I can see in the 
> partition parser - there is no other overflow I've identified. I just stated 
> that in order to place a partition towards the end of a 2 TB disk, the offset 
> calculation will overflow regardless of what combination of rdb->rdb_BlockBytes 
> and sector addresses stored in the RDB (in units of 512 byte blocks) we use:
> 
>          blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
> 
> 
>                  nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
>                              be32_to_cpu(pb->pb_Environment[9])) *
>                             be32_to_cpu(pb->pb_Environment[3]) *
>                             be32_to_cpu(pb->pb_Environment[5]) *
>                             blksize;
>                  if (!nr_sects)
>                          continue;
>                  start_sect = be32_to_cpu(pb->pb_Environment[9]) *
>                               be32_to_cpu(pb->pb_Environment[3]) *
>                               be32_to_cpu(pb->pb_Environment[5]) *
>                               blksize;
> 
> But in the interest of avoiding any accidental use of a RDB partition where 
> calculations currently overflow, I'll make the default behaviour to bail out 
> (instead of using wrong offset or size as we currently do). Given the 
> 'eat_my_RDB_disk=1' boot option, the user may proceed at their own risk (though 
> I still can't see what harm should result from now translating a well formed 
> v4.1 2 TB disk RDB correctly for the first time).
> 
> Whether or not Linux correctly handles AFFS filesystems larger than 1 TB is a 
> matter for VFS experts. Bailing out on nr_sects overflowing ought to prevent 
> accidental use of AFFS filesystems on RDB disks which I suppose is the majority 
> of use cases.
> 
> Bugs in partitioning tools on Linux are entirely out of scope - the partitioning 
> tools bypass the partition structure discovered by the kernel, and work straight 
> on the raw device. No protecting against that.
> 
> If you can point out a way to cause data loss with these precautions, for a disk 
> 2 TB or larger that was partitioned and used on a recent version or AmigaOS 
> supporting such large disks, I'd consider omitting the 'eat_my_RDB_disk' boot 
> option, and just bail out as the only safe option.
> 
> Cheers,
> 
>      Michael
> 
> 
> Am 27.06.2018 um 18:24 schrieb jdow:
>> You allergic to using a GPT solution? It will get away from some of the
>> evils that RDB has inherent in it because they are also features?
>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>> system that sees an RDSK block is going to try to translate it. If you
>> add a new RDB format you are going to get bizarre and probably quite
>> destructive results from the mistake. Fail safe is a rather good notion,
>> methinks.
>>
>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>> system seems utterly outlandish to me. You cited another overflow
>> potential. There are at least three we've identified, I believe. Are you
>> 100% sure there are no more? The specific one you mention of translating
>> RDB to Linux has a proper solution in the RDB reader. It should recover
>> such overflow errors in the RDB as it can with due care and polish. It
>> should flag any other overflow error it detects within the RDBs and
>> return an error such as to leave the disk unmounted or mounted read-only
>> if you feel like messing up a poor sod's backups. The simple solution is
>> to read each of the variables with the nominal RDB size and convert it
>> to uint64_t before calculating byte indices.
>>
>> However, consider my inputs as advice from an adult who has seen the
>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>> you wish; but, I would plead with you to avoid ANY chance you can for
>> the user to make a bonehead stupid move and lose all his treasured disk
>> archives. Doing otherwise is very poor form.
>>
>> {o.o}   Joanne "Said enough, she has" Dow
>>
>> On 20180626 18:07, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> As far as I have been able to test, the change is backwards compatible
>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>> That"s only been done on an emulator though.
>>>
>>> Your advice about the dangers of using RDB disks that would have
>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>> in question ever used on a 32 bit Amiga system?
>>>
>>> RDB disk format is meant for legacy use only, so I think we can get
>>> away with printing a big fat warning during boot, advising the user
>>> that the oversize RDB partition(s) scanned are not compatible with
>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>> partition at disk end and trashing valid partitions that overlap,
>>> which is what Martin was after.
>>>
>>> If that still seems too risky, we can make the default behaviour to
>>> bail out once a potential overflow is detected, and allow the user to
>>> override that through a boot parameter. I'd leave that decision up for
>>> the code review on linux-block.
>>>
>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>> start and size calculations, so a change of the RDB blocksize to
>>> reduce the block counts stored in the RDB would still result in the
>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>> updating (along with probably most legacy m68k partitioners). Adrian
>>> has advertised parted as replacement for the old tools - maybe this
>>> would make a nice test case for parted?
>>>
>>> Cheers,
>>>
>>>    Michael
>>>
>>>
>>>
>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>>> If it is not backwards compatible I for one would refuse to use it.
>>>> And if
>>>> it still mattered that much to me I'd also generate a reasonable
>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>> good idea.
>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>> uint32_t
>>>> only system. If it is for your personal use then you're entirely free to
>>>> reject my advice and are probably smart enough to keep it working for
>>>> yourself.
>>>>
>>>> GPT is probably the right way to go. Preserve the ability to read
>>>> RDBs for
>>>> legacy disks only.
>>>>
>>>> {^_^}
>>>>
>>>>
>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>
>>>>> Joanne,
>>>>>
>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>>>> issues Martin reported. Your patch addresses that by using the correct
>>>>> data type for the calculations (as do other partition parsers that may
>>>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>>>> the right thing to do.
>>>>>
>>>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>>>> don't currently overflow is not expected to cause new issues, other
>>>>> than
>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>> ramifications with filesystems on these partitions). So comptibility is
>>>>> preserved.
>>>>>
>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>> issues in filesystems as well, but I can't see how the block size
>>>>> stored
>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>> Linux
>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>> Viro
>>>>> may have found a few issues when he looked at the AFFS code so I won't
>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>> patch.
>>>>>
>>>>> If you feel strongly about unknown ramifications of any filesystems on
>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>> warning about these partitions.
>>>>>
>>>>> I'll get this patch tested on Martin's test case image as well as on a
>>>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>> use
>>>>> for tests is a long way away from my home indeed.
>>>>>
>>>>> Cheers,
>>>>>
>>>>>       Michael
>>>>>
>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>
>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>> {^_-}
>>>>>>
>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>> change. I remember thinking about this for awhile and then determining
>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>> into a gordian knot.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>
>>>>>>> Joanne,
>>>>>>>
>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>>>> 4)
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>> Attached SCSI disk
>>>>>>>
>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>> size.
>>>>>>>
>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>>      Michael
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>>>
>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>>>> a famn
>>>>>>>> dool.
>>>>>>>>
>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>> should
>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>> block maps.
>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>> so if
>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>
>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>> preparing
>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>
>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>> which
>>>>>>>>> I still have in my apartment.
>>>>>>>>>
>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>> unless
>>>>>>>>> someone else does.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>> -- 
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>> -- 
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-28  3:44                             ` jdow
@ 2018-06-28  5:43                               ` Michael Schmitz
  2018-06-28  6:39                                 ` jdow
  2018-06-28  8:07                                 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
  0 siblings, 2 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-28  5:43 UTC (permalink / raw)
  To: jdow
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Joanne,

Linux on m68k has supported lseek64 (or llseek) for a long time (from 
glibc version 2.1 according to what I found). About the only area where 
we are limited by 32 bits is the virtual memory size.

I'm not proposing to modify the RDB format definition, though an 
extension to store 64 bit offsets separate from the 32 bit ones would be 
one way to make certain such disks are safe to use on 3.1 and earlier 
versions of AmigaOS. (Another one would be to modify the disk drivers on 
older versions to do the offset calculation in 64 bit, and check for 
overflow just as we do here. Not sure whether that's feasible. And as 
you so eloquently describe, we can't rely on users listening.)

Either way, we need the cooperation of partitioning tool writers to 
ensure partition information that is prone to overflows is never stored 
in the 32 bit, classic RDB. That appears to have failed already, as 
Martin's experience illustrates.

I'm only concerned with fixing the (dangerous) but in the Linux 
partition format parser for RDB. Refusing to use any partitions that 
will cause havoc on old AmigaOS versions is all I can do to try and get 
the users' attention.

Your warning makes me wonder whether the log message should just say 
'report this bug to linux-m68k@vger.kernel.org' to at least try and 
educate any that respond about the dangers of their partitioning scheme 
before telling them about the override option. Problem with that is, in 
about three years no one will remember any of this ...

Cheers,

	Michael


Am 28.06.2018 um 15:44 schrieb jdow:
> Michael, as long as m68k only supports int fseek( int ) 4 GB * block
> size is your HARD limit. Versions that support __int64 fseek64( __int64
> ) can work with larger disks. RDBs could include RDSK and a new set of
> other blocks that replace the last two characters with "64" and use
> __int64 where needed in the various values. That way a clever disk
> partitioner could give allow normal (32 bit) RDB definitions where
> possible. Then at least SOME of the disk could be supported AND a very
> clever filesystem that abstracts very large disks properly could give
> access to the whole disk. (Read the RDBs first 32 bits. Then if a
> filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit
> partitions are revealed.
>
> I could be wrong but I do not think RDBs could be safely modified any
> other way to work. And, trust me as I bet this is still true, you will
> need a SERIOUSLY good bomb shelter on the Moon if you change RDBs.
> Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4
> because he wants to run a game that crashes on anything newer. Then Joe
> got far enough something writes to the disk and data is corrupted. Note
> further that Amigoids do NOT, repeat NOT, listen to facts in such cases.
> Hell, some of them never listened to facts about an incident at Jerry
> Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we
> lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried
> to tell them I was there, it was my machine, and 1.1 was, indeed, crap.
>
> {o.o}
>
> On 20180627 02:00, Michael Schmitz wrote:
>> Joanne,
>>
>> I'm not at all allergic to avoiding RDB at all cost for new disks. If
>> AmigaOS 4.1 supports more recent partition formats, all the better.
>> This is all about supporting use of legacy RDB disks on Linux (though
>> 2 TB does stretch the definition of 'legacy' a little). My interest in
>> this is to ensure we can continue to use RDB format disks on m68k
>> Amiga computers which have no other way to boot Linux from disk.
>>
>> Not proposing to change the RDB format at all, either. Just trying to
>> make sure we translate RDB info into Linux 512-byte block offset and
>> size numbers correctly. The kernel won't modify the RDB at all
>> (intentionally, that is - with the correct choice of partition sizes,
>> Martin might well have wiped out his RDB with the current version of
>> the parser).
>>
>> The choice of refusing to mount a disk (or mounting read-only) rests
>> with the VFS drivers alone - AFFS in that case. Not touching any of
>> that. At partition scan time, we only have the option of making the
>> partition available (with a warning printed), or refusing to make it
>> available to the kernel. Once it's made available, all bets are off.
>>
>>  From what Martin writes, his test case RDB was valid and worked as
>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
>> necessary extensions to handle the large offsets resulting from 2 TB
>> disks. Not sure what safeguards are in place when connecting such a
>> disk to older versions of AmigaOS, but that is a different matter
>> entirely.
>>
>> The overflows in partition offset and size are the only ones I can see
>> in the partition parser - there is no other overflow I've identified.
>> I just stated that in order to place a partition towards the end of a
>> 2 TB disk, the offset calculation will overflow regardless of what
>> combination of rdb->rdb_BlockBytes and sector addresses stored in the
>> RDB (in units of 512 byte blocks) we use:
>>
>>          blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
>>
>>
>>                  nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
>>                              be32_to_cpu(pb->pb_Environment[9])) *
>>                             be32_to_cpu(pb->pb_Environment[3]) *
>>                             be32_to_cpu(pb->pb_Environment[5]) *
>>                             blksize;
>>                  if (!nr_sects)
>>                          continue;
>>                  start_sect = be32_to_cpu(pb->pb_Environment[9]) *
>>                               be32_to_cpu(pb->pb_Environment[3]) *
>>                               be32_to_cpu(pb->pb_Environment[5]) *
>>                               blksize;
>>
>> But in the interest of avoiding any accidental use of a RDB partition
>> where calculations currently overflow, I'll make the default behaviour
>> to bail out (instead of using wrong offset or size as we currently
>> do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed
>> at their own risk (though I still can't see what harm should result
>> from now translating a well formed v4.1 2 TB disk RDB correctly for
>> the first time).
>>
>> Whether or not Linux correctly handles AFFS filesystems larger than 1
>> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
>> ought to prevent accidental use of AFFS filesystems on RDB disks which
>> I suppose is the majority of use cases.
>>
>> Bugs in partitioning tools on Linux are entirely out of scope - the
>> partitioning tools bypass the partition structure discovered by the
>> kernel, and work straight on the raw device. No protecting against that.
>>
>> If you can point out a way to cause data loss with these precautions,
>> for a disk 2 TB or larger that was partitioned and used on a recent
>> version or AmigaOS supporting such large disks, I'd consider omitting
>> the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
>> option.
>>
>> Cheers,
>>
>>      Michael
>>
>>
>> Am 27.06.2018 um 18:24 schrieb jdow:
>>> You allergic to using a GPT solution? It will get away from some of the
>>> evils that RDB has inherent in it because they are also features?
>>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>>> system that sees an RDSK block is going to try to translate it. If you
>>> add a new RDB format you are going to get bizarre and probably quite
>>> destructive results from the mistake. Fail safe is a rather good notion,
>>> methinks.
>>>
>>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>>> system seems utterly outlandish to me. You cited another overflow
>>> potential. There are at least three we've identified, I believe. Are you
>>> 100% sure there are no more? The specific one you mention of translating
>>> RDB to Linux has a proper solution in the RDB reader. It should recover
>>> such overflow errors in the RDB as it can with due care and polish. It
>>> should flag any other overflow error it detects within the RDBs and
>>> return an error such as to leave the disk unmounted or mounted read-only
>>> if you feel like messing up a poor sod's backups. The simple solution is
>>> to read each of the variables with the nominal RDB size and convert it
>>> to uint64_t before calculating byte indices.
>>>
>>> However, consider my inputs as advice from an adult who has seen the
>>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>>> you wish; but, I would plead with you to avoid ANY chance you can for
>>> the user to make a bonehead stupid move and lose all his treasured disk
>>> archives. Doing otherwise is very poor form.
>>>
>>> {o.o}   Joanne "Said enough, she has" Dow
>>>
>>> On 20180626 18:07, Michael Schmitz wrote:
>>>> Joanne,
>>>>
>>>> As far as I have been able to test, the change is backwards compatible
>>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>>> That"s only been done on an emulator though.
>>>>
>>>> Your advice about the dangers of using RDB disks that would have
>>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>>> in question ever used on a 32 bit Amiga system?
>>>>
>>>> RDB disk format is meant for legacy use only, so I think we can get
>>>> away with printing a big fat warning during boot, advising the user
>>>> that the oversize RDB partition(s) scanned are not compatible with
>>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>>> partition at disk end and trashing valid partitions that overlap,
>>>> which is what Martin was after.
>>>>
>>>> If that still seems too risky, we can make the default behaviour to
>>>> bail out once a potential overflow is detected, and allow the user to
>>>> override that through a boot parameter. I'd leave that decision up for
>>>> the code review on linux-block.
>>>>
>>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>>> start and size calculations, so a change of the RDB blocksize to
>>>> reduce the block counts stored in the RDB would still result in the
>>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>>> updating (along with probably most legacy m68k partitioners). Adrian
>>>> has advertised parted as replacement for the old tools - maybe this
>>>> would make a nice test case for parted?
>>>>
>>>> Cheers,
>>>>
>>>>    Michael
>>>>
>>>>
>>>>
>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>>>> If it is not backwards compatible I for one would refuse to use it.
>>>>> And if
>>>>> it still mattered that much to me I'd also generate a reasonable
>>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>>> good idea.
>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>>> uint32_t
>>>>> only system. If it is for your personal use then you're entirely
>>>>> free to
>>>>> reject my advice and are probably smart enough to keep it working for
>>>>> yourself.
>>>>>
>>>>> GPT is probably the right way to go. Preserve the ability to read
>>>>> RDBs for
>>>>> legacy disks only.
>>>>>
>>>>> {^_^}
>>>>>
>>>>>
>>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>>
>>>>>> Joanne,
>>>>>>
>>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>>> the
>>>>>> issues Martin reported. Your patch addresses that by using the
>>>>>> correct
>>>>>> data type for the calculations (as do other partition parsers that
>>>>>> may
>>>>>> have to deal with large disks) and fixes Martin's bug, so appears
>>>>>> to be
>>>>>> the right thing to do.
>>>>>>
>>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>>> calculations
>>>>>> don't currently overflow is not expected to cause new issues, other
>>>>>> than
>>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>>> ramifications with filesystems on these partitions). So
>>>>>> comptibility is
>>>>>> preserved.
>>>>>>
>>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>>> issues in filesystems as well, but I can't see how the block size
>>>>>> stored
>>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>>> Linux
>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>>> Viro
>>>>>> may have found a few issues when he looked at the AFFS code so I
>>>>>> won't
>>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>>> patch.
>>>>>>
>>>>>> If you feel strongly about unknown ramifications of any
>>>>>> filesystems on
>>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>>> warning about these partitions.
>>>>>>
>>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>>> on a
>>>>>> RDB image from a disk known to currently work under Linux (thanks
>>>>>> Geert
>>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>>> use
>>>>>> for tests is a long way away from my home indeed.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>       Michael
>>>>>>
>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>>
>>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>>> block
>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>>> under
>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>>> {^_-}
>>>>>>>
>>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>>> change. I remember thinking about this for awhile and then
>>>>>>> determining
>>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>>> into a gordian knot.
>>>>>>>
>>>>>>> {^_^}
>>>>>>>
>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>>
>>>>>>>> Joanne,
>>>>>>>>
>>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>>
>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)
>>>>>>>> sdb1
>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>>> 2 spb
>>>>>>>> 4)
>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>>> Attached SCSI disk
>>>>>>>>
>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>>> 512
>>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>>> size.
>>>>>>>>
>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>>> yet.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>>      Michael
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>>>>
>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>>> system is
>>>>>>>>> a famn
>>>>>>>>> dool.
>>>>>>>>>
>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>>> should
>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>>> block maps.
>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>>
>>>>>>>>> {^_^}
>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>>
>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> test results at
>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>>> so if
>>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>>
>>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>>> preparing
>>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>>
>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>>> which
>>>>>>>>>> I still have in my apartment.
>>>>>>>>>>
>>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>>> unless
>>>>>>>>>> someone else does.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>> linux-m68k" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: moving affs + RDB partition support to staging?
  2018-06-28  5:43                               ` Michael Schmitz
@ 2018-06-28  6:39                                 ` jdow
  2018-06-28  8:16                                   ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
  2018-06-28  8:07                                 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
  1 sibling, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-28  6:39 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Anything done to RDBs for Linux must remain 100.000% compatible with existing 
Amiga equipment. Otherwise, what's the point of bothering to use RDBs?

Note, as an experiment I had a CrossDOS partition on a hard disk briefly. So 
some amazing nonsense is possible. I concluded that for me it was not something 
I wanted to do. Alternative techniques worked well enough.

That brings to the fore an interesting question. Why bother with RDBs over 2TB 
unless you want a disk with one single partition? This Win10 monster I am using 
has a modest BIOS driver partition for the OS and a giant data partition. That 
smaller partition would easily work with any RDB/Filesystem combination since 
2.0. So there are some good workarounds that are probably "safer" and at least 
as flexible as RDBs, one Linux has used for a very long time, too.

With that it is easy in concept to put together a dual boot system with all the 
disks and partitions readable if you play the proper games at the filesystem 
level and keep individual partitions small. The filesystem would have to take on 
some of the attributes of a device driver, though. A 281 TB disk made up of 2 TB 
partitions should be possible to create if you think it through. (Note that with 
RDBs the CHS disk description is purely an abstraction that has no relationship 
to anything inside the disk. The same tricks could exist at the filesystem level.)

As for existing tools screwing up, do you think that is an excuse to perpetuate 
the failure?

If we are speaking Linux for m68k, what does it do when confronted with which is 
described with a size larger than 32 bits? Before it had everything inside it 
that refers to positions within a file by byte index with an __int64 type of 
storage did it throw up it's hands, corrupt the disk, or what? If the m68k Linux 
is a failure perhaps it doesn't matter what you do for Linuxoids. (I'll stick 
with the x64 version in that case. And I never go back to earlier versions for 
games since I consider a good compiler is the best game around. It's better than 
a blank piece of paper for possibilities.)

As I have said, for the RDB parser fix the famndool thing. Do fix it right in 
such a manner that if somebody compiles it against a version with no 64 bit 
device code it will throw a proper overflow error and protect the user. Users 
are dumb. We like to think of ourselves as smart. Let's try to be smart about 
this where we can so fingers can't point back at us rather than the fool that 
made some other error.

And do remember, I am merely (and vociferously) advising rather than dictating. 
You don't need my approval to proceed. I may want my name noted as an early 
contributor only. Meanwhile I spit out ideas as they come to me. One or more of 
them might be good. And offering alternatives is better than simply saying "No" 
most of the time.

If people are using RDBs for TB level disks I doubt they can remember which is 
the left shoe when they are getting dressed in the morning before going out in 
the yard to beat some dead horses. Or else maybe they just want to see how far 
they can flog the m68k architecture as a mental challenge. In that case, taking 
it too seriously could hurt. Note that I am mostly ignoring m68k Linux. People 
using that are hard core. People using x86/x64 Linux aren't such hard core 
folks. And I bet most of them want to read the disks so they can copy stuff to 
Amiga Forever or WinUAE running on other architectures. So TB is not likely to 
be an issue for them, either.

{^_^}

On 20180627 22:43, Michael Schmitz wrote:
> Joanne,
> 
> Linux on m68k has supported lseek64 (or llseek) for a long time (from glibc 
> version 2.1 according to what I found). About the only area where we are limited 
> by 32 bits is the virtual memory size.
> 
> I'm not proposing to modify the RDB format definition, though an extension to 
> store 64 bit offsets separate from the 32 bit ones would be one way to make 
> certain such disks are safe to use on 3.1 and earlier versions of AmigaOS. 
> (Another one would be to modify the disk drivers on older versions to do the 
> offset calculation in 64 bit, and check for overflow just as we do here. Not 
> sure whether that's feasible. And as you so eloquently describe, we can't rely 
> on users listening.)
> 
> Either way, we need the cooperation of partitioning tool writers to ensure 
> partition information that is prone to overflows is never stored in the 32 bit, 
> classic RDB. That appears to have failed already, as Martin's experience 
> illustrates.
> 
> I'm only concerned with fixing the (dangerous) but in the Linux partition format 
> parser for RDB. Refusing to use any partitions that will cause havoc on old 
> AmigaOS versions is all I can do to try and get the users' attention.
> 
> Your warning makes me wonder whether the log message should just say 'report 
> this bug to linux-m68k@vger.kernel.org' to at least try and educate any that 
> respond about the dangers of their partitioning scheme before telling them about 
> the override option. Problem with that is, in about three years no one will 
> remember any of this ...
> 
> Cheers,
> 
>      Michael
> 
> 
> Am 28.06.2018 um 15:44 schrieb jdow:
>> Michael, as long as m68k only supports int fseek( int ) 4 GB * block
>> size is your HARD limit. Versions that support __int64 fseek64( __int64
>> ) can work with larger disks. RDBs could include RDSK and a new set of
>> other blocks that replace the last two characters with "64" and use
>> __int64 where needed in the various values. That way a clever disk
>> partitioner could give allow normal (32 bit) RDB definitions where
>> possible. Then at least SOME of the disk could be supported AND a very
>> clever filesystem that abstracts very large disks properly could give
>> access to the whole disk. (Read the RDBs first 32 bits. Then if a
>> filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit
>> partitions are revealed.
>>
>> I could be wrong but I do not think RDBs could be safely modified any
>> other way to work. And, trust me as I bet this is still true, you will
>> need a SERIOUSLY good bomb shelter on the Moon if you change RDBs.
>> Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4
>> because he wants to run a game that crashes on anything newer. Then Joe
>> got far enough something writes to the disk and data is corrupted. Note
>> further that Amigoids do NOT, repeat NOT, listen to facts in such cases.
>> Hell, some of them never listened to facts about an incident at Jerry
>> Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we
>> lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried
>> to tell them I was there, it was my machine, and 1.1 was, indeed, crap.
>>
>> {o.o}
>>
>> On 20180627 02:00, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> I'm not at all allergic to avoiding RDB at all cost for new disks. If
>>> AmigaOS 4.1 supports more recent partition formats, all the better.
>>> This is all about supporting use of legacy RDB disks on Linux (though
>>> 2 TB does stretch the definition of 'legacy' a little). My interest in
>>> this is to ensure we can continue to use RDB format disks on m68k
>>> Amiga computers which have no other way to boot Linux from disk.
>>>
>>> Not proposing to change the RDB format at all, either. Just trying to
>>> make sure we translate RDB info into Linux 512-byte block offset and
>>> size numbers correctly. The kernel won't modify the RDB at all
>>> (intentionally, that is - with the correct choice of partition sizes,
>>> Martin might well have wiped out his RDB with the current version of
>>> the parser).
>>>
>>> The choice of refusing to mount a disk (or mounting read-only) rests
>>> with the VFS drivers alone - AFFS in that case. Not touching any of
>>> that. At partition scan time, we only have the option of making the
>>> partition available (with a warning printed), or refusing to make it
>>> available to the kernel. Once it's made available, all bets are off.
>>>
>>>  From what Martin writes, his test case RDB was valid and worked as
>>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
>>> necessary extensions to handle the large offsets resulting from 2 TB
>>> disks. Not sure what safeguards are in place when connecting such a
>>> disk to older versions of AmigaOS, but that is a different matter
>>> entirely.
>>>
>>> The overflows in partition offset and size are the only ones I can see
>>> in the partition parser - there is no other overflow I've identified.
>>> I just stated that in order to place a partition towards the end of a
>>> 2 TB disk, the offset calculation will overflow regardless of what
>>> combination of rdb->rdb_BlockBytes and sector addresses stored in the
>>> RDB (in units of 512 byte blocks) we use:
>>>
>>>          blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
>>>
>>>
>>>                  nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
>>>                              be32_to_cpu(pb->pb_Environment[9])) *
>>>                             be32_to_cpu(pb->pb_Environment[3]) *
>>>                             be32_to_cpu(pb->pb_Environment[5]) *
>>>                             blksize;
>>>                  if (!nr_sects)
>>>                          continue;
>>>                  start_sect = be32_to_cpu(pb->pb_Environment[9]) *
>>>                               be32_to_cpu(pb->pb_Environment[3]) *
>>>                               be32_to_cpu(pb->pb_Environment[5]) *
>>>                               blksize;
>>>
>>> But in the interest of avoiding any accidental use of a RDB partition
>>> where calculations currently overflow, I'll make the default behaviour
>>> to bail out (instead of using wrong offset or size as we currently
>>> do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed
>>> at their own risk (though I still can't see what harm should result
>>> from now translating a well formed v4.1 2 TB disk RDB correctly for
>>> the first time).
>>>
>>> Whether or not Linux correctly handles AFFS filesystems larger than 1
>>> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
>>> ought to prevent accidental use of AFFS filesystems on RDB disks which
>>> I suppose is the majority of use cases.
>>>
>>> Bugs in partitioning tools on Linux are entirely out of scope - the
>>> partitioning tools bypass the partition structure discovered by the
>>> kernel, and work straight on the raw device. No protecting against that.
>>>
>>> If you can point out a way to cause data loss with these precautions,
>>> for a disk 2 TB or larger that was partitioned and used on a recent
>>> version or AmigaOS supporting such large disks, I'd consider omitting
>>> the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
>>> option.
>>>
>>> Cheers,
>>>
>>>      Michael
>>>
>>>
>>> Am 27.06.2018 um 18:24 schrieb jdow:
>>>> You allergic to using a GPT solution? It will get away from some of the
>>>> evils that RDB has inherent in it because they are also features?
>>>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>>>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>>>> system that sees an RDSK block is going to try to translate it. If you
>>>> add a new RDB format you are going to get bizarre and probably quite
>>>> destructive results from the mistake. Fail safe is a rather good notion,
>>>> methinks.
>>>>
>>>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>>>> system seems utterly outlandish to me. You cited another overflow
>>>> potential. There are at least three we've identified, I believe. Are you
>>>> 100% sure there are no more? The specific one you mention of translating
>>>> RDB to Linux has a proper solution in the RDB reader. It should recover
>>>> such overflow errors in the RDB as it can with due care and polish. It
>>>> should flag any other overflow error it detects within the RDBs and
>>>> return an error such as to leave the disk unmounted or mounted read-only
>>>> if you feel like messing up a poor sod's backups. The simple solution is
>>>> to read each of the variables with the nominal RDB size and convert it
>>>> to uint64_t before calculating byte indices.
>>>>
>>>> However, consider my inputs as advice from an adult who has seen the
>>>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>>>> you wish; but, I would plead with you to avoid ANY chance you can for
>>>> the user to make a bonehead stupid move and lose all his treasured disk
>>>> archives. Doing otherwise is very poor form.
>>>>
>>>> {o.o}   Joanne "Said enough, she has" Dow
>>>>
>>>> On 20180626 18:07, Michael Schmitz wrote:
>>>>> Joanne,
>>>>>
>>>>> As far as I have been able to test, the change is backwards compatible
>>>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>>>> That"s only been done on an emulator though.
>>>>>
>>>>> Your advice about the dangers of using RDB disks that would have
>>>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>>>> in question ever used on a 32 bit Amiga system?
>>>>>
>>>>> RDB disk format is meant for legacy use only, so I think we can get
>>>>> away with printing a big fat warning during boot, advising the user
>>>>> that the oversize RDB partition(s) scanned are not compatible with
>>>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>>>> partition at disk end and trashing valid partitions that overlap,
>>>>> which is what Martin was after.
>>>>>
>>>>> If that still seems too risky, we can make the default behaviour to
>>>>> bail out once a potential overflow is detected, and allow the user to
>>>>> override that through a boot parameter. I'd leave that decision up for
>>>>> the code review on linux-block.
>>>>>
>>>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>>>> start and size calculations, so a change of the RDB blocksize to
>>>>> reduce the block counts stored in the RDB would still result in the
>>>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>>>> updating (along with probably most legacy m68k partitioners). Adrian
>>>>> has advertised parted as replacement for the old tools - maybe this
>>>>> would make a nice test case for parted?
>>>>>
>>>>> Cheers,
>>>>>
>>>>>    Michael
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>>>>> If it is not backwards compatible I for one would refuse to use it.
>>>>>> And if
>>>>>> it still mattered that much to me I'd also generate a reasonable
>>>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>>>> good idea.
>>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>>>> uint32_t
>>>>>> only system. If it is for your personal use then you're entirely
>>>>>> free to
>>>>>> reject my advice and are probably smart enough to keep it working for
>>>>>> yourself.
>>>>>>
>>>>>> GPT is probably the right way to go. Preserve the ability to read
>>>>>> RDBs for
>>>>>> legacy disks only.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>>
>>>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>>>
>>>>>>> Joanne,
>>>>>>>
>>>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>>>> the
>>>>>>> issues Martin reported. Your patch addresses that by using the
>>>>>>> correct
>>>>>>> data type for the calculations (as do other partition parsers that
>>>>>>> may
>>>>>>> have to deal with large disks) and fixes Martin's bug, so appears
>>>>>>> to be
>>>>>>> the right thing to do.
>>>>>>>
>>>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>>>> calculations
>>>>>>> don't currently overflow is not expected to cause new issues, other
>>>>>>> than
>>>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>>>> ramifications with filesystems on these partitions). So
>>>>>>> comptibility is
>>>>>>> preserved.
>>>>>>>
>>>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>>>> issues in filesystems as well, but I can't see how the block size
>>>>>>> stored
>>>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>>>> Linux
>>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>>>> Viro
>>>>>>> may have found a few issues when he looked at the AFFS code so I
>>>>>>> won't
>>>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>>>> patch.
>>>>>>>
>>>>>>> If you feel strongly about unknown ramifications of any
>>>>>>> filesystems on
>>>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>>>> warning about these partitions.
>>>>>>>
>>>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>>>> on a
>>>>>>> RDB image from a disk known to currently work under Linux (thanks
>>>>>>> Geert
>>>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>>>> use
>>>>>>> for tests is a long way away from my home indeed.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>>       Michael
>>>>>>>
>>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>>>
>>>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>>>> block
>>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>>>> under
>>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>>>> {^_-}
>>>>>>>>
>>>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>>>> change. I remember thinking about this for awhile and then
>>>>>>>> determining
>>>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>>>> into a gordian knot.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>>
>>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>>>
>>>>>>>>> Joanne,
>>>>>>>>>
>>>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>>>
>>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)
>>>>>>>>> sdb1
>>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>>>> 2 spb
>>>>>>>>> 4)
>>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>>>> Attached SCSI disk
>>>>>>>>>
>>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>>>> 512
>>>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>>>> size.
>>>>>>>>>
>>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>>>> yet.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>>      Michael
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>>>>>
>>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>>>> system is
>>>>>>>>>> a famn
>>>>>>>>>> dool.
>>>>>>>>>>
>>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>>>> should
>>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>>>> block maps.
>>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>>>
>>>>>>>>>> {^_^}
>>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi.
>>>>>>>>>>>
>>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> test results at
>>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>>>> so if
>>>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>>>
>>>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>>>> preparing
>>>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>>>
>>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>>>> which
>>>>>>>>>>> I still have in my apartment.
>>>>>>>>>>>
>>>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>>>> unless
>>>>>>>>>>> someone else does.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>> linux-m68k" in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>> -- 
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>> -- 
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)
  2018-06-28  2:57                             ` jdow
@ 2018-06-28  7:40                               ` Martin Steigerwald
  0 siblings, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28  7:40 UTC (permalink / raw)
  To: jdow
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Changing subject, so that there is at least a chance for someone to find 
this discussions with a search engine :)

Joanne,

jdow - 28.06.18, 04:57:
> The issue is what happens when one of those disks appears on a 3.1
> system. {^_^}

That is right, so I think the warning about 64 bit support in native OS 
is okay, but that issue already exists *without* Linux.

Remember, I created that RDB with Media Toolbox on AmigaOS 4.0. I did 
not even use Linux to create that beast :). If it booms in AmigaOS < 4 
without NSD64, TD64 or SCSI direct, that would happen with or without 
the warning in Linux, even without the disk ever have been seen by a 
Linux kernel.

I´d say the warning about support in native OS does not harm, even when 
it is about educating Amiga users who, in case they use that large 
drives – and I pretty much bet, some of them do –, better know the 
limitations beforehand.

I do think the extra kernel option does not make all that much sense, 
but I accept it anyway.

Cause if you argue like that, what would need fixing is AmigaOS < 4 
without NSD64, TD64 or SCSI direct, but then that is what NSD64 and TD64 
was made for more than 10 years ago.

Of course, if a partitioning tool for Linux ever allows to create such 
an RDB, it makes sense to add a big fat warning about that. As… I think 
would make sense to have in Media Toolbox and AmigaOS partitioning 
tools.

However Linux here just reads the RDB, so I´d personally go with the 
warning about support in native OS, but spare myself the extra kernel 
option stuff.

It is Michael´s call tough, as he submits the patch. And if he chooses 
to be on a safer side than this, that is fine with me.

Thanks,
Martin

> On 20180627 01:03, Martin Steigerwald wrote:
> > Dear Joanne.
> > 
> > jdow - 27.06.18, 08:24:
> >> You allergic to using a GPT solution? It will get away from some of
> >> the evils that RDB has inherent in it because they are also
> >> features?
> >> (Loading a filesystem or DriveInit code from RDBs is just asking
> >> for
> >> a nearly impossible to remove malware infection.) Furthermore, any
> >> 32
> >> bit system that sees an RDSK block is going to try to translate it.
> >> If you add a new RDB format you are going to get bizarre and
> >> probably
> >> quite destructive results from the mistake. Fail safe is a rather
> >> good notion, methinks.
> >> 
> >> Personally I figure this is all rather surreal. 2TG of junk on an
> >> Amiga system seems utterly outlandish to me. You cited another
> >> overflow potential. There are at least three we've identified, I
> >> believe. Are you 100% sure there are no more? The specific one you
> >> mention of translating RDB to Linux has a proper solution in the
> >> RDB
> >> reader. It should recover such overflow errors in the RDB as it can
> >> with due care and polish. It should flag any other overflow error
> >> it
> >> detects within the RDBs and return an error such as to leave the
> >> disk
> >> unmounted or mounted read-only if you feel like messing up a poor
> >> sod's backups. The simple solution is to read each of the variables
> >> with the nominal RDB size and convert it to uint64_t before
> >> calculating byte indices.
> >> 
> >> However, consider my inputs as advice from an adult who has seen
> >> the
> >> Amiga Elephant so to speak. I am not trying to assert any control.
> >> Do
> >> as you wish; but, I would plead with you to avoid ANY chance you
> >> can
> >> for the user to make a bonehead stupid move and lose all his
> >> treasured disk archives. Doing otherwise is very poor form.
> > 
> > I am pretty confident that larger than 2 TiB disks are fully
> > supported within AmigaOS 4, as I outlined in my other mail.
> > 
> > So with all due respect: I used a larger than 2 TiB disk in AmigaOS
> > 4 in 2012 already *just* fine. I even found I had the same
> > questions back then, and researched it. Which lead to this official
> > article back then:
> > 
> > http://wiki.amigaos.net/wiki/RDB
> > 
> > I am also pretty sure that AmigaOS still uses RDB as partitioning
> > format. They support MBR. I don´t think AmigaOS 4.1 supports GPT.
> > Whether to implement that of course is the decision of AmigaOS 4
> > development team. I am no longer a member of it since some time.
> > 
> > Linux m68k should already be able to use disks in GPT format, but
> > you
> > likely won´t be able to read them in AmigaOS, unless there is some
> > third party support for it meanwhile.
> > 
> > Thanks,
> > Martin
> > 
> >> {o.o}   Joanne "Said enough, she has" Dow
> >> 
> >> On 20180626 18:07, Michael Schmitz wrote:
> >>> Joanne,
> >>> 
> >>> As far as I have been able to test, the change is backwards
> >>> compatible (RDB partitions from an old disk 80 GB disk are still
> >>> recognized OK). That"s only been done on an emulator though.
> >>> 
> >>> Your advice about the dangers of using RDB disks that would have
> >>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>> well
> >>> taken though. Maybe Martin can clarify that for me - was the 2 TB
> >>> disk in question ever used on a 32 bit Amiga system?
> >>> 
> >>> RDB disk format is meant for legacy use only, so I think we can
> >>> get
> >>> away with printing a big fat warning during boot, advising the
> >>> user
> >>> that the oversize RDB partition(s) scanned are not compatible with
> >>> legacy 32 bit AmigaOS. With the proposed fix they will work under
> >>> both AmigaOS 4.1 and Linux instead of truncating the first
> >>> overflowing partition at disk end and trashing valid partitions
> >>> that overlap, which is what Martin was after.
> >>> 
> >>> If that still seems too risky, we can make the default behaviour
> >>> to
> >>> bail out once a potential overflow is detected, and allow the user
> >>> to
> >>> override that through a boot parameter. I'd leave that decision up
> >>> for the code review on linux-block.
> >>> 
> >>> Two more comments: Linux uses 512 byte block sizes for the
> >>> partition
> >>> start and size calculations, so a change of the RDB blocksize to
> >>> reduce the block counts stored in the RDB would still result in
> >>> the
> >>> same overflow. And amiga-fdisk is indeed utterly broken and needs
> >>> updating (along with probably most legacy m68k partitioners).
> >>> Adrian
> >>> has advertised parted as replacement for the old tools - maybe
> >>> this
> >>> would make a nice test case for parted?
> >>> 
> >>> Cheers,
> >>> 
> >>>     Michael
> >>> 
> >>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> >>>> If it is not backwards compatible I for one would refuse to use
> >>>> it.
> >>>> And if it still mattered that much to me I'd also generate a
> >>>> reasonable alternative. Modifying RDBs nay not be even an
> >>>> approximation of a good idea. You'd discover that as soon as an
> >>>> RDB uint64_t disk is tasted by a uint32_t only system. If it is
> >>>> for your personal use then you're entirely free to reject my
> >>>> advice and are probably smart enough to keep it working for
> >>>> yourself.
> >>>> 
> >>>> GPT is probably the right way to go. Preserve the ability to read
> >>>> RDBs for legacy disks only.
> >>>> 
> >>>> {^_^}
> >>>> 
> >>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>> 
> >>>>> I think we all agree that doing 32 bit calculations on 512-byte
> >>>>> block
> >>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>> causing
> >>>>> the issues Martin reported. Your patch addresses that by using
> >>>>> the correct data type for the calculations (as do other
> >>>>> partition
> >>>>> parsers that may have to deal with large disks) and fixes
> >>>>> Martin's bug, so appears to be the right thing to do.
> >>>>> 
> >>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>> calculations don't currently overflow is not expected to cause
> >>>>> new issues, other than enabling use of disk and partitions
> >>>>> larger
> >>>>> than 2 TB (which may have ramifications with filesystems on
> >>>>> these
> >>>>> partitions). So comptibility is preserved.
> >>>>> 
> >>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>> overflow
> >>>>> issues in filesystems as well, but I can't see how the block
> >>>>> size
> >>>>> stored in the RDB would enforce use of the same block size in
> >>>>> filesystems. We'll have to rely on the filesystem tools to get
> >>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >>>>> limitation) so this should allow partitions larger than 2 TB to
> >>>>> work already (but I suspect Al Viro may have found a few issues
> >>>>> when he looked at the AFFS code so I won't say more). Anyway
> >>>>> partitioning tools and filesystems are unrelated to the Linux
> >>>>> partition parser code which is all we aim to fix in this patch.
> >>>>> 
> >>>>> If you feel strongly about unknown ramifications of any
> >>>>> filesystems on partitions larger than 2 TB, say so and I'll have
> >>>>> the kernel print a warning about these partitions.
> >>>>> 
> >>>>> I'll get this patch tested on Martin's test case image as well
> >>>>> as
> >>>>> on a RDB image from a disk known to currently work under Linux
> >>>>> (thanks Geert for the losetup hint). Can't do much more without
> >>>>> procuring a working Amiga disk image to use with an emulator,
> >>>>> sorry. The Amiga I plan to use for tests is a long way away from
> >>>>> my home indeed.
> >>>>> 
> >>>>> Cheers,
> >>>>> 
> >>>>>        Michael
> >>>>> 
> >>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>> suppose.
> >>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>> force
> >>>>>> the
> >>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>> systems
> >>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>> blocks
> >>>>>> list will have to go beyond a uint32_t size, for example. But a
> >>>>>> block
> >>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>> tad
> >>>>>> under 1% of the disk all by itself. A block bitmap is not quite
> >>>>>> so bad. {^_-}
> >>>>>> 
> >>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>> make
> >>>>>> a
> >>>>>> change. I remember thinking about this for awhile and then
> >>>>>> determining I REALLY did not want to think about it as my brain
> >>>>>> was getting tied into a gordian knot.
> >>>>>> 
> >>>>>> {^_^}
> >>>>>> 
> >>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>> Joanne,
> >>>>>>> 
> >>>>>>> Martin's boot log (including your patch) says:
> >>>>>>> 
> >>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK
> >>>>>>> (512)
> >>>>>>> sdb1
> >>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>> (DOS^C)(res
> >>>>>>> 2 spb
> >>>>>>> 4)
> >>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>> [sdb]
> >>>>>>> Attached SCSI disk
> >>>>>>> 
> >>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>> means
> >>>>>>> 512
> >>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>> block
> >>>>>>> size.
> >>>>>>> 
> >>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>> units.
> >>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>> others
> >>>>>>> yet.
> >>>>>>> 
> >>>>>>> Cheers,
> >>>>>>> 
> >>>>>>>       Michael
> >>>>>>> 
> >>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
> > 
> > wrote:
> >>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>> system is
> >>>>>>>> a famn
> >>>>>>>> dool.
> >>>>>>>> 
> >>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>> so
> >>>>>>>> it
> >>>>>>>> should
> >>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>> blocks
> >>>>>>>> is
> >>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
> >>>>>>>> in
> >>>>>>>> block maps.
> >>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>> 
> >>>>>>>> {^_^}
> >>>>>>>> 
> >>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>> Hi.
> >>>>>>>>> 
> >>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>> test results at
> >>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>> there, so if
> >>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>> 
> >>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>> 
> >>>>>>>>> I do not care enough about this, in order to motivate myself
> >>>>>>>>> preparing the a patch from Joanne Dow´s fix.
> >>>>>>>>> 
> >>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>> Sam440ep
> >>>>>>>>> which
> >>>>>>>>> I still have in my apartment.
> >>>>>>>>> 
> >>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>>>>> TB,
> >>>>>>>>> unless
> >>>>>>>>> someone else does.
> >>>>>>>>> 
> >>>>>>>>> Thanks.
> >>>>>>>> 
> >>>>>>>> --
> >>>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>>> linux-m68k" in
> >>>>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>>>> More majordomo info at
> >>>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>>> 
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>> linux-m68k" in the body of a message to
> >>>>>> majordomo@vger.kernel.org
> >>>>>> More majordomo info at
> >>>>>> http://vger.kernel.org/majordomo-info.html
> >>>> 
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe
> >>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
> >>>> More majordomo info at 
> >>>> http://vger.kernel.org/majordomo-info.html


-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)
  2018-06-28  5:43                               ` Michael Schmitz
  2018-06-28  6:39                                 ` jdow
@ 2018-06-28  8:07                                 ` Martin Steigerwald
  1 sibling, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28  8:07 UTC (permalink / raw)
  To: Michael Schmitz
  Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
	Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
	linux-m68k

Michael Schmitz - 28.06.18, 07:43:
> Joanne,
> 
> Linux on m68k has supported lseek64 (or llseek) for a long time (from
> glibc version 2.1 according to what I found). About the only area
> where we are limited by 32 bits is the virtual memory size.
> 
> I'm not proposing to modify the RDB format definition, though an
> extension to store 64 bit offsets separate from the 32 bit ones would
> be one way to make certain such disks are safe to use on 3.1 and
> earlier versions of AmigaOS. (Another one would be to modify the disk
> drivers on older versions to do the offset calculation in 64 bit, and
> check for overflow just as we do here. Not sure whether that's
> feasible. And as you so eloquently describe, we can't rely on users
> listening.)

I think that would be up to upstream developers to decide. That would be 
AmigaOS developers and/or MorphOS, AROS developers. I could try to at 
least point them to this discussion here. Whether they choose to take 
the time to look at it? I don´t know. They are developing that stuff in 
their spare time.

> Either way, we need the cooperation of partitioning tool writers to
> ensure partition information that is prone to overflows is never
> stored in the 32 bit, classic RDB. That appears to have failed
> already, as Martin's experience illustrates.

Heh :)

But yeah, I´d say the damage is done already.

> The raw, theoretical limit on the maximum device capacity is about
> 2^105 bytes:
> 
> 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> bytes/sector for the HD size in struct RigidDiskBlock\0

http://wiki.amigaos.net/wiki/RDB

Of course that only holds true for as long as >32 bit math is used :). 
So yeah.

I wonder how many Amiga users may try to use such a large disk with a 
native operating system that does not support this. NSD64 and TD64 are 
at least 10 years old, if not older. Just see the dates in:

http://aminet.net/search?query=nsdpatch

http://aminet.net/package/disk/misc/td64patch

Forget about 10 years. 20 years are more accurate. This stuff is 
*ancient*. Officially support is in AmigaOS since version 3.5, which has 
been released more than 20 years ago as well:

https://en.wikipedia.org/wiki/AmigaOS_version_history#AmigaOS_3.5,_3.9

Even AmigaOS 4.0 is older than 10 years already.

In any case, it is the responsibility of the tool that creates or change 
the RDB to take care of warning the user or using a new format.

Here the kernel just reads that which already exists in the wild.

> I'm only concerned with fixing the (dangerous) but in the Linux
> partition format parser for RDB. Refusing to use any partitions that
> will cause havoc on old AmigaOS versions is all I can do to try and
> get the users' attention.

Exactly.

And I do think, that no tool on Linux can create these kind of RDBs, and 
if they can… a big fat warning belongs into that tool.

Not even Media Toolbox warns about that currently, so yes… adding an 
additional warning in the kernel may be helpful.

However without refusing to parse this, the user may never notice the 
warning. So that is an argument for refusing to parse this without a 
kernel option.

I still think that is too much hand-holding for the few native OS users 
out there. Cause in order to see the warning, the user would have to 
stuff the disk into a Linux machine anyway. So the user is still 
perfectly able to create such an RDB on AmigaOS 4.x or even AmigaOS 
3.5/3.9 or even earlier, and then stuff the disk into a machine with 
AmigaOS 1.3 and probably let it go boom there. Now, how many users who 
do not know about these limits will ever put the disk into a Linux 
machine, receive the warning and then be saved from data destruction? I 
bet: None. It is just as unlikely as it can get. :) We are talking about 
maybe a few thousand Amiga users here. And there would be even the 
question how many of them will try to use disks larger than 2 TiB. I bet 
there will be some, but I bet there won´t be many.

So it would be way more important to warn in Media Toolbox, amiga-fdisk, 
parted and whatever other partitioning tool users may use.

I´d still leave in the warning anyway, but I think the kernel option… is 
over-protecting users.

Anyway your call and I perfectly get it, in case you choose to be on a 
100% safe side. You are submitting the patch.

> Your warning makes me wonder whether the log message should just say
> 'report this bug to linux-m68k@vger.kernel.org' to at least try and
> educate any that respond about the dangers of their partitioning
> scheme before telling them about the override option. Problem with
> that is, in about three years no one will remember any of this ...
> 
> Cheers,
> 
> 	Michael
> 
> Am 28.06.2018 um 15:44 schrieb jdow:
> > Michael, as long as m68k only supports int fseek( int ) 4 GB * block
> > size is your HARD limit. Versions that support __int64 fseek64(
> > __int64 ) can work with larger disks. RDBs could include RDSK and a
> > new set of other blocks that replace the last two characters with
> > "64" and use __int64 where needed in the various values. That way a
> > clever disk partitioner could give allow normal (32 bit) RDB
> > definitions where possible. Then at least SOME of the disk could be
> > supported AND a very clever filesystem that abstracts very large
> > disks properly could give access to the whole disk. (Read the RDBs
> > first 32 bits. Then if a filesystem or driveinit was loaded re-read
> > the RDBs to see if new 64 bit partitions are revealed.
> > 
> > I could be wrong but I do not think RDBs could be safely modified
> > any
> > other way to work. And, trust me as I bet this is still true, you
> > will need a SERIOUSLY good bomb shelter on the Moon if you change
> > RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads
> > Amigados 2.4 because he wants to run a game that crashes on
> > anything newer. Then Joe got far enough something writes to the
> > disk and data is corrupted. Note further that Amigoids do NOT,
> > repeat NOT, listen to facts in such cases. Hell, some of them never
> > listened to facts about an incident at Jerry Pournelle's place when
> > a 1.1 DPaint session with Kelly Freas hung and we lost a delightful
> > drawing. Jerry reported it. Amigoids screamed. I tried to tell them
> > I was there, it was my machine, and 1.1 was, indeed, crap.
> > 
> > {o.o}
> > 
> > On 20180627 02:00, Michael Schmitz wrote:
> >> Joanne,
> >> 
> >> I'm not at all allergic to avoiding RDB at all cost for new disks.
> >> If
> >> AmigaOS 4.1 supports more recent partition formats, all the better.
> >> This is all about supporting use of legacy RDB disks on Linux
> >> (though
> >> 2 TB does stretch the definition of 'legacy' a little). My interest
> >> in this is to ensure we can continue to use RDB format disks on
> >> m68k Amiga computers which have no other way to boot Linux from
> >> disk.
> >> 
> >> Not proposing to change the RDB format at all, either. Just trying
> >> to
> >> make sure we translate RDB info into Linux 512-byte block offset
> >> and
> >> size numbers correctly. The kernel won't modify the RDB at all
> >> (intentionally, that is - with the correct choice of partition
> >> sizes,
> >> Martin might well have wiped out his RDB with the current version
> >> of
> >> the parser).
> >> 
> >> The choice of refusing to mount a disk (or mounting read-only)
> >> rests
> >> with the VFS drivers alone - AFFS in that case. Not touching any of
> >> that. At partition scan time, we only have the option of making the
> >> partition available (with a warning printed), or refusing to make
> >> it
> >> available to the kernel. Once it's made available, all bets are
> >> off.
> >> 
> >>  From what Martin writes, his test case RDB was valid and worked as
> >> 
> >> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
> >> necessary extensions to handle the large offsets resulting from 2
> >> TB
> >> disks. Not sure what safeguards are in place when connecting such a
> >> disk to older versions of AmigaOS, but that is a different matter
> >> entirely.
> >> 
> >> The overflows in partition offset and size are the only ones I can
> >> see in the partition parser - there is no other overflow I've
> >> identified. I just stated that in order to place a partition
> >> towards the end of a 2 TB disk, the offset calculation will
> >> overflow regardless of what combination of rdb->rdb_BlockBytes and
> >> sector addresses stored in the>> 
> >> RDB (in units of 512 byte blocks) we use:
> >>          blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
> >>          
> >>                  nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
> >>                  1 -
> >>                  
> >>                              be32_to_cpu(pb->pb_Environment[9])) *
> >>                             
> >>                             be32_to_cpu(pb->pb_Environment[3]) *
> >>                             be32_to_cpu(pb->pb_Environment[5]) *
> >>                             blksize;
> >>                  
> >>                  if (!nr_sects)
> >>                  
> >>                          continue;
> >>                  
> >>                  start_sect = be32_to_cpu(pb->pb_Environment[9]) *
> >>                  
> >>                               be32_to_cpu(pb->pb_Environment[3]) *
> >>                               be32_to_cpu(pb->pb_Environment[5]) *
> >>                               blksize;
> >> 
> >> But in the interest of avoiding any accidental use of a RDB
> >> partition
> >> where calculations currently overflow, I'll make the default
> >> behaviour to bail out (instead of using wrong offset or size as we
> >> currently do). Given the 'eat_my_RDB_disk=1' boot option, the user
> >> may proceed at their own risk (though I still can't see what harm
> >> should result from now translating a well formed v4.1 2 TB disk
> >> RDB correctly for the first time).
> >> 
> >> Whether or not Linux correctly handles AFFS filesystems larger than
> >> 1
> >> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
> >> ought to prevent accidental use of AFFS filesystems on RDB disks
> >> which I suppose is the majority of use cases.
> >> 
> >> Bugs in partitioning tools on Linux are entirely out of scope - the
> >> partitioning tools bypass the partition structure discovered by the
> >> kernel, and work straight on the raw device. No protecting against
> >> that.
> >> 
> >> If you can point out a way to cause data loss with these
> >> precautions,
> >> for a disk 2 TB or larger that was partitioned and used on a recent
> >> version or AmigaOS supporting such large disks, I'd consider
> >> omitting
> >> the 'eat_my_RDB_disk' boot option, and just bail out as the only
> >> safe
> >> option.
> >> 
> >> Cheers,
> >> 
> >>      Michael
> >> 
> >> Am 27.06.2018 um 18:24 schrieb jdow:
> >>> You allergic to using a GPT solution? It will get away from some
> >>> of the evils that RDB has inherent in it because they are also
> >>> features? (Loading a filesystem or DriveInit code from RDBs is
> >>> just asking for a nearly impossible to remove malware infection.)
> >>> Furthermore, any 32 bit system that sees an RDSK block is going
> >>> to try to translate it. If you add a new RDB format you are going
> >>> to get bizarre and probably quite destructive results from the
> >>> mistake. Fail safe is a rather good notion, methinks.
> >>> 
> >>> Personally I figure this is all rather surreal. 2TG of junk on an
> >>> Amiga system seems utterly outlandish to me. You cited another
> >>> overflow potential. There are at least three we've identified, I
> >>> believe. Are you 100% sure there are no more? The specific one
> >>> you mention of translating RDB to Linux has a proper solution in
> >>> the RDB reader. It should recover such overflow errors in the RDB
> >>> as it can with due care and polish. It should flag any other
> >>> overflow error it detects within the RDBs and return an error
> >>> such as to leave the disk unmounted or mounted read-only if you
> >>> feel like messing up a poor sod's backups. The simple solution is
> >>> to read each of the variables with the nominal RDB size and
> >>> convert it to uint64_t before calculating byte indices.
> >>> 
> >>> However, consider my inputs as advice from an adult who has seen
> >>> the
> >>> Amiga Elephant so to speak. I am not trying to assert any control.
> >>> Do as you wish; but, I would plead with you to avoid ANY chance
> >>> you can for the user to make a bonehead stupid move and lose all
> >>> his treasured disk archives. Doing otherwise is very poor form.
> >>> 
> >>> {o.o}   Joanne "Said enough, she has" Dow
> >>> 
> >>> On 20180626 18:07, Michael Schmitz wrote:
> >>>> Joanne,
> >>>> 
> >>>> As far as I have been able to test, the change is backwards
> >>>> compatible (RDB partitions from an old disk 80 GB disk are still
> >>>> recognized OK). That"s only been done on an emulator though.
> >>>> 
> >>>> Your advice about the dangers of using RDB disks that would have
> >>>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>>> well
> >>>> taken though. Maybe Martin can clarify that for me - was the 2 TB
> >>>> disk in question ever used on a 32 bit Amiga system?
> >>>> 
> >>>> RDB disk format is meant for legacy use only, so I think we can
> >>>> get
> >>>> away with printing a big fat warning during boot, advising the
> >>>> user
> >>>> that the oversize RDB partition(s) scanned are not compatible
> >>>> with
> >>>> legacy 32 bit AmigaOS. With the proposed fix they will work under
> >>>> both AmigaOS 4.1 and Linux instead of truncating the first
> >>>> overflowing partition at disk end and trashing valid partitions
> >>>> that overlap, which is what Martin was after.
> >>>> 
> >>>> If that still seems too risky, we can make the default behaviour
> >>>> to
> >>>> bail out once a potential overflow is detected, and allow the
> >>>> user to
> >>>> override that through a boot parameter. I'd leave that decision
> >>>> up for the code review on linux-block.
> >>>> 
> >>>> Two more comments: Linux uses 512 byte block sizes for the
> >>>> partition
> >>>> start and size calculations, so a change of the RDB blocksize to
> >>>> reduce the block counts stored in the RDB would still result in
> >>>> the
> >>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
> >>>> updating (along with probably most legacy m68k partitioners).
> >>>> Adrian
> >>>> has advertised parted as replacement for the old tools - maybe
> >>>> this
> >>>> would make a nice test case for parted?
> >>>> 
> >>>> Cheers,
> >>>> 
> >>>>    Michael
> >>>> 
> >>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> >>>>> If it is not backwards compatible I for one would refuse to use
> >>>>> it.
> >>>>> And if
> >>>>> it still mattered that much to me I'd also generate a reasonable
> >>>>> alternative. Modifying RDBs nay not be even an approximation of
> >>>>> a
> >>>>> good idea.
> >>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by
> >>>>> a
> >>>>> uint32_t
> >>>>> only system. If it is for your personal use then you're entirely
> >>>>> free to
> >>>>> reject my advice and are probably smart enough to keep it
> >>>>> working for
> >>>>> yourself.
> >>>>> 
> >>>>> GPT is probably the right way to go. Preserve the ability to
> >>>>> read
> >>>>> RDBs for
> >>>>> legacy disks only.
> >>>>> 
> >>>>> {^_^}
> >>>>> 
> >>>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>>> Joanne,
> >>>>>> 
> >>>>>> I think we all agree that doing 32 bit calculations on 512-byte
> >>>>>> block
> >>>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>>> causing
> >>>>>> the
> >>>>>> issues Martin reported. Your patch addresses that by using the
> >>>>>> correct
> >>>>>> data type for the calculations (as do other partition parsers
> >>>>>> that
> >>>>>> may
> >>>>>> have to deal with large disks) and fixes Martin's bug, so
> >>>>>> appears
> >>>>>> to be
> >>>>>> the right thing to do.
> >>>>>> 
> >>>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>>> calculations
> >>>>>> don't currently overflow is not expected to cause new issues,
> >>>>>> other
> >>>>>> than
> >>>>>> enabling use of disk and partitions larger than 2 TB (which may
> >>>>>> have
> >>>>>> ramifications with filesystems on these partitions). So
> >>>>>> comptibility is
> >>>>>> preserved.
> >>>>>> 
> >>>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>>> overflow
> >>>>>> issues in filesystems as well, but I can't see how the block
> >>>>>> size
> >>>>>> stored
> >>>>>> in the RDB would enforce use of the same block size in
> >>>>>> filesystems.
> >>>>>> We'll have to rely on the filesystem tools to get that right,
> >>>>>> too.
> >>>>>> Linux
> >>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this
> >>>>>> should
> >>>>>> allow partitions larger than 2 TB to work already (but I
> >>>>>> suspect Al
> >>>>>> Viro
> >>>>>> may have found a few issues when he looked at the AFFS code so
> >>>>>> I
> >>>>>> won't
> >>>>>> say more). Anyway partitioning tools and filesystems are
> >>>>>> unrelated to
> >>>>>> the Linux partition parser code which is all we aim to fix in
> >>>>>> this
> >>>>>> patch.
> >>>>>> 
> >>>>>> If you feel strongly about unknown ramifications of any
> >>>>>> filesystems on
> >>>>>> partitions larger than 2 TB, say so and I'll have the kernel
> >>>>>> print a
> >>>>>> warning about these partitions.
> >>>>>> 
> >>>>>> I'll get this patch tested on Martin's test case image as well
> >>>>>> as
> >>>>>> on a
> >>>>>> RDB image from a disk known to currently work under Linux
> >>>>>> (thanks
> >>>>>> Geert
> >>>>>> for the losetup hint). Can't do much more without procuring a
> >>>>>> working
> >>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I
> >>>>>> plan to
> >>>>>> use
> >>>>>> for tests is a long way away from my home indeed.
> >>>>>> 
> >>>>>> Cheers,
> >>>>>> 
> >>>>>>       Michael
> >>>>>> 
> >>>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>>> suppose.
> >>>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>>> force the
> >>>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>>> systems
> >>>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>>> blocks
> >>>>>>> list will have to go beyond a uint32_t size, for example. But
> >>>>>>> a
> >>>>>>> block
> >>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>>> tad
> >>>>>>> under
> >>>>>>> 1% of the disk all by itself. A block bitmap is not quite so
> >>>>>>> bad.
> >>>>>>> {^_-}
> >>>>>>> 
> >>>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>>> make a
> >>>>>>> change. I remember thinking about this for awhile and then
> >>>>>>> determining
> >>>>>>> I REALLY did not want to think about it as my brain was
> >>>>>>> getting tied
> >>>>>>> into a gordian knot.
> >>>>>>> 
> >>>>>>> {^_^}
> >>>>>>> 
> >>>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>>> Joanne,
> >>>>>>>> 
> >>>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>> 
> >>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK
> >>>>>>>> (512)
> >>>>>>>> sdb1
> >>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>>> (DOS^C)(res
> >>>>>>>> 2 spb
> >>>>>>>> 4)
> >>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>>> [sdb]
> >>>>>>>> Attached SCSI disk
> >>>>>>>> 
> >>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>>> means
> >>>>>>>> 512
> >>>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>>> block
> >>>>>>>> size.
> >>>>>>>> 
> >>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>>> units.
> >>>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>>> others
> >>>>>>>> yet.
> >>>>>>>> 
> >>>>>>>> Cheers,
> >>>>>>>> 
> >>>>>>>>      Michael
> >>>>>>>> 
> >>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> 
wrote:
> >>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>>> system is
> >>>>>>>>> a famn
> >>>>>>>>> dool.
> >>>>>>>>> 
> >>>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>>> so it
> >>>>>>>>> should
> >>>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>>> blocks is
> >>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >>>>>>>>> disk in
> >>>>>>>>> block maps.
> >>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>> 
> >>>>>>>>> {^_^}
> >>>>>>>>> 
> >>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>>> Hi.
> >>>>>>>>>> 
> >>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>>> test results at
> >>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>>> there,
> >>>>>>>>>>> so if
> >>>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>> 
> >>>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>> 
> >>>>>>>>>> I do not care enough about this, in order to motivate
> >>>>>>>>>> myself
> >>>>>>>>>> preparing
> >>>>>>>>>> the a patch from Joanne Dow´s fix.
> >>>>>>>>>> 
> >>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>>> Sam440ep
> >>>>>>>>>> which
> >>>>>>>>>> I still have in my apartment.
> >>>>>>>>>> 
> >>>>>>>>>> So RDB support in Linux it remains broken for disks larger
> >>>>>>>>>> 2 TB,
> >>>>>>>>>> unless
> >>>>>>>>>> someone else does.
> >>>>>>>>>> 
> >>>>>>>>>> Thanks.
> >>>>>>>>> 
> >>>>>>>>> --
> >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>>>> linux-m68k" in
> >>>>>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>>>>> More majordomo info at 
> >>>>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>>>> 
> >>>>>>> --
> >>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>> linux-m68k" in
> >>>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>>> More majordomo info at 
> >>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>> 
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>> linux-m68k" in
> >>>>> the body of a message to majordomo@vger.kernel.org
> >>>>> More majordomo info at 
> >>>>> http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)
  2018-06-28  6:39                                 ` jdow
@ 2018-06-28  8:16                                   ` Martin Steigerwald
  2018-06-28 10:00                                     ` Amiga RDB partition support for disks >= 2 TB jdow
  0 siblings, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28  8:16 UTC (permalink / raw)
  To: jdow
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Dear Joanne.

jdow - 28.06.18, 08:39:
> Anything done to RDBs for Linux must remain 100.000% compatible with
> existing Amiga equipment. Otherwise, what's the point of bothering to
> use RDBs?

Done to, in the sense of written to: Yes. I completely agree. But that 
is for amiga-fdisk and parted. And for partitioning tools on native OS.

[…]
> That brings to the fore an interesting question. Why bother with RDBs
> over 2TB unless you want a disk with one single partition? This Win10
> monster I am using has a modest BIOS driver partition for the OS and
> a giant data partition. That smaller partition would easily work with
> any RDB/Filesystem combination since 2.0. So there are some good
> workarounds that are probably "safer" and at least as flexible as
> RDBs, one Linux has used for a very long time, too.

Well, my use case was simple:

I had this 2 TB disk and I choose to share it as a backup disk for Linux 
*and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

As AmigaOS up to my knowledge does not have GPT support and MBR with 2 
TB disk is asking for even more trouble and is not supported in any 
sensible partitioning tool on AmigaOS and I choose to use Media Toolbox, 
I went with RDB as I thought it is nicely supported in Linux, which it 
is, apart from the overflowing issue.

So I did it this way.

> As I have said, for the RDB parser fix the famndool thing. Do fix it
> right in such a manner that if somebody compiles it against a version
> with no 64 bit device code it will throw a proper overflow error and
> protect the user. Users are dumb. We like to think of ourselves as

Sure, if the Linux kernel can´t handle it due to missing CONFIG_LBDAF or 
so… by all means *bail* out. Without even a kernel option to parse this 
anyway. No overflowing. Period. That is what I wrote from the beginning.

> smart. Let's try to be smart about this where we can so fingers can't
> point back at us rather than the fool that made some other error.

I completely agree.

> And do remember, I am merely (and vociferously) advising rather than
> dictating. You don't need my approval to proceed. I may want my name
> noted as an early contributor only. Meanwhile I spit out ideas as
> they come to me. One or more of them might be good. And offering
> alternatives is better than simply saying "No" most of the time.
> 
> If people are using RDBs for TB level disks I doubt they can remember
> which is the left shoe when they are getting dressed in the morning
> before going out in the yard to beat some dead horses. Or else maybe

Heh, I remembered my shoes back then. And I informed myself about what I 
was doing.

> they just want to see how far they can flog the m68k architecture as
> a mental challenge. In that case, taking it too seriously could hurt.

And well, yes… I wanted to see how far I can get away with it :)

> Note that I am mostly ignoring m68k Linux. People using that are hard
> core. People using x86/x64 Linux aren't such hard core folks. And I
> bet most of them want to read the disks so they can copy stuff to
> Amiga Forever or WinUAE running on other architectures. So TB is not
> likely to be an issue for them, either.

Yes.

Print a warning and be done with it in the RDB parser code.

Put a big fat warning into amiga-fdisk and parted!

Thanks,
Martin

> {^_^}
> 
> On 20180627 22:43, Michael Schmitz wrote:
> > Joanne,
> > 
> > Linux on m68k has supported lseek64 (or llseek) for a long time
> > (from glibc version 2.1 according to what I found). About the only
> > area where we are limited by 32 bits is the virtual memory size.
> > 
> > I'm not proposing to modify the RDB format definition, though an
> > extension to store 64 bit offsets separate from the 32 bit ones
> > would be one way to make certain such disks are safe to use on 3.1
> > and earlier versions of AmigaOS. (Another one would be to modify
> > the disk drivers on older versions to do the offset calculation in
> > 64 bit, and check for overflow just as we do here. Not sure whether
> > that's feasible. And as you so eloquently describe, we can't rely
> > on users listening.)
> > 
> > Either way, we need the cooperation of partitioning tool writers to
> > ensure partition information that is prone to overflows is never
> > stored in the 32 bit, classic RDB. That appears to have failed
> > already, as Martin's experience illustrates.
> > 
> > I'm only concerned with fixing the (dangerous) but in the Linux
> > partition format parser for RDB. Refusing to use any partitions
> > that will cause havoc on old AmigaOS versions is all I can do to
> > try and get the users' attention.
> > 
> > Your warning makes me wonder whether the log message should just say
> > 'report this bug to linux-m68k@vger.kernel.org' to at least try and
> > educate any that respond about the dangers of their partitioning
> > scheme before telling them about the override option. Problem with
> > that is, in about three years no one will remember any of this ...
> > 
> > Cheers,
> > 
> >      Michael
> > 
> > Am 28.06.2018 um 15:44 schrieb jdow:
> >> Michael, as long as m68k only supports int fseek( int ) 4 GB *
> >> block
> >> size is your HARD limit. Versions that support __int64 fseek64(
> >> __int64 ) can work with larger disks. RDBs could include RDSK and
> >> a new set of other blocks that replace the last two characters
> >> with "64" and use __int64 where needed in the various values. That
> >> way a clever disk partitioner could give allow normal (32 bit) RDB
> >> definitions where possible. Then at least SOME of the disk could
> >> be supported AND a very clever filesystem that abstracts very
> >> large disks properly could give access to the whole disk. (Read
> >> the RDBs first 32 bits. Then if a filesystem or driveinit was
> >> loaded re-read the RDBs to see if new 64 bit partitions are
> >> revealed.
> >> 
> >> I could be wrong but I do not think RDBs could be safely modified
> >> any
> >> other way to work. And, trust me as I bet this is still true, you
> >> will need a SERIOUSLY good bomb shelter on the Moon if you change
> >> RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads
> >> Amigados 2.4 because he wants to run a game that crashes on
> >> anything newer. Then Joe got far enough something writes to the
> >> disk and data is corrupted. Note further that Amigoids do NOT,
> >> repeat NOT, listen to facts in such cases. Hell, some of them
> >> never listened to facts about an incident at Jerry Pournelle's
> >> place when a 1.1 DPaint session with Kelly Freas hung and we lost
> >> a delightful drawing. Jerry reported it. Amigoids screamed. I
> >> tried to tell them I was there, it was my machine, and 1.1 was,
> >> indeed, crap.
> >> 
> >> {o.o}
> >> 
> >> On 20180627 02:00, Michael Schmitz wrote:
> >>> Joanne,
> >>> 
> >>> I'm not at all allergic to avoiding RDB at all cost for new disks.
> >>> If
> >>> AmigaOS 4.1 supports more recent partition formats, all the
> >>> better.
> >>> This is all about supporting use of legacy RDB disks on Linux
> >>> (though
> >>> 2 TB does stretch the definition of 'legacy' a little). My
> >>> interest in this is to ensure we can continue to use RDB format
> >>> disks on m68k Amiga computers which have no other way to boot
> >>> Linux from disk.
> >>> 
> >>> Not proposing to change the RDB format at all, either. Just trying
> >>> to
> >>> make sure we translate RDB info into Linux 512-byte block offset
> >>> and
> >>> size numbers correctly. The kernel won't modify the RDB at all
> >>> (intentionally, that is - with the correct choice of partition
> >>> sizes,
> >>> Martin might well have wiped out his RDB with the current version
> >>> of
> >>> the parser).
> >>> 
> >>> The choice of refusing to mount a disk (or mounting read-only)
> >>> rests
> >>> with the VFS drivers alone - AFFS in that case. Not touching any
> >>> of
> >>> that. At partition scan time, we only have the option of making
> >>> the
> >>> partition available (with a warning printed), or refusing to make
> >>> it
> >>> available to the kernel. Once it's made available, all bets are
> >>> off.
> >>> 
> >>>  From what Martin writes, his test case RDB was valid and worked
> >>> as
> >>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
> >>> necessary extensions to handle the large offsets resulting from 2
> >>> TB
> >>> disks. Not sure what safeguards are in place when connecting such
> >>> a
> >>> disk to older versions of AmigaOS, but that is a different matter
> >>> entirely.
> >>> 
> >>> The overflows in partition offset and size are the only ones I can
> >>> see in the partition parser - there is no other overflow I've
> >>> identified. I just stated that in order to place a partition
> >>> towards the end of a 2 TB disk, the offset calculation will
> >>> overflow regardless of what combination of rdb->rdb_BlockBytes
> >>> and sector addresses stored in the RDB (in units of 512 byte
> >>> blocks) we use:
> >>> 
> >>>          blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
> >>> 
> >>> 
> >>>                  nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
> >>> 1 - be32_to_cpu(pb->pb_Environment[9])) *
> >>> be32_to_cpu(pb->pb_Environment[3]) *
> >>> be32_to_cpu(pb->pb_Environment[5]) * blksize;
> >>>                  if (!nr_sects)
> >>>                          continue;
> >>>                  start_sect = be32_to_cpu(pb->pb_Environment[9]) *
> >>>                               be32_to_cpu(pb->pb_Environment[3]) *
> >>>                               be32_to_cpu(pb->pb_Environment[5]) *
> >>>                               blksize;
> >>> 
> >>> But in the interest of avoiding any accidental use of a RDB
> >>> partition
> >>> where calculations currently overflow, I'll make the default
> >>> behaviour to bail out (instead of using wrong offset or size as
> >>> we currently do). Given the 'eat_my_RDB_disk=1' boot option, the
> >>> user may proceed at their own risk (though I still can't see what
> >>> harm should result from now translating a well formed v4.1 2 TB
> >>> disk RDB correctly for the first time).
> >>> 
> >>> Whether or not Linux correctly handles AFFS filesystems larger
> >>> than 1
> >>> TB is a matter for VFS experts. Bailing out on nr_sects
> >>> overflowing
> >>> ought to prevent accidental use of AFFS filesystems on RDB disks
> >>> which I suppose is the majority of use cases.
> >>> 
> >>> Bugs in partitioning tools on Linux are entirely out of scope -
> >>> the
> >>> partitioning tools bypass the partition structure discovered by
> >>> the
> >>> kernel, and work straight on the raw device. No protecting against
> >>> that.
> >>> 
> >>> If you can point out a way to cause data loss with these
> >>> precautions,
> >>> for a disk 2 TB or larger that was partitioned and used on a
> >>> recent
> >>> version or AmigaOS supporting such large disks, I'd consider
> >>> omitting
> >>> the 'eat_my_RDB_disk' boot option, and just bail out as the only
> >>> safe
> >>> option.
> >>> 
> >>> Cheers,
> >>> 
> >>>      Michael
> >>> 
> >>> Am 27.06.2018 um 18:24 schrieb jdow:
> >>>> You allergic to using a GPT solution? It will get away from some
> >>>> of the evils that RDB has inherent in it because they are also
> >>>> features? (Loading a filesystem or DriveInit code from RDBs is
> >>>> just asking for a nearly impossible to remove malware
> >>>> infection.) Furthermore, any 32 bit system that sees an RDSK
> >>>> block is going to try to translate it. If you add a new RDB
> >>>> format you are going to get bizarre and probably quite
> >>>> destructive results from the mistake. Fail safe is a rather good
> >>>> notion, methinks.
> >>>> 
> >>>> Personally I figure this is all rather surreal. 2TG of junk on an
> >>>> Amiga system seems utterly outlandish to me. You cited another
> >>>> overflow potential. There are at least three we've identified, I
> >>>> believe. Are you 100% sure there are no more? The specific one
> >>>> you mention of translating RDB to Linux has a proper solution in
> >>>> the RDB reader. It should recover such overflow errors in the
> >>>> RDB as it can with due care and polish. It should flag any other
> >>>> overflow error it detects within the RDBs and return an error
> >>>> such as to leave the disk unmounted or mounted read-only if you
> >>>> feel like messing up a poor sod's backups. The simple solution
> >>>> is to read each of the variables with the nominal RDB size and
> >>>> convert it to uint64_t before calculating byte indices.
> >>>> 
> >>>> However, consider my inputs as advice from an adult who has seen
> >>>> the
> >>>> Amiga Elephant so to speak. I am not trying to assert any
> >>>> control. Do as you wish; but, I would plead with you to avoid
> >>>> ANY chance you can for the user to make a bonehead stupid move
> >>>> and lose all his treasured disk archives. Doing otherwise is
> >>>> very poor form.
> >>>> 
> >>>> {o.o}   Joanne "Said enough, she has" Dow
> >>>> 
> >>>> On 20180626 18:07, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>> 
> >>>>> As far as I have been able to test, the change is backwards
> >>>>> compatible (RDB partitions from an old disk 80 GB disk are
> >>>>> still recognized OK). That"s only been done on an emulator
> >>>>> though.
> >>>>> 
> >>>>> Your advice about the dangers of using RDB disks that would have
> >>>>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>>>> well
> >>>>> taken though. Maybe Martin can clarify that for me - was the 2
> >>>>> TB disk in question ever used on a 32 bit Amiga system?
> >>>>> 
> >>>>> RDB disk format is meant for legacy use only, so I think we can
> >>>>> get
> >>>>> away with printing a big fat warning during boot, advising the
> >>>>> user
> >>>>> that the oversize RDB partition(s) scanned are not compatible
> >>>>> with
> >>>>> legacy 32 bit AmigaOS. With the proposed fix they will work
> >>>>> under both AmigaOS 4.1 and Linux instead of truncating the
> >>>>> first overflowing partition at disk end and trashing valid
> >>>>> partitions that overlap, which is what Martin was after.
> >>>>> 
> >>>>> If that still seems too risky, we can make the default behaviour
> >>>>> to
> >>>>> bail out once a potential overflow is detected, and allow the
> >>>>> user to
> >>>>> override that through a boot parameter. I'd leave that decision
> >>>>> up for the code review on linux-block.
> >>>>> 
> >>>>> Two more comments: Linux uses 512 byte block sizes for the
> >>>>> partition
> >>>>> start and size calculations, so a change of the RDB blocksize to
> >>>>> reduce the block counts stored in the RDB would still result in
> >>>>> the
> >>>>> same overflow. And amiga-fdisk is indeed utterly broken and
> >>>>> needs
> >>>>> updating (along with probably most legacy m68k partitioners).
> >>>>> Adrian
> >>>>> has advertised parted as replacement for the old tools - maybe
> >>>>> this
> >>>>> would make a nice test case for parted?
> >>>>> 
> >>>>> Cheers,
> >>>>> 
> >>>>>    Michael
> >>>>> 
> >>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> 
wrote:
> >>>>>> If it is not backwards compatible I for one would refuse to use
> >>>>>> it.
> >>>>>> And if
> >>>>>> it still mattered that much to me I'd also generate a
> >>>>>> reasonable
> >>>>>> alternative. Modifying RDBs nay not be even an approximation of
> >>>>>> a
> >>>>>> good idea.
> >>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted
> >>>>>> by a
> >>>>>> uint32_t
> >>>>>> only system. If it is for your personal use then you're
> >>>>>> entirely
> >>>>>> free to
> >>>>>> reject my advice and are probably smart enough to keep it
> >>>>>> working for
> >>>>>> yourself.
> >>>>>> 
> >>>>>> GPT is probably the right way to go. Preserve the ability to
> >>>>>> read
> >>>>>> RDBs for
> >>>>>> legacy disks only.
> >>>>>> 
> >>>>>> {^_^}
> >>>>>> 
> >>>>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>>>> Joanne,
> >>>>>>> 
> >>>>>>> I think we all agree that doing 32 bit calculations on
> >>>>>>> 512-byte block
> >>>>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>>>> causing
> >>>>>>> the
> >>>>>>> issues Martin reported. Your patch addresses that by using the
> >>>>>>> correct
> >>>>>>> data type for the calculations (as do other partition parsers
> >>>>>>> that
> >>>>>>> may
> >>>>>>> have to deal with large disks) and fixes Martin's bug, so
> >>>>>>> appears
> >>>>>>> to be
> >>>>>>> the right thing to do.
> >>>>>>> 
> >>>>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>>>> calculations
> >>>>>>> don't currently overflow is not expected to cause new issues,
> >>>>>>> other
> >>>>>>> than
> >>>>>>> enabling use of disk and partitions larger than 2 TB (which
> >>>>>>> may have
> >>>>>>> ramifications with filesystems on these partitions). So
> >>>>>>> comptibility is
> >>>>>>> preserved.
> >>>>>>> 
> >>>>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>>>> overflow
> >>>>>>> issues in filesystems as well, but I can't see how the block
> >>>>>>> size
> >>>>>>> stored
> >>>>>>> in the RDB would enforce use of the same block size in
> >>>>>>> filesystems.
> >>>>>>> We'll have to rely on the filesystem tools to get that right,
> >>>>>>> too.
> >>>>>>> Linux
> >>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this
> >>>>>>> should
> >>>>>>> allow partitions larger than 2 TB to work already (but I
> >>>>>>> suspect Al
> >>>>>>> Viro
> >>>>>>> may have found a few issues when he looked at the AFFS code so
> >>>>>>> I
> >>>>>>> won't
> >>>>>>> say more). Anyway partitioning tools and filesystems are
> >>>>>>> unrelated to
> >>>>>>> the Linux partition parser code which is all we aim to fix in
> >>>>>>> this
> >>>>>>> patch.
> >>>>>>> 
> >>>>>>> If you feel strongly about unknown ramifications of any
> >>>>>>> filesystems on
> >>>>>>> partitions larger than 2 TB, say so and I'll have the kernel
> >>>>>>> print a
> >>>>>>> warning about these partitions.
> >>>>>>> 
> >>>>>>> I'll get this patch tested on Martin's test case image as well
> >>>>>>> as
> >>>>>>> on a
> >>>>>>> RDB image from a disk known to currently work under Linux
> >>>>>>> (thanks
> >>>>>>> Geert
> >>>>>>> for the losetup hint). Can't do much more without procuring a
> >>>>>>> working
> >>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I
> >>>>>>> plan to
> >>>>>>> use
> >>>>>>> for tests is a long way away from my home indeed.
> >>>>>>> 
> >>>>>>> Cheers,
> >>>>>>> 
> >>>>>>>       Michael
> >>>>>>> 
> >>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>>>> suppose.
> >>>>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>>>> force the
> >>>>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>>>> systems
> >>>>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>>>> blocks
> >>>>>>>> list will have to go beyond a uint32_t size, for example. But
> >>>>>>>> a
> >>>>>>>> block
> >>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>>>> tad
> >>>>>>>> under
> >>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so
> >>>>>>>> bad.
> >>>>>>>> {^_-}
> >>>>>>>> 
> >>>>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>>>> make a
> >>>>>>>> change. I remember thinking about this for awhile and then
> >>>>>>>> determining
> >>>>>>>> I REALLY did not want to think about it as my brain was
> >>>>>>>> getting tied
> >>>>>>>> into a gordian knot.
> >>>>>>>> 
> >>>>>>>> {^_^}
> >>>>>>>> 
> >>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>>>> Joanne,
> >>>>>>>>> 
> >>>>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>>> 
> >>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK
> >>>>>>>>> (512)
> >>>>>>>>> sdb1
> >>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>>>> (DOS^C)(res
> >>>>>>>>> 2 spb
> >>>>>>>>> 4)
> >>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>>>> [sdb]
> >>>>>>>>> Attached SCSI disk
> >>>>>>>>> 
> >>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>>>> means
> >>>>>>>>> 512
> >>>>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>>>> block
> >>>>>>>>> size.
> >>>>>>>>> 
> >>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>>>> units.
> >>>>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>>>> others
> >>>>>>>>> yet.
> >>>>>>>>> 
> >>>>>>>>> Cheers,
> >>>>>>>>> 
> >>>>>>>>>      Michael
> >>>>>>>>> 
> >>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> 
wrote:
> >>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>>>> system is
> >>>>>>>>>> a famn
> >>>>>>>>>> dool.
> >>>>>>>>>> 
> >>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>>>> so it
> >>>>>>>>>> should
> >>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>>>> blocks is
> >>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >>>>>>>>>> disk in
> >>>>>>>>>> block maps.
> >>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>>> 
> >>>>>>>>>> {^_^}
> >>>>>>>>>> 
> >>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>>>> Hi.
> >>>>>>>>>>> 
> >>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>>>> test results at
> >>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>>>> there,
> >>>>>>>>>>>> so if
> >>>>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>>> 
> >>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>>> 
> >>>>>>>>>>> I do not care enough about this, in order to motivate
> >>>>>>>>>>> myself
> >>>>>>>>>>> preparing
> >>>>>>>>>>> the a patch from Joanne Dow´s fix.
> >>>>>>>>>>> 
> >>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>>>> Sam440ep
> >>>>>>>>>>> which
> >>>>>>>>>>> I still have in my apartment.
> >>>>>>>>>>> 
> >>>>>>>>>>> So RDB support in Linux it remains broken for disks larger
> >>>>>>>>>>> 2 TB,
> >>>>>>>>>>> unless
> >>>>>>>>>>> someone else does.
> >>>>>>>>>>> 
> >>>>>>>>>>> Thanks.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Amiga RDB partition support for disks >= 2 TB
  2018-06-28  8:16                                   ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
@ 2018-06-28 10:00                                     ` jdow
  2018-06-28 11:30                                       ` Martin Steigerwald
  0 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-28 10:00 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k



On 20180628 01:16, Martin Steigerwald wrote:
> Dear Joanne.
> 
> jdow - 28.06.18, 08:39:
>> Anything done to RDBs for Linux must remain 100.000% compatible with
>> existing Amiga equipment. Otherwise, what's the point of bothering to
>> use RDBs?
> 
> Done to, in the sense of written to: Yes. I completely agree. But that
> is for amiga-fdisk and parted. And for partitioning tools on native OS.

Design changes, too.

> […]
>> That brings to the fore an interesting question. Why bother with RDBs
>> over 2TB unless you want a disk with one single partition? This Win10
>> monster I am using has a modest BIOS driver partition for the OS and
>> a giant data partition. That smaller partition would easily work with
>> any RDB/Filesystem combination since 2.0. So there are some good
>> workarounds that are probably "safer" and at least as flexible as
>> RDBs, one Linux has used for a very long time, too.
> 
> Well, my use case was simple:
> 
> I had this 2 TB disk and I choose to share it as a backup disk for Linux
> *and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

EEEEEEK! The hair on my neck is standing up straight! Have you heard of SAMBA? 
The linux mail server firewall etc machine has an extra 4TB disk on it as a 
backup for the other systems, although a piddly 4TB is small when I save the 
entire 3G RAID system I have. It's a proof of concept so.... A full backup on a 
1gig Ethernet still takes a looooong time. But backing up even an 18GB disk on 
an Amiga via 100Base-t isn't too bad. And disk speeds of the era being what they 
were it's about all you can do anyway.

{o.o}

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Amiga RDB partition support for disks >= 2 TB
  2018-06-28 10:00                                     ` Amiga RDB partition support for disks >= 2 TB jdow
@ 2018-06-28 11:30                                       ` Martin Steigerwald
  2018-06-28 11:38                                         ` Martin Steigerwald
  0 siblings, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28 11:30 UTC (permalink / raw)
  To: jdow
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

jdow - 28.06.18, 12:00:
> On 20180628 01:16, Martin Steigerwald wrote:
[…]
> >> That brings to the fore an interesting question. Why bother with
> >> RDBs
> >> over 2TB unless you want a disk with one single partition? This
> >> Win10
> >> monster I am using has a modest BIOS driver partition for the OS
> >> and
> >> a giant data partition. That smaller partition would easily work
> >> with
> >> any RDB/Filesystem combination since 2.0. So there are some good
> >> workarounds that are probably "safer" and at least as flexible as
> >> RDBs, one Linux has used for a very long time, too.
> > 
> > Well, my use case was simple:
> > 
> > I had this 2 TB disk and I choose to share it as a backup disk for
> > Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me
> > desk here.
> EEEEEEK! The hair on my neck is standing up straight! Have you heard
> of SAMBA? The linux mail server firewall etc machine has an extra 4TB
> disk on it as a backup for the other systems, although a piddly 4TB
> is small when I save the entire 3G RAID system I have. It's a proof
> of concept so.... A full backup on a 1gig Ethernet still takes a
> looooong time. But backing up even an 18GB disk on an Amiga via
> 100Base-t isn't too bad. And disk speeds of the era being what they
> were it's about all you can do anyway.

Heh, the thing worked just fine in Amiga OS 4. I got away with it 
without an issue, until I plugged the disk to my Linux laptop and wrote 
data onto the Linux file system. Mind you, I think in that partition 
marked LNX\0 I even created a Linux LVM with pvcreate. Do you call that 
insane? Well it probably is. :)

And as an Amiga user I could just return to you: I clicked it, it did 
not warn, so all is good :)

But yeah, as mentioned I researched the topic before. And I think there 
has not even been an overflow within the RDB:

> The raw, theoretical limit on the maximum device capacity is about
> 2^105 bytes:
> 
> 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> bytes/sector for the HD size in struct RigidDiskBlock\0

http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)

Confirmed by:

The .ADF (Amiga Disk File) format FAQ:
http://lclevy.free.fr/adflib/adf_info.html#p6

But what do I write, you know the RDB format :)

So just do the calculation in 96 Bit and you all are set :)

Now that is a reason for 128 Bit CPUs :).

Muuhahaha.

Ciao,
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Amiga RDB partition support for disks >= 2 TB
  2018-06-28 11:30                                       ` Martin Steigerwald
@ 2018-06-28 11:38                                         ` Martin Steigerwald
  2018-06-28 12:31                                           ` jdow
  0 siblings, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28 11:38 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: jdow, Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

Martin Steigerwald - 28.06.18, 13:30:
> jdow - 28.06.18, 12:00:
> > On 20180628 01:16, Martin Steigerwald wrote:
> […]
> 
> > >> That brings to the fore an interesting question. Why bother with
> > >> RDBs
> > >> over 2TB unless you want a disk with one single partition? This
> > >> Win10
> > >> monster I am using has a modest BIOS driver partition for the OS
> > >> and
> > >> a giant data partition. That smaller partition would easily work
> > >> with
> > >> any RDB/Filesystem combination since 2.0. So there are some good
> > >> workarounds that are probably "safer" and at least as flexible as
> > >> RDBs, one Linux has used for a very long time, too.
> > > 
> > > Well, my use case was simple:
> > > 
> > > I had this 2 TB disk and I choose to share it as a backup disk for
> > > Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me
> > > desk here.
> > 
> > EEEEEEK! The hair on my neck is standing up straight! Have you heard
> > of SAMBA? The linux mail server firewall etc machine has an extra
> > 4TB
> > disk on it as a backup for the other systems, although a piddly 4TB
> > is small when I save the entire 3G RAID system I have. It's a proof
> > of concept so.... A full backup on a 1gig Ethernet still takes a
> > looooong time. But backing up even an 18GB disk on an Amiga via
> > 100Base-t isn't too bad. And disk speeds of the era being what they
> > were it's about all you can do anyway.
> 
> Heh, the thing worked just fine in Amiga OS 4. I got away with it
> without an issue, until I plugged the disk to my Linux laptop and
> wrote data onto the Linux file system. Mind you, I think in that
> partition marked LNX\0 I even created a Linux LVM with pvcreate. Do
> you call that insane? Well it probably is. :)
> 
> And as an Amiga user I could just return to you: I clicked it, it did
> not warn, so all is good :)
> 
> But yeah, as mentioned I researched the topic before. And I think
> there
> has not even been an overflow within the RDB:
> > The raw, theoretical limit on the maximum device capacity is about
> > 2^105 bytes:
> > 
> > 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> > bytes/sector for the HD size in struct RigidDiskBlock\0
> 
> http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)
> 
> Confirmed by:
> 
> The .ADF (Amiga Disk File) format FAQ:
> http://lclevy.free.fr/adflib/adf_info.html#p6
> 
> But what do I write, you know the RDB format :)
> 
> So just do the calculation in 96 Bit and you all are set :)

For sectors.

Or

3*32+9 (for 512 bytes per sector) = 105 bits

for bytes.

> Now that is a reason for 128 Bit CPUs :).
> 
> Muuhahaha.
-- 
Martin

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: Amiga RDB partition support for disks >= 2 TB
  2018-06-28 11:38                                         ` Martin Steigerwald
@ 2018-06-28 12:31                                           ` jdow
  0 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-28 12:31 UTC (permalink / raw)
  To: Martin Steigerwald
  Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
	David Sterba, Linux FS Devel, Linux Kernel Mailing List,
	Jens Axboe, linux-m68k

On 20180628 04:38, Martin Steigerwald wrote:
> Martin Steigerwald - 28.06.18, 13:30:
>> jdow - 28.06.18, 12:00:
>>> On 20180628 01:16, Martin Steigerwald wrote:
>> […]
>>
>>>>> That brings to the fore an interesting question. Why bother with
>>>>> RDBs
>>>>> over 2TB unless you want a disk with one single partition? This
>>>>> Win10
>>>>> monster I am using has a modest BIOS driver partition for the OS
>>>>> and
>>>>> a giant data partition. That smaller partition would easily work
>>>>> with
>>>>> any RDB/Filesystem combination since 2.0. So there are some good
>>>>> workarounds that are probably "safer" and at least as flexible as
>>>>> RDBs, one Linux has used for a very long time, too.
>>>>
>>>> Well, my use case was simple:
>>>>
>>>> I had this 2 TB disk and I choose to share it as a backup disk for
>>>> Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me
>>>> desk here.
>>>
>>> EEEEEEK! The hair on my neck is standing up straight! Have you heard
>>> of SAMBA? The linux mail server firewall etc machine has an extra
>>> 4TB
>>> disk on it as a backup for the other systems, although a piddly 4TB
>>> is small when I save the entire 3G RAID system I have. It's a proof
>>> of concept so.... A full backup on a 1gig Ethernet still takes a
>>> looooong time. But backing up even an 18GB disk on an Amiga via
>>> 100Base-t isn't too bad. And disk speeds of the era being what they
>>> were it's about all you can do anyway.
>>
>> Heh, the thing worked just fine in Amiga OS 4. I got away with it
>> without an issue, until I plugged the disk to my Linux laptop and
>> wrote data onto the Linux file system. Mind you, I think in that
>> partition marked LNX\0 I even created a Linux LVM with pvcreate. Do
>> you call that insane? Well it probably is. :)
>>
>> And as an Amiga user I could just return to you: I clicked it, it did
>> not warn, so all is good :)
>>
>> But yeah, as mentioned I researched the topic before. And I think
>> there
>> has not even been an overflow within the RDB:
>>> The raw, theoretical limit on the maximum device capacity is about
>>> 2^105 bytes:
>>>
>>> 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
>>> bytes/sector for the HD size in struct RigidDiskBlock
>>
>> http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)
>>
>> Confirmed by:
>>
>> The .ADF (Amiga Disk File) format FAQ:
>> http://lclevy.free.fr/adflib/adf_info.html#p6
>>
>> But what do I write, you know the RDB format :)
>>
>> So just do the calculation in 96 Bit and you all are set :)
> 
> For sectors.
> 
> Or
> 
> 3*32+9 (for 512 bytes per sector) = 105 bits
> 
> for bytes.
> 
>> Now that is a reason for 128 Bit CPUs :).
>>
>> Muuhahaha.

And if you make the logical block size 65536 you need 128 bits or 10^something 
big = 2^128.
{^_-}

^ permalink raw reply	[flat|nested] 73+ messages in thread

end of thread, other threads:[~2018-06-28 12:31 UTC | newest]

Thread overview: 73+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-25 15:46 Moving unmaintained filesystems to staging Matthew Wilcox
2018-04-25 15:47 ` Christoph Hellwig
2018-04-25 20:30 ` David Sterba
2018-04-26  2:57   ` Matthew Wilcox
2018-04-26 10:28     ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Martin Steigerwald
2018-04-26 10:45       ` moving affs + RDB partition support to staging? John Paul Adrian Glaubitz
2018-04-26 10:59         ` David Sterba
2018-04-26 11:06           ` John Paul Adrian Glaubitz
2018-05-06  0:59         ` Al Viro
2018-05-06  7:40           ` Al Viro
2018-05-06 20:46             ` Al Viro
2018-05-06 20:49               ` John Paul Adrian Glaubitz
2018-05-06 21:32               ` Al Viro
2018-05-07  2:15                 ` Al Viro
2018-05-07  2:40                   ` Michael Schmitz
2018-05-07  7:08                     ` Martin Steigerwald
2018-05-07 20:50                       ` Michael Schmitz
2018-05-07 20:56                         ` Ingo Jürgensmann
2018-05-07 20:58                           ` John Paul Adrian Glaubitz
2018-05-06  8:40           ` John Paul Adrian Glaubitz
2018-05-06 10:12           ` Martin Steigerwald
2018-04-26 11:00       ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Christoph Hellwig
2018-04-26 11:08       ` Geert Uytterhoeven
2018-04-26 23:56         ` Finn Thain
2018-04-27  1:43           ` moving affs + RDB partition support to staging? jdow
2018-04-27  1:26         ` jdow
2018-05-06  8:52           ` John Paul Adrian Glaubitz
2018-05-06 10:10             ` Martin Steigerwald
2018-05-07  4:54             ` jdow
2018-04-27  2:11         ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
2018-06-24  9:06           ` Martin Steigerwald
2018-06-24 11:33             ` moving affs + RDB partition support to staging? jdow
2018-06-24 11:40             ` jdow
2018-06-26  2:23               ` Michael Schmitz
2018-06-26  5:17                 ` jdow
2018-06-26  8:12                   ` Martin Steigerwald
2018-06-26  9:46                     ` jdow
2018-06-26  8:31                   ` Michael Schmitz
2018-06-26  9:45                     ` jdow
2018-06-27  1:07                       ` Michael Schmitz
2018-06-27  6:24                         ` jdow
2018-06-27  8:03                           ` Martin Steigerwald
2018-06-28  2:57                             ` jdow
2018-06-28  7:40                               ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
2018-06-27  9:00                           ` moving affs + RDB partition support to staging? Michael Schmitz
2018-06-28  3:44                             ` jdow
2018-06-28  5:43                               ` Michael Schmitz
2018-06-28  6:39                                 ` jdow
2018-06-28  8:16                                   ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
2018-06-28 10:00                                     ` Amiga RDB partition support for disks >= 2 TB jdow
2018-06-28 11:30                                       ` Martin Steigerwald
2018-06-28 11:38                                         ` Martin Steigerwald
2018-06-28 12:31                                           ` jdow
2018-06-28  8:07                                 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
2018-06-27  7:57                         ` moving affs + RDB partition support to staging? Martin Steigerwald
2018-06-28  2:56                           ` jdow
2018-06-26  8:02                 ` Martin Steigerwald
2018-06-26  8:40                   ` Michael Schmitz
2018-06-26  9:31                   ` jdow
2018-06-25  7:53             ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
2018-06-25  8:26               ` Martin Steigerwald
2018-06-25  8:40               ` Geert Uytterhoeven
2018-04-27  8:01         ` Martin Steigerwald
2018-04-26  4:58   ` Moving unmaintained filesystems to staging Nikolay Borisov
2018-04-26  5:30     ` Willy Tarreau
2018-04-26  6:11 ` Pavel Machek
2018-04-26 10:36   ` Martin Steigerwald
2018-05-03  9:18     ` Pavel Machek
2018-04-27  1:10   ` Luis R. Rodriguez
2018-04-29 12:07   ` Greg KH
2018-04-29 20:07     ` Ondrej Zary
2018-04-29 23:37       ` Greg KH
2018-05-01 10:14         ` Pavel Machek

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).