* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
2018-04-26 11:08 ` Geert Uytterhoeven
@ 2018-04-26 23:56 ` Finn Thain
2018-04-27 1:43 ` moving affs + RDB partition support to staging? jdow
2018-04-27 1:26 ` jdow
` (2 subsequent siblings)
3 siblings, 1 reply; 73+ messages in thread
From: Finn Thain @ 2018-04-26 23:56 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Martin Steigerwald, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow
On Thu, 26 Apr 2018, Geert Uytterhoeven wrote:
>
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
> handled by FUSE
Moving to FUSE is a great divide-and-conquer strategy for those who just
want the code to die and don't care about any of the data in that format.
If there is a maintainence burden that can be shared then it should be
shared -- until it can be established that there is no data of value in
that format.
> moving RDB partition support to staging is not an option, as it is the
> only partitioning scheme that Amigas can boot from.
>
Whether or not the original hardware is in use is mostly irrelevant.
As long as the old format is accessible using current hardware, the data
in that format remains accessible (to archivists, to curators, to your
decendents, etc).
> If there are bugs in the RDB parser that people run into, they should be
> fixed. If there are limitations in the RDB format on large disks, that's
> still not a reason to move it to staging (hi msdos partitioning!).
>
--
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-04-26 23:56 ` Finn Thain
@ 2018-04-27 1:43 ` jdow
0 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-04-27 1:43 UTC (permalink / raw)
To: Finn Thain, Geert Uytterhoeven
Cc: Martin Steigerwald, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k
On 20180426 16:56, Finn Thain wrote:
> On Thu, 26 Apr 2018, Geert Uytterhoeven wrote:
>
>>
>> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
>> handled by FUSE
>
> Moving to FUSE is a great divide-and-conquer strategy for those who just
> want the code to die and don't care about any of the data in that format.
>
> If there is a maintainence burden that can be shared then it should be
> shared -- until it can be established that there is no data of value in
> that format.
>
>> moving RDB partition support to staging is not an option, as it is the
>> only partitioning scheme that Amigas can boot from.
>>
>
> Whether or not the original hardware is in use is mostly irrelevant.
>
> As long as the old format is accessible using current hardware, the data
> in that format remains accessible (to archivists, to curators, to your
> decendents, etc).
>
>> If there are bugs in the RDB parser that people run into, they should be
>> fixed. If there are limitations in the RDB format on large disks, that's
>> still not a reason to move it to staging (hi msdos partitioning!).
>>
This intrepid cyberunit is inclined to suggest that understanding the RDBs can
go a long way towards defining if there is a bug somewhere and whether it is in
the RDB description or its misuse.
There are some things RDBs can do that REALLY REALLY don't make sense until you
run across the situation which called for it creation. There are two variables
that suggest some blocks at the beginning and the end, respectively, of a
partition are not accessible by the OS. I have used these facilities to
"interleave" partitions and RDBs. I have built a disk which reserved about 128
512 byte blocks for RDBs plus filesystem code (which probably should be
abandoned) which embedded the RDBs describing the partition within the
partition. Then I reserved space at the end of the partition and embedded a
second partition in that space. As absurd as it sounds this had at one time a
decent use case. Disk space was an expensive premium in those days so wasting
space to get nice integer numbers in the disk description, which was phony for a
hard disk in any case, we allowed any numbers and if that went past the end of
the disk we reserved the necessary space so that it would never be used. The
space at the beginning of a partition was needed in any case because a one block
partition signature needed space at the start of the partition. It held the
filesystem's signature, OFS, AFS, SFS, etc.
There is also a good reason for allowing the anchor for the RDBs to start in any
of the first 16 blocks with a recommendation not to use block 0 as other FSs
used that. And we wanted to accommodate at least two different partition
description technologies to work on the disk. My code always placed the RDBs at
block 3.
I hope passing along some of this history will mitigate some fo the feelings
that RDBs are inherently flawed or full of bugs or whatnot. (Full pf security
holes is another story. DriveInit code and filesystem code have worried me from
day one.)
{^_^} Joanne
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-04-26 11:08 ` Geert Uytterhoeven
2018-04-26 23:56 ` Finn Thain
@ 2018-04-27 1:26 ` jdow
2018-05-06 8:52 ` John Paul Adrian Glaubitz
2018-04-27 2:11 ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
2018-04-27 8:01 ` Martin Steigerwald
3 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-04-27 1:26 UTC (permalink / raw)
To: Geert Uytterhoeven, Martin Steigerwald
Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k
OK guys, I am working from th ememory of a 74 year old with a severe head cold.
The filesystems were always limited to 2 TB since the seek command was signed 32
bit integer number of BYTEs. So block size has nothing to do with this 2TB
issue. That's something else in the descriptions.
The RDBs grew a block size and sector size concept in describing partitions. One
had to be a power of two multiple of the other, as I recall. The relationship
worked two ways. If you had a Fujitsu Magneto Optical drive with 2k physical
sectors you could still allocate space in virtual sectors, block sizes, of 512
bytes. Or you could use much larger blocks than sector size to cut down on the
unallocated blocks list. Of course, all OSs do that these days. With a single
unsigned 16 bit integer your address space would fill up a 32 GB with 512 byte
blocks. We could see this would blow out so we decribed block size and disk size
in blocks with 32 bit integers giving 4 TB of 1 BYTE blocks. Supposing the file
sizes stored are such as to support an 8k block size that means describing a
very large disk should be very easy, even if block size was a 16 bit unsigned
integer.
To carry this forward more accurately I'll have to dig up some old archival data
and see what it actually says. This isn't going to happen when I sit here
sniffling. Maybe in a few days I can dredge up the actual numbers. I seem to
remember at the time that without a 64 bit OS the disk addressing limit was 2
TB, since seeks were signed integers. The disk descriptions could be
considerably larger.
And before I forget there are two features of the RDBs that I heartily recommend
never implementing on Linux. They were good ideas at the time; but, times
changed. The RDBs are capable of storing a filesystem driver and some drive init
code for the plugin disk driver card. That is giving malware authors entirely
goo easy a shot at owning a machine. Martin S., I would strongly suggest that
going forward those two capabilities be removed from the RDB readers in AmigaOS
as well as Linux OS.
I also note that at one time the RDB parsing facility in Linux benefitted from
my hands on the code and handled the block size/sector size issues very nicely.
I had an 18 GB disk I wanted to be able to read on Linux. I seem to remember
somebody at one time had a "this doesn't make any real world sense" event and
had to be reminded that the real world was wider than he envisioned. I don't
know what has happened since then.
{^_^} Joanne Dow
On 20180426 04:08, Geert Uytterhoeven wrote:
> Hi Martin,
>
> CC jdow
>
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
> <martin@lichtvoll.de> wrote:
>> You probably put your stick into a cave with ancient sleeping dragons :)
>>
>> Added in linux-m68k mailing list, as they likely have an opinion on how
>> to treat affs + RDB partition support. Also added in Jens Axboe about
>> patching that RDB support broken with 2 TB or larger harddisks issue
>> which had been in Linux kernel for 6 years while a patch exists that to
>> my testing back then solves the issue.
>
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
> handled by FUSE, moving RDB partition support to staging is not an option,
> as it is the only partitioning scheme that Amigas can boot from.
>
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's still not
> a reason to move it to staging (hi msdos partitioning!).
>
>> Matthew Wilcox - 26.04.18, 04:57:
>>> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
>>>> I had similar toughts some time ago while browsing the fs/
>>>> directory.
>>>> Access to the filesystem images can be reimplemented in FUSE, but
>>>> other than that, I don't think the in-kernel code would be missed.
>>>>
>>>> It's hard to know how many users are there. I was curious eg. about
>>>> bfs, befs, coda or feevxfs, looked at the last commits and searched
>>>> around web if there are any mentions or user community. But as long
>>>> as there's somebody listed in MAINTAINERS, the above are not
>>>> candidates for moving to staging or deletion.
>>>
>>> Yeah, it's pretty sad how few commits some of these filesystems have
>>> had in recent years. One can argue that they're stable and don't need
>>> to be fixed because they aren't broken, but I find it hard to believe
>>> that any of them were better-implemented than ext2 which still sees
>>> regular bugfixes.
>>
>> Regarding affs there is a severe issue which is not in affs itself but
>> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
>> partitioning standard, which is far more advanced than MBR: It overruns
>> for 2 TB or larger drives and then wraps over to the beginning of the
>> drive – I bet you can imagine what happens if you write to an area
>> larger than 2 TB. I learned this with an external 2TB RDB partitioned
>> harddisk back then, which I used for Sam440ep (a kind of successor for
>> old, classic Amiga hardware) backup + some Linux related stuff in
>> another partition.
>>
>> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
>> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
>> but never officially put it officially through upstreaming as I offered
>> to make a good description and upstream it through Jens Axboe.
>>
>> I may take this as a reason to… actually follow through this time,
>> hopefully remembering all the details in order to provide a meaningful
>> patch description – but I think mostly I can do just careful copy and
>> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
>> 43511, but not 43511 which is more about a safeguarding issue in case of
>> future overflows, I still think it would be good to go in in case affs +
>> RDB stays in their current places.
>>
>> However, in case you move affs to staging, I may be less motivated to do
>> so, but then I suggest you also move RDB partitioning support to
>> staging, cause this is the one that is known to be dangerously badly for
>> 2 TB or larger disks. And yeah, I admit I did not follow through with
>> having that patch upstreamed. Probably I did not want to be responsible
>> in case my description would not have been absolutely accurate or the
>> patch breaks something else. I do not have that 2 TB drive anymore and
>> don´t feel like setting one up in a suitable way in order to go about
>> this patch, but my testing back then was quite elaborate and I still
>> feel pretty confident about it.
>>
>> I totally get your motivation, but I would find it somewhat sad to see
>> the filesystems you mentioned go into staging. However, as I just shown
>> clearly, for the user it may be better, cause there may be unfixed
>> dangerous bugs. FUSE might be an interesting approach, but I bet it will
>> not solve the maintenance issue. If there is no one maintaining it in
>> the kernel, I think its unlikely to find someone adapting it to be a
>> FUSE filesystem and maintaining it. And then I am not aware of FUSE
>> based partitioning support. (And I think think ideally we´d had a
>> microkernel and run all filesystems in userspace processes with a
>> defined set of privileges, but that is simply not Linux as it is.)
>>
>> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
>> AmigaOS 4.1
>> https://lkml.org/lkml/2012/6/17/6
>>
>> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
>> instead of refusing to use it
>> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>>
>> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>
>> I forward the relevant mail of Joanne, in
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>>
>> I even have the patch in diff format. And I just checked, the issue is
>> still unpatched as of 4.16.3.
>>
>> ---------- Weitergeleitete Nachricht ----------
>>
>> Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> Datum: Montag, 18. Juni 2012, 03:28:48 CEST
>> Von: jdow <jdow@earthlink.net>
>> An: Martin Steigerwald <Martin@lichtvoll.de>
>> Kopie: Geert Uytterhoeven <geert@linux-m68k.org>, linux-
>> kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>, linux-
>> m68k@vger.kernel.org
>>
>> On 2012/06/17 14:20, Martin Steigerwald wrote:
>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>>> <Martin@lichtvoll.de> wrote:
>>>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>>>> | JXFS 64 bit file system
>>>>>> |
>>>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>>>> | system, which means that it reduces data loss if data writes to
>>>>>> | the disk are interrupted. It is the fastest and most reliable
>>>>>> | file system ever created for AmigaOS.
>>>>>>
>>>>>> http://www.amigaos.net/content/1/features
>>>>>>
>>>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
>> see
>>>>>> what they say about 2 TB limits.
>>>>>
>>>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>>>> 4096?
>>>>>
>>>>> block/partitions/amiga.c reads the block size from
>>>>> RigidDiskBlock.rdb_BlockBytes,
>>>>> but after conversion to 512-byte blocks, all further calculations
>> are
>>>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>>>
>>>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>>>> bytes per block,
>>>>> so I'll have to get a deeper look into your RDB first...
>>> […]
>>>> Note that the work I did on the Linux RDB code eons ago took full
>>>> cognizance of the physical and virtual block sizes.
>>> […]
>>>> I've asked Martin for a digital copy of his RDBs and what he thinks
>> the
>>>> partition(s) should look like. I should also be told whether the disk
>>>> is supposed to be solely Amiga OSs or not. I gather it's not.
>>>
>>> Its all in the bug report. profile-binary.img should be it.
>>>
>>> Its small so I attach it.
>>>
>>> Meanwhile I try to get the currently supported maximum disk size out
>> from
>>> the OS 4 developers. Maybe JXFS is playing tricks that other
>> filesystems do
>>> not play or simply has a different implementation.
>>>
>>> Thanks,
>>
>> I've sent Jens a proposed fix changing these lines in amiga.c.
>> ===8<--- was
>> struct PartitionBlock *pb;
>> int start_sect, nr_sects, blk, part, res = 0;
>> int blksize = 1; /* Multiplier for disk block size */
>> ===8<--- becomes changing line 338 into lines 338-339
>> struct PartitionBlock *pb;
>> sector_t start_sect, nr_sects;
>> int blk, part, res = 0;
>> int blksize = 1; /* Multiplier for disk block size */
>> ===8<---
>>
>> (I'm working without proper tools at the moment or I'd make a real diff.
>> Sorry.)
>>
>> This fix may not be complete. The RDBs contain provisions for describing
>> the physical disk block size and how many physical blocks are used in a
>> file system's logical blocks. This MAY not work quite right large
>> physical
>> block sizes.
>>
>> {^_^} Joanne Dow
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> -------------------------------------------------------------
>> […]
>>
>>> As long as there's someone who can be pestered with bugs, I don't see
>>> the need to push their baby out of the tree. I'm a bit more nervous
>>> about hfsplus; if it has users and no maintainer, those users are at
>>> risk.
>>>
>>> Perhaps we could advertise on Craigslist or something ... maintainer
>>> wanted for LTR.
>
> Gr{oetje,eeting}s,
>
> Geert
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-04-27 1:26 ` jdow
@ 2018-05-06 8:52 ` John Paul Adrian Glaubitz
2018-05-06 10:10 ` Martin Steigerwald
2018-05-07 4:54 ` jdow
0 siblings, 2 replies; 73+ messages in thread
From: John Paul Adrian Glaubitz @ 2018-05-06 8:52 UTC (permalink / raw)
To: jdow, Geert Uytterhoeven, Martin Steigerwald
Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k
On 04/27/2018 03:26 AM, jdow wrote:
> And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times
> changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely
> goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS
> as well as Linux OS.
I assume removing the feature for AmigaOS isn't really possible since we don't have
the source code for that, do we?
Also, if I remember correctly, Mac partitions can store filesystem drivers as well
and its actually a feature being used in MacOS. parted received a patch some time
ago to fix the correct handling for storing the filesystem driver in the partition
table.
I would be generally against removing these features as I don't think the security
risk is relevant for the majority of users. The Amiga is a hobbyist machine these
days and AmigaOS has certainly way more on than way to be compromised through
vulnerabilities.
Adrian
--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-05-06 8:52 ` John Paul Adrian Glaubitz
@ 2018-05-06 10:10 ` Martin Steigerwald
2018-05-07 4:54 ` jdow
1 sibling, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-05-06 10:10 UTC (permalink / raw)
To: John Paul Adrian Glaubitz
Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
linux-m68k
John Paul Adrian Glaubitz - 06.05.18, 10:52:
> On 04/27/2018 03:26 AM, jdow wrote:
> > And before I forget there are two features of the RDBs that I
> > heartily recommend never implementing on Linux. They were good
> > ideas at the time; but, times changed. The RDBs are capable of
> > storing a filesystem driver and some drive init code for the plugin
> > disk driver card. That is giving malware authors entirely goo easy
> > a shot at owning a machine. Martin S., I would strongly suggest
> > that going forward those two capabilities be removed from the RDB
> > readers in AmigaOS as well as Linux OS.
>
> I assume removing the feature for AmigaOS isn't really possible since
> we don't have the source code for that, do we?
AmigaOS 4.x does not support loading filesystems from RDB anymore as far
as I know. Meanwhile I am not involved with the AmigaOS development
anymore, but that is the last state I know. Similarily to Linux
filesystems drivers are loaded as "modules" into the kernel. Its also
still possible to load a filesystem as a file from disk, but that does
not work for the filesystem the kernel boots from. The AmigaOS kernel
still decides which of the kernel filesystem to use according to the
DOSType of the partition.
Thanks,
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-05-06 8:52 ` John Paul Adrian Glaubitz
2018-05-06 10:10 ` Martin Steigerwald
@ 2018-05-07 4:54 ` jdow
1 sibling, 0 replies; 73+ messages in thread
From: jdow @ 2018-05-07 4:54 UTC (permalink / raw)
To: John Paul Adrian Glaubitz, Geert Uytterhoeven, Martin Steigerwald
Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k
On 20180506 01:52, John Paul Adrian Glaubitz wrote:
> On 04/27/2018 03:26 AM, jdow wrote:
>> And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times
>> changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely
>> goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS
>> as well as Linux OS.
>
> I assume removing the feature for AmigaOS isn't really possible since we don't have
> the source code for that, do we?
>
> Also, if I remember correctly, Mac partitions can store filesystem drivers as well
> and its actually a feature being used in MacOS. parted received a patch some time
> ago to fix the correct handling for storing the filesystem driver in the partition
> table.
>
> I would be generally against removing these features as I don't think the security
> risk is relevant for the majority of users. The Amiga is a hobbyist machine these
> days and AmigaOS has certainly way more on than way to be compromised through
> vulnerabilities.
>
> Adrian
You do not necessarily have the source for the device drivers. However the
DriveInit code and the filesystem code get executed by the OS initialization
code. The objection I have to the concept is that it's invisible to the user.
The Linux filesystem code is either compiled into the kernel or is available in
the libraries where it can be monitored at several levels from source code on
up. Within AmigaDOS it can be monitored fairly easily by an AV tool - in theory.
Alas, this is trying to lock the barn door after the barn has burned to the
ground with a clever enough piece of malware. At least AmigaDOS AV tools should
be expected to examine DriveInit and filesystem images on disk in the RDBs for
malware modifications to those blocks. This is a burden Linux should not be
forced to bear. So loading filesystems from RDBs instead of the more usual and
accepted Linux practices should be disabled. And at least a portion of this
discussion is Linux related. That's why I mentioned disabling the feature. While
I cannot see much money in AmigaDOS related malware I can see it in Linux
malware. And there's no real "glory" in launching malware on AmigaDOS. It's too
easy a problem last I knew. "Whoopie, you have just proven you can ride a
tricycle. Can't you do better?" (One could argue that AmigaDOS 1.0 was
self-inflicted malware foisted on marvelous hardware for the era.)
{^_^} Joanne Dow
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
2018-04-26 11:08 ` Geert Uytterhoeven
2018-04-26 23:56 ` Finn Thain
2018-04-27 1:26 ` jdow
@ 2018-04-27 2:11 ` Michael Schmitz
2018-06-24 9:06 ` Martin Steigerwald
2018-04-27 8:01 ` Martin Steigerwald
3 siblings, 1 reply; 73+ messages in thread
From: Michael Schmitz @ 2018-04-27 2:11 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Martin Steigerwald, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow
Hi Geert,
test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?
(MSDOS partition support is not the only one with limitations - the
SGI partition parser also uses int types. Brings back memories of /me
hacking a large disk into many small partitions because Irix couldn't
digest it whole!)
Going to look into Atari partition format limitations now ...
Cheers,
Michael
On Thu, Apr 26, 2018 at 11:08 PM, Geert Uytterhoeven
<geert@linux-m68k.org> wrote:
> Hi Martin,
>
> CC jdow
>
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
> <martin@lichtvoll.de> wrote:
>> You probably put your stick into a cave with ancient sleeping dragons :)
>>
>> Added in linux-m68k mailing list, as they likely have an opinion on how
>> to treat affs + RDB partition support. Also added in Jens Axboe about
>> patching that RDB support broken with 2 TB or larger harddisks issue
>> which had been in Linux kernel for 6 years while a patch exists that to
>> my testing back then solves the issue.
>
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
> handled by FUSE, moving RDB partition support to staging is not an option,
> as it is the only partitioning scheme that Amigas can boot from.
>
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's still not
> a reason to move it to staging (hi msdos partitioning!).
>
>> Matthew Wilcox - 26.04.18, 04:57:
>>> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
>>> > I had similar toughts some time ago while browsing the fs/
>>> > directory.
>>> > Access to the filesystem images can be reimplemented in FUSE, but
>>> > other than that, I don't think the in-kernel code would be missed.
>>> >
>>> > It's hard to know how many users are there. I was curious eg. about
>>> > bfs, befs, coda or feevxfs, looked at the last commits and searched
>>> > around web if there are any mentions or user community. But as long
>>> > as there's somebody listed in MAINTAINERS, the above are not
>>> > candidates for moving to staging or deletion.
>>>
>>> Yeah, it's pretty sad how few commits some of these filesystems have
>>> had in recent years. One can argue that they're stable and don't need
>>> to be fixed because they aren't broken, but I find it hard to believe
>>> that any of them were better-implemented than ext2 which still sees
>>> regular bugfixes.
>>
>> Regarding affs there is a severe issue which is not in affs itself but
>> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
>> partitioning standard, which is far more advanced than MBR: It overruns
>> for 2 TB or larger drives and then wraps over to the beginning of the
>> drive – I bet you can imagine what happens if you write to an area
>> larger than 2 TB. I learned this with an external 2TB RDB partitioned
>> harddisk back then, which I used for Sam440ep (a kind of successor for
>> old, classic Amiga hardware) backup + some Linux related stuff in
>> another partition.
>>
>> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
>> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
>> but never officially put it officially through upstreaming as I offered
>> to make a good description and upstream it through Jens Axboe.
>>
>> I may take this as a reason to… actually follow through this time,
>> hopefully remembering all the details in order to provide a meaningful
>> patch description – but I think mostly I can do just careful copy and
>> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
>> 43511, but not 43511 which is more about a safeguarding issue in case of
>> future overflows, I still think it would be good to go in in case affs +
>> RDB stays in their current places.
>>
>> However, in case you move affs to staging, I may be less motivated to do
>> so, but then I suggest you also move RDB partitioning support to
>> staging, cause this is the one that is known to be dangerously badly for
>> 2 TB or larger disks. And yeah, I admit I did not follow through with
>> having that patch upstreamed. Probably I did not want to be responsible
>> in case my description would not have been absolutely accurate or the
>> patch breaks something else. I do not have that 2 TB drive anymore and
>> don´t feel like setting one up in a suitable way in order to go about
>> this patch, but my testing back then was quite elaborate and I still
>> feel pretty confident about it.
>>
>> I totally get your motivation, but I would find it somewhat sad to see
>> the filesystems you mentioned go into staging. However, as I just shown
>> clearly, for the user it may be better, cause there may be unfixed
>> dangerous bugs. FUSE might be an interesting approach, but I bet it will
>> not solve the maintenance issue. If there is no one maintaining it in
>> the kernel, I think its unlikely to find someone adapting it to be a
>> FUSE filesystem and maintaining it. And then I am not aware of FUSE
>> based partitioning support. (And I think think ideally we´d had a
>> microkernel and run all filesystems in userspace processes with a
>> defined set of privileges, but that is simply not Linux as it is.)
>>
>> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
>> AmigaOS 4.1
>> https://lkml.org/lkml/2012/6/17/6
>>
>> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
>> instead of refusing to use it
>> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>>
>> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>
>> I forward the relevant mail of Joanne, in
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>>
>> I even have the patch in diff format. And I just checked, the issue is
>> still unpatched as of 4.16.3.
>>
>> ---------- Weitergeleitete Nachricht ----------
>>
>> Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> Datum: Montag, 18. Juni 2012, 03:28:48 CEST
>> Von: jdow <jdow@earthlink.net>
>> An: Martin Steigerwald <Martin@lichtvoll.de>
>> Kopie: Geert Uytterhoeven <geert@linux-m68k.org>, linux-
>> kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>, linux-
>> m68k@vger.kernel.org
>>
>> On 2012/06/17 14:20, Martin Steigerwald wrote:
>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>>> <Martin@lichtvoll.de> wrote:
>>>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>>>> | JXFS 64 bit file system
>>>>>> |
>>>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>>>> | system, which means that it reduces data loss if data writes to
>>>>>> | the disk are interrupted. It is the fastest and most reliable
>>>>>> | file system ever created for AmigaOS.
>>>>>>
>>>>>> http://www.amigaos.net/content/1/features
>>>>>>
>>>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
>> see
>>>>>> what they say about 2 TB limits.
>>>>>
>>>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>>>> 4096?
>>>>>
>>>>> block/partitions/amiga.c reads the block size from
>>>>> RigidDiskBlock.rdb_BlockBytes,
>>>>> but after conversion to 512-byte blocks, all further calculations
>> are
>>>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>>>
>>>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>>>> bytes per block,
>>>>> so I'll have to get a deeper look into your RDB first...
>>> […]
>>>> Note that the work I did on the Linux RDB code eons ago took full
>>>> cognizance of the physical and virtual block sizes.
>>> […]
>>>> I've asked Martin for a digital copy of his RDBs and what he thinks
>> the
>>>> partition(s) should look like. I should also be told whether the disk
>>>> is supposed to be solely Amiga OSs or not. I gather it's not.
>>>
>>> Its all in the bug report. profile-binary.img should be it.
>>>
>>> Its small so I attach it.
>>>
>>> Meanwhile I try to get the currently supported maximum disk size out
>> from
>>> the OS 4 developers. Maybe JXFS is playing tricks that other
>> filesystems do
>>> not play or simply has a different implementation.
>>>
>>> Thanks,
>>
>> I've sent Jens a proposed fix changing these lines in amiga.c.
>> ===8<--- was
>> struct PartitionBlock *pb;
>> int start_sect, nr_sects, blk, part, res = 0;
>> int blksize = 1; /* Multiplier for disk block size */
>> ===8<--- becomes changing line 338 into lines 338-339
>> struct PartitionBlock *pb;
>> sector_t start_sect, nr_sects;
>> int blk, part, res = 0;
>> int blksize = 1; /* Multiplier for disk block size */
>> ===8<---
>>
>> (I'm working without proper tools at the moment or I'd make a real diff.
>> Sorry.)
>>
>> This fix may not be complete. The RDBs contain provisions for describing
>> the physical disk block size and how many physical blocks are used in a
>> file system's logical blocks. This MAY not work quite right large
>> physical
>> block sizes.
>>
>> {^_^} Joanne Dow
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> -------------------------------------------------------------
>> […]
>>
>>> As long as there's someone who can be pestered with bugs, I don't see
>>> the need to push their baby out of the tree. I'm a bit more nervous
>>> about hfsplus; if it has users and no maintainer, those users are at
>>> risk.
>>>
>>> Perhaps we could advertise on Craigslist or something ... maintainer
>>> wanted for LTR.
>
> Gr{oetje,eeting}s,
>
> Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> -- Linus Torvalds
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
2018-04-27 2:11 ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
@ 2018-06-24 9:06 ` Martin Steigerwald
2018-06-24 11:33 ` moving affs + RDB partition support to staging? jdow
` (2 more replies)
0 siblings, 3 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-24 9:06 UTC (permalink / raw)
To: Michael Schmitz
Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow
Hi.
Michael Schmitz - 27.04.18, 04:11:
> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> indicate the RDB parser bug is fixed by the patch given there, so if
> Martin now submits the patch, all should be well?
Ok, better be honest than having anyone waiting for it:
I do not care enough about this, in order to motivate myself preparing
the a patch from Joanne Dow�s fix.
I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.
So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.
Thanks.
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-24 9:06 ` Martin Steigerwald
@ 2018-06-24 11:33 ` jdow
2018-06-24 11:40 ` jdow
2018-06-25 7:53 ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
2 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-24 11:33 UTC (permalink / raw)
To: Martin Steigerwald, Michael Schmitz
Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, jdow
It's more likely the filesystem unless I didn't get the RDBs up to 3.1 status. I
am pretty damn sure I did as I had an 18 gigabyte disk I was using at the time.
And I made sure it would mount with Linux. Files greater than 2 gigs were sure
to get messed up. But that was not RDBs. The disk had 9 gigs worth of AFS
partitions and 9 gigs worth of SFS partitions. The big SFS partition was most of
the 9 gigabytes. (The hdf file is 8,932,818,944 bytes.) There was a tiny system
partition on it. It used some of Bill Hawes' goodies to overlay some useful paths.
I'm not sure the machine still boots. The disk in question no longer spins up.
{^_^}
On 20180624 02:06, Martin Steigerwald wrote:
> Hi.
>
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
>
> Ok, better be honest than having anyone waiting for it:
>
> I do not care enough about this, in order to motivate myself preparing
> the a patch from Joanne Dow´s fix.
>
> I am not even using my Amiga boxes anymore, not even the Sam440ep which
> I still have in my apartment.
>
> So RDB support in Linux it remains broken for disks larger 2 TB, unless
> someone else does.
>
> Thanks.
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-24 9:06 ` Martin Steigerwald
2018-06-24 11:33 ` moving affs + RDB partition support to staging? jdow
@ 2018-06-24 11:40 ` jdow
2018-06-26 2:23 ` Michael Schmitz
2018-06-25 7:53 ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
2 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-24 11:40 UTC (permalink / raw)
To: Martin Steigerwald, Michael Schmitz
Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, jdow
BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn dool.
If memory serves the RDBs think in blocks rather than bytes so it should work up
to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes.
But that wastes just a WHOLE LOT of disk in block maps. Go up to 4096 or 8192.
The latter is 35 TB.
{^_^}
On 20180624 02:06, Martin Steigerwald wrote:
> Hi.
>
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
>
> Ok, better be honest than having anyone waiting for it:
>
> I do not care enough about this, in order to motivate myself preparing
> the a patch from Joanne Dow´s fix.
>
> I am not even using my Amiga boxes anymore, not even the Sam440ep which
> I still have in my apartment.
>
> So RDB support in Linux it remains broken for disks larger 2 TB, unless
> someone else does.
>
> Thanks.
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-24 11:40 ` jdow
@ 2018-06-26 2:23 ` Michael Schmitz
2018-06-26 5:17 ` jdow
2018-06-26 8:02 ` Martin Steigerwald
0 siblings, 2 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-26 2:23 UTC (permalink / raw)
To: jdow
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Joanne,
Martin's boot log (including your patch) says:
Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
Attached SCSI disk
so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.
Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others yet.
Cheers,
Michael
On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
> BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
> dool.
>
> If memory serves the RDBs think in blocks rather than bytes so it should
> work up to 2 gigablocks whatever your block size is. 512 blocks is
> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
> Go up to 4096 or 8192. The latter is 35 TB.
>
> {^_^}
> On 20180624 02:06, Martin Steigerwald wrote:
>>
>> Hi.
>>
>> Michael Schmitz - 27.04.18, 04:11:
>>>
>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>> Martin now submits the patch, all should be well?
>>
>>
>> Ok, better be honest than having anyone waiting for it:
>>
>> I do not care enough about this, in order to motivate myself preparing
>> the a patch from Joanne Dow´s fix.
>>
>> I am not even using my Amiga boxes anymore, not even the Sam440ep which
>> I still have in my apartment.
>>
>> So RDB support in Linux it remains broken for disks larger 2 TB, unless
>> someone else does.
>>
>> Thanks.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 2:23 ` Michael Schmitz
@ 2018-06-26 5:17 ` jdow
2018-06-26 8:12 ` Martin Steigerwald
2018-06-26 8:31 ` Michael Schmitz
2018-06-26 8:02 ` Martin Steigerwald
1 sibling, 2 replies; 73+ messages in thread
From: jdow @ 2018-06-26 5:17 UTC (permalink / raw)
To: Michael Schmitz
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
As long as it preserves compatibility it should be OK, I suppose. Personally I'd
make any partitioning tool front end gently force the block size towards 8k as
the disk size gets larger. The file systems may also run into 2TB issues that
are not obvious. An unused blocks list will have to go beyond a uint32_t size,
for example. But a block list (OFS for sure, don't remember for the newer AFS)
uses a tad under 1% of the disk all by itself. A block bitmap is not quite so
bad. {^_-}
Just be sure you are aware of all the ramifications when you make a change. I
remember thinking about this for awhile and then determining I REALLY did not
want to think about it as my brain was getting tied into a gordian knot.
{^_^}
On 20180625 19:23, Michael Schmitz wrote:
> Joanne,
>
> Martin's boot log (including your patch) says:
>
> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
> 4)
> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> Attached SCSI disk
>
> so it's indeed a case of self inflicted damage (RDSK (512) means 512
> byte blocks) and can be worked around by using a different block size.
>
> Your memory serves right indeed - blocksize is in 512 bytes units.
> I'll still submit a patch to Jens anyway as this may bite others yet.
>
> Cheers,
>
> Michael
>
>
> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>> BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
>> dool.
>>
>> If memory serves the RDBs think in blocks rather than bytes so it should
>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
>> Go up to 4096 or 8192. The latter is 35 TB.
>>
>> {^_^}
>> On 20180624 02:06, Martin Steigerwald wrote:
>>>
>>> Hi.
>>>
>>> Michael Schmitz - 27.04.18, 04:11:
>>>>
>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>> Martin now submits the patch, all should be well?
>>>
>>>
>>> Ok, better be honest than having anyone waiting for it:
>>>
>>> I do not care enough about this, in order to motivate myself preparing
>>> the a patch from Joanne Dow´s fix.
>>>
>>> I am not even using my Amiga boxes anymore, not even the Sam440ep which
>>> I still have in my apartment.
>>>
>>> So RDB support in Linux it remains broken for disks larger 2 TB, unless
>>> someone else does.
>>>
>>> Thanks.
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 5:17 ` jdow
@ 2018-06-26 8:12 ` Martin Steigerwald
2018-06-26 9:46 ` jdow
2018-06-26 8:31 ` Michael Schmitz
1 sibling, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-26 8:12 UTC (permalink / raw)
To: jdow
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Joanne.
jdow - 26.06.18, 07:17:
> As long as it preserves compatibility it should be OK, I suppose.
> Personally I'd make any partitioning tool front end gently force the
> block size towards 8k as the disk size gets larger. The file systems
> may also run into 2TB issues that are not obvious. An unused blocks
> list will have to go beyond a uint32_t size, for example. But a block
> list (OFS for sure, don't remember for the newer AFS) uses a tad
> under 1% of the disk all by itself. A block bitmap is not quite so
> bad. {^_-}
>
> Just be sure you are aware of all the ramifications when you make a
> change. I remember thinking about this for awhile and then
> determining I REALLY did not want to think about it as my brain was
> getting tied into a gordian knot.
Heh… :)
Well, all I can say it that it just worked on AmigaOS 4. I did not see
any data corruption in any of the filesystems. Well as far as I have
been able to check. There has been no repair tool for JXFS I think. But
as I migrated the data on it to SFS, I was able to copy everything.
Famous last words.
Well especially the disk size was detected properly and there was no
overflow like on Linux. So I´d say rather have on error less than one
error more.
Of course, it could also be an option to outright *refuse* to detect
such disks with a big fat warning in kernel log that it may unsafe. But
overflowing and thus on writes overwriting existing data is not safe.
I do think it is safe enough, but what I do know about the internals
about RDB (more than the average user certainly, but not as much as you
or some other AmigaOS developer who digged deeper into that).
So in case you´d rather see Linux to refuse to handle disks like that,
that would also be fine with me. Just do not handle them in the broken
way that they are handled in Linux now. I.e. do not deliberately corrupt
things as in "Its dangerous, so let´s overwrite data straight away, so
the user gets it." :)
Anyway, in my opinion RDB still just so much more advanced than MBR and
in some parts even on par with the much later GPT. With some limitations
it is a quite brilliant partition format, if you ask me.
> {^_^}
>
> On 20180625 19:23, Michael Schmitz wrote:
> > Joanne,
> >
> > Martin's boot log (including your patch) says:
> >
> > Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
> > (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
> > spb
> > 4)
> > Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> > Attached SCSI disk
> >
> > so it's indeed a case of self inflicted damage (RDSK (512) means 512
> > byte blocks) and can be worked around by using a different block
> > size.
> >
> > Your memory serves right indeed - blocksize is in 512 bytes units.
> > I'll still submit a patch to Jens anyway as this may bite others
> > yet.
> >
> > Cheers,
> >
> > Michael
> >
> > On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
> >> BTW - anybody who uses 512 byte blocks with an Amiga file system is
> >> a famn dool.
> >>
> >> If memory serves the RDBs think in blocks rather than bytes so it
> >> should work up to 2 gigablocks whatever your block size is. 512
> >> blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >> disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.
> >>
> >> {^_^}
> >>
> >> On 20180624 02:06, Martin Steigerwald wrote:
> >>> Hi.
> >>>
> >>> Michael Schmitz - 27.04.18, 04:11:
> >>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>> indicate the RDB parser bug is fixed by the patch given there, so
> >>>> if
> >>>> Martin now submits the patch, all should be well?
> >>>
> >>> Ok, better be honest than having anyone waiting for it:
> >>>
> >>> I do not care enough about this, in order to motivate myself
> >>> preparing the a patch from Joanne Dow´s fix.
> >>>
> >>> I am not even using my Amiga boxes anymore, not even the Sam440ep
> >>> which I still have in my apartment.
> >>>
> >>> So RDB support in Linux it remains broken for disks larger 2 TB,
> >>> unless someone else does.
> >>>
> >>> Thanks.
[…]
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 8:12 ` Martin Steigerwald
@ 2018-06-26 9:46 ` jdow
0 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-26 9:46 UTC (permalink / raw)
To: Martin Steigerwald
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
(Get everybody)
Speak to the high level driver folks about that. The low level stuff is
basically dumb. It tells you what it finds. It tells the rest of the OS (the
file systems) what it found. The amiga-makefs (or whatever it is called) needs
to grow some added intelligence just as HDToolBox needs to grow more
intelligence. The step from RDB uint32_t to RDB uint64_t probably needs a great
deal of thought. Personally I'd be tempted to adopt GUID based partitioning
system. It has already considered this stuff.
{^_^}
On 20180626 01:12, Martin Steigerwald wrote:
> Joanne.
>
> jdow - 26.06.18, 07:17:
>> As long as it preserves compatibility it should be OK, I suppose.
>> Personally I'd make any partitioning tool front end gently force the
>> block size towards 8k as the disk size gets larger. The file systems
>> may also run into 2TB issues that are not obvious. An unused blocks
>> list will have to go beyond a uint32_t size, for example. But a block
>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>> under 1% of the disk all by itself. A block bitmap is not quite so
>> bad. {^_-}
>>
>> Just be sure you are aware of all the ramifications when you make a
>> change. I remember thinking about this for awhile and then
>> determining I REALLY did not want to think about it as my brain was
>> getting tied into a gordian knot.
>
> Heh… :)
>
> Well, all I can say it that it just worked on AmigaOS 4. I did not see
> any data corruption in any of the filesystems. Well as far as I have
> been able to check. There has been no repair tool for JXFS I think. But
> as I migrated the data on it to SFS, I was able to copy everything.
>
> Famous last words.
>
> Well especially the disk size was detected properly and there was no
> overflow like on Linux. So I´d say rather have on error less than one
> error more.
>
> Of course, it could also be an option to outright *refuse* to detect
> such disks with a big fat warning in kernel log that it may unsafe. But
> overflowing and thus on writes overwriting existing data is not safe.
>
> I do think it is safe enough, but what I do know about the internals
> about RDB (more than the average user certainly, but not as much as you
> or some other AmigaOS developer who digged deeper into that).
>
> So in case you´d rather see Linux to refuse to handle disks like that,
> that would also be fine with me. Just do not handle them in the broken
> way that they are handled in Linux now. I.e. do not deliberately corrupt
> things as in "Its dangerous, so let´s overwrite data straight away, so
> the user gets it." :)
>
> Anyway, in my opinion RDB still just so much more advanced than MBR and
> in some parts even on par with the much later GPT. With some limitations
> it is a quite brilliant partition format, if you ask me.
>
>> {^_^}
>>
>> On 20180625 19:23, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> Martin's boot log (including your patch) says:
>>>
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
>>> spb
>>> 4)
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>> Attached SCSI disk
>>>
>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>> byte blocks) and can be worked around by using a different block
>>> size.
>>>
>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>> I'll still submit a patch to Jens anyway as this may bite others
>>> yet.
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>> a famn dool.
>>>>
>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>> should work up to 2 gigablocks whatever your block size is. 512
>>>> blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of
>>>> disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>> Hi.
>>>>>
>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>> indicate the RDB parser bug is fixed by the patch given there, so
>>>>>> if
>>>>>> Martin now submits the patch, all should be well?
>>>>>
>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>
>>>>> I do not care enough about this, in order to motivate myself
>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>
>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>> which I still have in my apartment.
>>>>>
>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>> unless someone else does.
>>>>>
>>>>> Thanks.
> […]
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 5:17 ` jdow
2018-06-26 8:12 ` Martin Steigerwald
@ 2018-06-26 8:31 ` Michael Schmitz
2018-06-26 9:45 ` jdow
1 sibling, 1 reply; 73+ messages in thread
From: Michael Schmitz @ 2018-06-26 8:31 UTC (permalink / raw)
To: jdow
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Joanne,
I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing the
issues Martin reported. Your patch addresses that by using the correct
data type for the calculations (as do other partition parsers that may
have to deal with large disks) and fixes Martin's bug, so appears to be
the right thing to do.
Using 64 bit data types for disks smaller than 2 TB where calculations
don't currently overflow is not expected to cause new issues, other than
enabling use of disk and partitions larger than 2 TB (which may have
ramifications with filesystems on these partitions). So comptibility is
preserved.
Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size stored
in the RDB would enforce use of the same block size in filesystems.
We'll have to rely on the filesystem tools to get that right, too. Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should
allow partitions larger than 2 TB to work already (but I suspect Al Viro
may have found a few issues when he looked at the AFFS code so I won't
say more). Anyway partitioning tools and filesystems are unrelated to
the Linux partition parser code which is all we aim to fix in this patch.
If you feel strongly about unknown ramifications of any filesystems on
partitions larger than 2 TB, say so and I'll have the kernel print a
warning about these partitions.
I'll get this patch tested on Martin's test case image as well as on a
RDB image from a disk known to currently work under Linux (thanks Geert
for the losetup hint). Can't do much more without procuring a working
Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
for tests is a long way away from my home indeed.
Cheers,
Michael
Am 26.06.18 um 17:17 schrieb jdow:
> As long as it preserves compatibility it should be OK, I suppose.
> Personally I'd make any partitioning tool front end gently force the
> block size towards 8k as the disk size gets larger. The file systems
> may also run into 2TB issues that are not obvious. An unused blocks
> list will have to go beyond a uint32_t size, for example. But a block
> list (OFS for sure, don't remember for the newer AFS) uses a tad under
> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>
> Just be sure you are aware of all the ramifications when you make a
> change. I remember thinking about this for awhile and then determining
> I REALLY did not want to think about it as my brain was getting tied
> into a gordian knot.
>
> {^_^}
>
> On 20180625 19:23, Michael Schmitz wrote:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
>>
>> Your memory serves right indeed - blocksize is in 512 bytes units.
>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>
>> Cheers,
>>
>> Michael
>>
>>
>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>> a famn
>>> dool.
>>>
>>> If memory serves the RDBs think in blocks rather than bytes so it
>>> should
>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>> block maps.
>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>
>>> {^_^}
>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>
>>>> Hi.
>>>>
>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>
>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>> Martin now submits the patch, all should be well?
>>>>
>>>>
>>>> Ok, better be honest than having anyone waiting for it:
>>>>
>>>> I do not care enough about this, in order to motivate myself preparing
>>>> the a patch from Joanne Dow´s fix.
>>>>
>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>> which
>>>> I still have in my apartment.
>>>>
>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>> unless
>>>> someone else does.
>>>>
>>>> Thanks.
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-m68k" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 8:31 ` Michael Schmitz
@ 2018-06-26 9:45 ` jdow
2018-06-27 1:07 ` Michael Schmitz
0 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-26 9:45 UTC (permalink / raw)
To: Michael Schmitz
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
If it is not backwards compatible I for one would refuse to use it. And if it
still mattered that much to me I'd also generate a reasonable alternative.
Modifying RDBs nay not be even an approximation of a good idea. You'd discover
that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it
is for your personal use then you're entirely free to reject my advice and are
probably smart enough to keep it working for yourself.
GPT is probably the right way to go. Preserve the ability to read RDBs for
legacy disks only.
{^_^}
On 20180626 01:31, Michael Schmitz wrote:
> Joanne,
>
> I think we all agree that doing 32 bit calculations on 512-byte block
> addresses that overflow on disks 2 TB and larger is a bug, causing the
> issues Martin reported. Your patch addresses that by using the correct
> data type for the calculations (as do other partition parsers that may
> have to deal with large disks) and fixes Martin's bug, so appears to be
> the right thing to do.
>
> Using 64 bit data types for disks smaller than 2 TB where calculations
> don't currently overflow is not expected to cause new issues, other than
> enabling use of disk and partitions larger than 2 TB (which may have
> ramifications with filesystems on these partitions). So comptibility is
> preserved.
>
> Forcing larger block sizes might be a good strategy to avoid overflow
> issues in filesystems as well, but I can't see how the block size stored
> in the RDB would enforce use of the same block size in filesystems.
> We'll have to rely on the filesystem tools to get that right, too. Linux
> AFFS does allow block sizes up to 4k (VFS limitation) so this should
> allow partitions larger than 2 TB to work already (but I suspect Al Viro
> may have found a few issues when he looked at the AFFS code so I won't
> say more). Anyway partitioning tools and filesystems are unrelated to
> the Linux partition parser code which is all we aim to fix in this patch.
>
> If you feel strongly about unknown ramifications of any filesystems on
> partitions larger than 2 TB, say so and I'll have the kernel print a
> warning about these partitions.
>
> I'll get this patch tested on Martin's test case image as well as on a
> RDB image from a disk known to currently work under Linux (thanks Geert
> for the losetup hint). Can't do much more without procuring a working
> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
> for tests is a long way away from my home indeed.
>
> Cheers,
>
> Michael
>
> Am 26.06.18 um 17:17 schrieb jdow:
>> As long as it preserves compatibility it should be OK, I suppose.
>> Personally I'd make any partitioning tool front end gently force the
>> block size towards 8k as the disk size gets larger. The file systems
>> may also run into 2TB issues that are not obvious. An unused blocks
>> list will have to go beyond a uint32_t size, for example. But a block
>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>
>> Just be sure you are aware of all the ramifications when you make a
>> change. I remember thinking about this for awhile and then determining
>> I REALLY did not want to think about it as my brain was getting tied
>> into a gordian knot.
>>
>> {^_^}
>>
>> On 20180625 19:23, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> Martin's boot log (including your patch) says:
>>>
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>> 4)
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>> Attached SCSI disk
>>>
>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>> byte blocks) and can be worked around by using a different block size.
>>>
>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>>
>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>> a famn
>>>> dool.
>>>>
>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>> should
>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>> block maps.
>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>
>>>> {^_^}
>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>
>>>>> Hi.
>>>>>
>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>
>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>> Martin now submits the patch, all should be well?
>>>>>
>>>>>
>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>
>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>> the a patch from Joanne Dow´s fix.
>>>>>
>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>> which
>>>>> I still have in my apartment.
>>>>>
>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>> unless
>>>>> someone else does.
>>>>>
>>>>> Thanks.
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-m68k" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 9:45 ` jdow
@ 2018-06-27 1:07 ` Michael Schmitz
2018-06-27 6:24 ` jdow
2018-06-27 7:57 ` moving affs + RDB partition support to staging? Martin Steigerwald
0 siblings, 2 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-27 1:07 UTC (permalink / raw)
To: jdow
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Joanne,
As far as I have been able to test, the change is backwards compatible
(RDB partitions from an old disk 80 GB disk are still recognized OK).
That"s only been done on an emulator though.
Your advice about the dangers of using RDB disks that would have
failed the current Linux RDB parser on legacy 32 bit systems is well
taken though. Maybe Martin can clarify that for me - was the 2 TB disk
in question ever used on a 32 bit Amiga system?
RDB disk format is meant for legacy use only, so I think we can get
away with printing a big fat warning during boot, advising the user
that the oversize RDB partition(s) scanned are not compatible with
legacy 32 bit AmigaOS. With the proposed fix they will work under both
AmigaOS 4.1 and Linux instead of truncating the first overflowing
partition at disk end and trashing valid partitions that overlap,
which is what Martin was after.
If that still seems too risky, we can make the default behaviour to
bail out once a potential overflow is detected, and allow the user to
override that through a boot parameter. I'd leave that decision up for
the code review on linux-block.
Two more comments: Linux uses 512 byte block sizes for the partition
start and size calculations, so a change of the RDB blocksize to
reduce the block counts stored in the RDB would still result in the
same overflow. And amiga-fdisk is indeed utterly broken and needs
updating (along with probably most legacy m68k partitioners). Adrian
has advertised parted as replacement for the old tools - maybe this
would make a nice test case for parted?
Cheers,
Michael
On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> If it is not backwards compatible I for one would refuse to use it. And if
> it still mattered that much to me I'd also generate a reasonable
> alternative. Modifying RDBs nay not be even an approximation of a good idea.
> You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
> only system. If it is for your personal use then you're entirely free to
> reject my advice and are probably smart enough to keep it working for
> yourself.
>
> GPT is probably the right way to go. Preserve the ability to read RDBs for
> legacy disks only.
>
> {^_^}
>
>
> On 20180626 01:31, Michael Schmitz wrote:
>>
>> Joanne,
>>
>> I think we all agree that doing 32 bit calculations on 512-byte block
>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>> issues Martin reported. Your patch addresses that by using the correct
>> data type for the calculations (as do other partition parsers that may
>> have to deal with large disks) and fixes Martin's bug, so appears to be
>> the right thing to do.
>>
>> Using 64 bit data types for disks smaller than 2 TB where calculations
>> don't currently overflow is not expected to cause new issues, other than
>> enabling use of disk and partitions larger than 2 TB (which may have
>> ramifications with filesystems on these partitions). So comptibility is
>> preserved.
>>
>> Forcing larger block sizes might be a good strategy to avoid overflow
>> issues in filesystems as well, but I can't see how the block size stored
>> in the RDB would enforce use of the same block size in filesystems.
>> We'll have to rely on the filesystem tools to get that right, too. Linux
>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>> allow partitions larger than 2 TB to work already (but I suspect Al Viro
>> may have found a few issues when he looked at the AFFS code so I won't
>> say more). Anyway partitioning tools and filesystems are unrelated to
>> the Linux partition parser code which is all we aim to fix in this patch.
>>
>> If you feel strongly about unknown ramifications of any filesystems on
>> partitions larger than 2 TB, say so and I'll have the kernel print a
>> warning about these partitions.
>>
>> I'll get this patch tested on Martin's test case image as well as on a
>> RDB image from a disk known to currently work under Linux (thanks Geert
>> for the losetup hint). Can't do much more without procuring a working
>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
>> for tests is a long way away from my home indeed.
>>
>> Cheers,
>>
>> Michael
>>
>> Am 26.06.18 um 17:17 schrieb jdow:
>>>
>>> As long as it preserves compatibility it should be OK, I suppose.
>>> Personally I'd make any partitioning tool front end gently force the
>>> block size towards 8k as the disk size gets larger. The file systems
>>> may also run into 2TB issues that are not obvious. An unused blocks
>>> list will have to go beyond a uint32_t size, for example. But a block
>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>>
>>> Just be sure you are aware of all the ramifications when you make a
>>> change. I remember thinking about this for awhile and then determining
>>> I REALLY did not want to think about it as my brain was getting tied
>>> into a gordian knot.
>>>
>>> {^_^}
>>>
>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>
>>>> Joanne,
>>>>
>>>> Martin's boot log (including your patch) says:
>>>>
>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>> 4)
>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>> Attached SCSI disk
>>>>
>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>> byte blocks) and can be worked around by using a different block size.
>>>>
>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>
>>>> Cheers,
>>>>
>>>> Michael
>>>>
>>>>
>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>
>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>> a famn
>>>>> dool.
>>>>>
>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>> should
>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>> block maps.
>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>
>>>>> {^_^}
>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>
>>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>
>>>>>>>
>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>>> Martin now submits the patch, all should be well?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>
>>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>
>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>> which
>>>>>> I still have in my apartment.
>>>>>>
>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>> unless
>>>>>> someone else does.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-27 1:07 ` Michael Schmitz
@ 2018-06-27 6:24 ` jdow
2018-06-27 8:03 ` Martin Steigerwald
2018-06-27 9:00 ` moving affs + RDB partition support to staging? Michael Schmitz
2018-06-27 7:57 ` moving affs + RDB partition support to staging? Martin Steigerwald
1 sibling, 2 replies; 73+ messages in thread
From: jdow @ 2018-06-27 6:24 UTC (permalink / raw)
To: Michael Schmitz
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
You allergic to using a GPT solution? It will get away from some of the evils
that RDB has inherent in it because they are also features? (Loading a
filesystem or DriveInit code from RDBs is just asking for a nearly impossible to
remove malware infection.) Furthermore, any 32 bit system that sees an RDSK
block is going to try to translate it. If you add a new RDB format you are going
to get bizarre and probably quite destructive results from the mistake. Fail
safe is a rather good notion, methinks.
Personally I figure this is all rather surreal. 2TG of junk on an Amiga system
seems utterly outlandish to me. You cited another overflow potential. There are
at least three we've identified, I believe. Are you 100% sure there are no more?
The specific one you mention of translating RDB to Linux has a proper solution
in the RDB reader. It should recover such overflow errors in the RDB as it can
with due care and polish. It should flag any other overflow error it detects
within the RDBs and return an error such as to leave the disk unmounted or
mounted read-only if you feel like messing up a poor sod's backups. The simple
solution is to read each of the variables with the nominal RDB size and convert
it to uint64_t before calculating byte indices.
However, consider my inputs as advice from an adult who has seen the Amiga
Elephant so to speak. I am not trying to assert any control. Do as you wish;
but, I would plead with you to avoid ANY chance you can for the user to make a
bonehead stupid move and lose all his treasured disk archives. Doing otherwise
is very poor form.
{o.o} Joanne "Said enough, she has" Dow
On 20180626 18:07, Michael Schmitz wrote:
> Joanne,
>
> As far as I have been able to test, the change is backwards compatible
> (RDB partitions from an old disk 80 GB disk are still recognized OK).
> That"s only been done on an emulator though.
>
> Your advice about the dangers of using RDB disks that would have
> failed the current Linux RDB parser on legacy 32 bit systems is well
> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
> in question ever used on a 32 bit Amiga system?
>
> RDB disk format is meant for legacy use only, so I think we can get
> away with printing a big fat warning during boot, advising the user
> that the oversize RDB partition(s) scanned are not compatible with
> legacy 32 bit AmigaOS. With the proposed fix they will work under both
> AmigaOS 4.1 and Linux instead of truncating the first overflowing
> partition at disk end and trashing valid partitions that overlap,
> which is what Martin was after.
>
> If that still seems too risky, we can make the default behaviour to
> bail out once a potential overflow is detected, and allow the user to
> override that through a boot parameter. I'd leave that decision up for
> the code review on linux-block.
>
> Two more comments: Linux uses 512 byte block sizes for the partition
> start and size calculations, so a change of the RDB blocksize to
> reduce the block counts stored in the RDB would still result in the
> same overflow. And amiga-fdisk is indeed utterly broken and needs
> updating (along with probably most legacy m68k partitioners). Adrian
> has advertised parted as replacement for the old tools - maybe this
> would make a nice test case for parted?
>
> Cheers,
>
> Michael
>
>
>
> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>> If it is not backwards compatible I for one would refuse to use it. And if
>> it still mattered that much to me I'd also generate a reasonable
>> alternative. Modifying RDBs nay not be even an approximation of a good idea.
>> You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
>> only system. If it is for your personal use then you're entirely free to
>> reject my advice and are probably smart enough to keep it working for
>> yourself.
>>
>> GPT is probably the right way to go. Preserve the ability to read RDBs for
>> legacy disks only.
>>
>> {^_^}
>>
>>
>> On 20180626 01:31, Michael Schmitz wrote:
>>>
>>> Joanne,
>>>
>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>> issues Martin reported. Your patch addresses that by using the correct
>>> data type for the calculations (as do other partition parsers that may
>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>> the right thing to do.
>>>
>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>> don't currently overflow is not expected to cause new issues, other than
>>> enabling use of disk and partitions larger than 2 TB (which may have
>>> ramifications with filesystems on these partitions). So comptibility is
>>> preserved.
>>>
>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>> issues in filesystems as well, but I can't see how the block size stored
>>> in the RDB would enforce use of the same block size in filesystems.
>>> We'll have to rely on the filesystem tools to get that right, too. Linux
>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>> allow partitions larger than 2 TB to work already (but I suspect Al Viro
>>> may have found a few issues when he looked at the AFFS code so I won't
>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>> the Linux partition parser code which is all we aim to fix in this patch.
>>>
>>> If you feel strongly about unknown ramifications of any filesystems on
>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>> warning about these partitions.
>>>
>>> I'll get this patch tested on Martin's test case image as well as on a
>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>> for the losetup hint). Can't do much more without procuring a working
>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
>>> for tests is a long way away from my home indeed.
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>
>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>> Personally I'd make any partitioning tool front end gently force the
>>>> block size towards 8k as the disk size gets larger. The file systems
>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>>>
>>>> Just be sure you are aware of all the ramifications when you make a
>>>> change. I remember thinking about this for awhile and then determining
>>>> I REALLY did not want to think about it as my brain was getting tied
>>>> into a gordian knot.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>
>>>>> Joanne,
>>>>>
>>>>> Martin's boot log (including your patch) says:
>>>>>
>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>> 4)
>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>> Attached SCSI disk
>>>>>
>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>> byte blocks) and can be worked around by using a different block size.
>>>>>
>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Michael
>>>>>
>>>>>
>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>
>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>> a famn
>>>>>> dool.
>>>>>>
>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>> should
>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>> block maps.
>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>
>>>>>> {^_^}
>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>
>>>>>>>>
>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>
>>>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>
>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>> which
>>>>>>> I still have in my apartment.
>>>>>>>
>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>> unless
>>>>>>> someone else does.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-27 6:24 ` jdow
@ 2018-06-27 8:03 ` Martin Steigerwald
2018-06-28 2:57 ` jdow
2018-06-27 9:00 ` moving affs + RDB partition support to staging? Michael Schmitz
1 sibling, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-27 8:03 UTC (permalink / raw)
To: jdow
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Dear Joanne.
jdow - 27.06.18, 08:24:
> You allergic to using a GPT solution? It will get away from some of
> the evils that RDB has inherent in it because they are also features?
> (Loading a filesystem or DriveInit code from RDBs is just asking for
> a nearly impossible to remove malware infection.) Furthermore, any 32
> bit system that sees an RDSK block is going to try to translate it.
> If you add a new RDB format you are going to get bizarre and probably
> quite destructive results from the mistake. Fail safe is a rather
> good notion, methinks.
>
> Personally I figure this is all rather surreal. 2TG of junk on an
> Amiga system seems utterly outlandish to me. You cited another
> overflow potential. There are at least three we've identified, I
> believe. Are you 100% sure there are no more? The specific one you
> mention of translating RDB to Linux has a proper solution in the RDB
> reader. It should recover such overflow errors in the RDB as it can
> with due care and polish. It should flag any other overflow error it
> detects within the RDBs and return an error such as to leave the disk
> unmounted or mounted read-only if you feel like messing up a poor
> sod's backups. The simple solution is to read each of the variables
> with the nominal RDB size and convert it to uint64_t before
> calculating byte indices.
>
> However, consider my inputs as advice from an adult who has seen the
> Amiga Elephant so to speak. I am not trying to assert any control. Do
> as you wish; but, I would plead with you to avoid ANY chance you can
> for the user to make a bonehead stupid move and lose all his
> treasured disk archives. Doing otherwise is very poor form.
I am pretty confident that larger than 2 TiB disks are fully supported
within AmigaOS 4, as I outlined in my other mail.
So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in
2012 already *just* fine. I even found I had the same questions back
then, and researched it. Which lead to this official article back then:
http://wiki.amigaos.net/wiki/RDB
I am also pretty sure that AmigaOS still uses RDB as partitioning
format. They support MBR. I don�t think AmigaOS 4.1 supports GPT.
Whether to implement that of course is the decision of AmigaOS 4
development team. I am no longer a member of it since some time.
Linux m68k should already be able to use disks in GPT format, but you
likely won�t be able to read them in AmigaOS, unless there is some third
party support for it meanwhile.
Thanks,
Martin
>
> {o.o} Joanne "Said enough, she has" Dow
>
> On 20180626 18:07, Michael Schmitz wrote:
> > Joanne,
> >
> > As far as I have been able to test, the change is backwards
> > compatible (RDB partitions from an old disk 80 GB disk are still
> > recognized OK). That"s only been done on an emulator though.
> >
> > Your advice about the dangers of using RDB disks that would have
> > failed the current Linux RDB parser on legacy 32 bit systems is well
> > taken though. Maybe Martin can clarify that for me - was the 2 TB
> > disk in question ever used on a 32 bit Amiga system?
> >
> > RDB disk format is meant for legacy use only, so I think we can get
> > away with printing a big fat warning during boot, advising the user
> > that the oversize RDB partition(s) scanned are not compatible with
> > legacy 32 bit AmigaOS. With the proposed fix they will work under
> > both AmigaOS 4.1 and Linux instead of truncating the first
> > overflowing partition at disk end and trashing valid partitions
> > that overlap, which is what Martin was after.
> >
> > If that still seems too risky, we can make the default behaviour to
> > bail out once a potential overflow is detected, and allow the user
> > to
> > override that through a boot parameter. I'd leave that decision up
> > for the code review on linux-block.
> >
> > Two more comments: Linux uses 512 byte block sizes for the partition
> > start and size calculations, so a change of the RDB blocksize to
> > reduce the block counts stored in the RDB would still result in the
> > same overflow. And amiga-fdisk is indeed utterly broken and needs
> > updating (along with probably most legacy m68k partitioners). Adrian
> > has advertised parted as replacement for the old tools - maybe this
> > would make a nice test case for parted?
> >
> > Cheers,
> >
> > Michael
> >
> > On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> >> If it is not backwards compatible I for one would refuse to use it.
> >> And if it still mattered that much to me I'd also generate a
> >> reasonable alternative. Modifying RDBs nay not be even an
> >> approximation of a good idea. You'd discover that as soon as an
> >> RDB uint64_t disk is tasted by a uint32_t only system. If it is
> >> for your personal use then you're entirely free to reject my
> >> advice and are probably smart enough to keep it working for
> >> yourself.
> >>
> >> GPT is probably the right way to go. Preserve the ability to read
> >> RDBs for legacy disks only.
> >>
> >> {^_^}
> >>
> >> On 20180626 01:31, Michael Schmitz wrote:
> >>> Joanne,
> >>>
> >>> I think we all agree that doing 32 bit calculations on 512-byte
> >>> block
> >>> addresses that overflow on disks 2 TB and larger is a bug, causing
> >>> the issues Martin reported. Your patch addresses that by using
> >>> the correct data type for the calculations (as do other partition
> >>> parsers that may have to deal with large disks) and fixes
> >>> Martin's bug, so appears to be the right thing to do.
> >>>
> >>> Using 64 bit data types for disks smaller than 2 TB where
> >>> calculations don't currently overflow is not expected to cause
> >>> new issues, other than enabling use of disk and partitions larger
> >>> than 2 TB (which may have ramifications with filesystems on these
> >>> partitions). So comptibility is preserved.
> >>>
> >>> Forcing larger block sizes might be a good strategy to avoid
> >>> overflow
> >>> issues in filesystems as well, but I can't see how the block size
> >>> stored in the RDB would enforce use of the same block size in
> >>> filesystems. We'll have to rely on the filesystem tools to get
> >>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >>> limitation) so this should allow partitions larger than 2 TB to
> >>> work already (but I suspect Al Viro may have found a few issues
> >>> when he looked at the AFFS code so I won't say more). Anyway
> >>> partitioning tools and filesystems are unrelated to the Linux
> >>> partition parser code which is all we aim to fix in this patch.
> >>>
> >>> If you feel strongly about unknown ramifications of any
> >>> filesystems on partitions larger than 2 TB, say so and I'll have
> >>> the kernel print a warning about these partitions.
> >>>
> >>> I'll get this patch tested on Martin's test case image as well as
> >>> on a RDB image from a disk known to currently work under Linux
> >>> (thanks Geert for the losetup hint). Can't do much more without
> >>> procuring a working Amiga disk image to use with an emulator,
> >>> sorry. The Amiga I plan to use for tests is a long way away from
> >>> my home indeed.
> >>>
> >>> Cheers,
> >>>
> >>> Michael
> >>>
> >>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>> As long as it preserves compatibility it should be OK, I suppose.
> >>>> Personally I'd make any partitioning tool front end gently force
> >>>> the
> >>>> block size towards 8k as the disk size gets larger. The file
> >>>> systems
> >>>> may also run into 2TB issues that are not obvious. An unused
> >>>> blocks
> >>>> list will have to go beyond a uint32_t size, for example. But a
> >>>> block
> >>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
> >>>> under 1% of the disk all by itself. A block bitmap is not quite
> >>>> so bad. {^_-}
> >>>>
> >>>> Just be sure you are aware of all the ramifications when you make
> >>>> a
> >>>> change. I remember thinking about this for awhile and then
> >>>> determining I REALLY did not want to think about it as my brain
> >>>> was getting tied into a gordian knot.
> >>>>
> >>>> {^_^}
> >>>>
> >>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>>
> >>>>> Martin's boot log (including your patch) says:
> >>>>>
> >>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
> >>>>> sdb1
> >>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
> >>>>> 2 spb
> >>>>> 4)
> >>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> >>>>> Attached SCSI disk
> >>>>>
> >>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
> >>>>> 512
> >>>>> byte blocks) and can be worked around by using a different block
> >>>>> size.
> >>>>>
> >>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>> units.
> >>>>> I'll still submit a patch to Jens anyway as this may bite others
> >>>>> yet.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
wrote:
> >>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>> system is
> >>>>>> a famn
> >>>>>> dool.
> >>>>>>
> >>>>>> If memory serves the RDBs think in blocks rather than bytes so
> >>>>>> it
> >>>>>> should
> >>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
> >>>>>> is
> >>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
> >>>>>> in
> >>>>>> block maps.
> >>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>
> >>>>>> {^_^}
> >>>>>>
> >>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>> Hi.
> >>>>>>>
> >>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>> test results at
> >>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>> there, so if
> >>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>
> >>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>
> >>>>>>> I do not care enough about this, in order to motivate myself
> >>>>>>> preparing the a patch from Joanne Dow�s fix.
> >>>>>>>
> >>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>> Sam440ep
> >>>>>>> which
> >>>>>>> I still have in my apartment.
> >>>>>>>
> >>>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>>> TB,
> >>>>>>> unless
> >>>>>>> someone else does.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>> linux-m68k" in
> >>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>> More majordomo info at
> >>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe
> >>>> linux-m68k" in the body of a message to
> >>>> majordomo@vger.kernel.org
> >>>> More majordomo info at
> >>>> http://vger.kernel.org/majordomo-info.html
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe
> >> linux-m68k" in the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-27 8:03 ` Martin Steigerwald
@ 2018-06-28 2:57 ` jdow
2018-06-28 7:40 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
0 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-28 2:57 UTC (permalink / raw)
To: Martin Steigerwald
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
The issue is what happens when one of those disks appears on a 3.1 system.
{^_^}
On 20180627 01:03, Martin Steigerwald wrote:
> Dear Joanne.
>
> jdow - 27.06.18, 08:24:
>> You allergic to using a GPT solution? It will get away from some of
>> the evils that RDB has inherent in it because they are also features?
>> (Loading a filesystem or DriveInit code from RDBs is just asking for
>> a nearly impossible to remove malware infection.) Furthermore, any 32
>> bit system that sees an RDSK block is going to try to translate it.
>> If you add a new RDB format you are going to get bizarre and probably
>> quite destructive results from the mistake. Fail safe is a rather
>> good notion, methinks.
>>
>> Personally I figure this is all rather surreal. 2TG of junk on an
>> Amiga system seems utterly outlandish to me. You cited another
>> overflow potential. There are at least three we've identified, I
>> believe. Are you 100% sure there are no more? The specific one you
>> mention of translating RDB to Linux has a proper solution in the RDB
>> reader. It should recover such overflow errors in the RDB as it can
>> with due care and polish. It should flag any other overflow error it
>> detects within the RDBs and return an error such as to leave the disk
>> unmounted or mounted read-only if you feel like messing up a poor
>> sod's backups. The simple solution is to read each of the variables
>> with the nominal RDB size and convert it to uint64_t before
>> calculating byte indices.
>>
>> However, consider my inputs as advice from an adult who has seen the
>> Amiga Elephant so to speak. I am not trying to assert any control. Do
>> as you wish; but, I would plead with you to avoid ANY chance you can
>> for the user to make a bonehead stupid move and lose all his
>> treasured disk archives. Doing otherwise is very poor form.
>
> I am pretty confident that larger than 2 TiB disks are fully supported
> within AmigaOS 4, as I outlined in my other mail.
>
> So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in
> 2012 already *just* fine. I even found I had the same questions back
> then, and researched it. Which lead to this official article back then:
>
> http://wiki.amigaos.net/wiki/RDB
>
> I am also pretty sure that AmigaOS still uses RDB as partitioning
> format. They support MBR. I don´t think AmigaOS 4.1 supports GPT.
> Whether to implement that of course is the decision of AmigaOS 4
> development team. I am no longer a member of it since some time.
>
> Linux m68k should already be able to use disks in GPT format, but you
> likely won´t be able to read them in AmigaOS, unless there is some third
> party support for it meanwhile.
>
> Thanks,
> Martin
>
>>
>> {o.o} Joanne "Said enough, she has" Dow
>>
>> On 20180626 18:07, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> As far as I have been able to test, the change is backwards
>>> compatible (RDB partitions from an old disk 80 GB disk are still
>>> recognized OK). That"s only been done on an emulator though.
>>>
>>> Your advice about the dangers of using RDB disks that would have
>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>> taken though. Maybe Martin can clarify that for me - was the 2 TB
>>> disk in question ever used on a 32 bit Amiga system?
>>>
>>> RDB disk format is meant for legacy use only, so I think we can get
>>> away with printing a big fat warning during boot, advising the user
>>> that the oversize RDB partition(s) scanned are not compatible with
>>> legacy 32 bit AmigaOS. With the proposed fix they will work under
>>> both AmigaOS 4.1 and Linux instead of truncating the first
>>> overflowing partition at disk end and trashing valid partitions
>>> that overlap, which is what Martin was after.
>>>
>>> If that still seems too risky, we can make the default behaviour to
>>> bail out once a potential overflow is detected, and allow the user
>>> to
>>> override that through a boot parameter. I'd leave that decision up
>>> for the code review on linux-block.
>>>
>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>> start and size calculations, so a change of the RDB blocksize to
>>> reduce the block counts stored in the RDB would still result in the
>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>> updating (along with probably most legacy m68k partitioners). Adrian
>>> has advertised parted as replacement for the old tools - maybe this
>>> would make a nice test case for parted?
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>>> If it is not backwards compatible I for one would refuse to use it.
>>>> And if it still mattered that much to me I'd also generate a
>>>> reasonable alternative. Modifying RDBs nay not be even an
>>>> approximation of a good idea. You'd discover that as soon as an
>>>> RDB uint64_t disk is tasted by a uint32_t only system. If it is
>>>> for your personal use then you're entirely free to reject my
>>>> advice and are probably smart enough to keep it working for
>>>> yourself.
>>>>
>>>> GPT is probably the right way to go. Preserve the ability to read
>>>> RDBs for legacy disks only.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>> Joanne,
>>>>>
>>>>> I think we all agree that doing 32 bit calculations on 512-byte
>>>>> block
>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>> the issues Martin reported. Your patch addresses that by using
>>>>> the correct data type for the calculations (as do other partition
>>>>> parsers that may have to deal with large disks) and fixes
>>>>> Martin's bug, so appears to be the right thing to do.
>>>>>
>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>> calculations don't currently overflow is not expected to cause
>>>>> new issues, other than enabling use of disk and partitions larger
>>>>> than 2 TB (which may have ramifications with filesystems on these
>>>>> partitions). So comptibility is preserved.
>>>>>
>>>>> Forcing larger block sizes might be a good strategy to avoid
>>>>> overflow
>>>>> issues in filesystems as well, but I can't see how the block size
>>>>> stored in the RDB would enforce use of the same block size in
>>>>> filesystems. We'll have to rely on the filesystem tools to get
>>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
>>>>> limitation) so this should allow partitions larger than 2 TB to
>>>>> work already (but I suspect Al Viro may have found a few issues
>>>>> when he looked at the AFFS code so I won't say more). Anyway
>>>>> partitioning tools and filesystems are unrelated to the Linux
>>>>> partition parser code which is all we aim to fix in this patch.
>>>>>
>>>>> If you feel strongly about unknown ramifications of any
>>>>> filesystems on partitions larger than 2 TB, say so and I'll have
>>>>> the kernel print a warning about these partitions.
>>>>>
>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>> on a RDB image from a disk known to currently work under Linux
>>>>> (thanks Geert for the losetup hint). Can't do much more without
>>>>> procuring a working Amiga disk image to use with an emulator,
>>>>> sorry. The Amiga I plan to use for tests is a long way away from
>>>>> my home indeed.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Michael
>>>>>
>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>> Personally I'd make any partitioning tool front end gently force
>>>>>> the
>>>>>> block size towards 8k as the disk size gets larger. The file
>>>>>> systems
>>>>>> may also run into 2TB issues that are not obvious. An unused
>>>>>> blocks
>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>> block
>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>> under 1% of the disk all by itself. A block bitmap is not quite
>>>>>> so bad. {^_-}
>>>>>>
>>>>>> Just be sure you are aware of all the ramifications when you make
>>>>>> a
>>>>>> change. I remember thinking about this for awhile and then
>>>>>> determining I REALLY did not want to think about it as my brain
>>>>>> was getting tied into a gordian knot.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>> Joanne,
>>>>>>>
>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
>>>>>>> sdb1
>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>> 2 spb
>>>>>>> 4)
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>> Attached SCSI disk
>>>>>>>
>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>> 512
>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>> size.
>>>>>>>
>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
>>>>>>> units.
>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>> yet.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Michael
>>>>>>>
>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
> wrote:
>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>> system is
>>>>>>>> a famn
>>>>>>>> dool.
>>>>>>>>
>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so
>>>>>>>> it
>>>>>>>> should
>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
>>>>>>>> is
>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
>>>>>>>> in
>>>>>>>> block maps.
>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>>
>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>> test results at
>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
>>>>>>>>>> there, so if
>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>
>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>
>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>>>>>
>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
>>>>>>>>> Sam440ep
>>>>>>>>> which
>>>>>>>>> I still have in my apartment.
>>>>>>>>>
>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
>>>>>>>>> TB,
>>>>>>>>> unless
>>>>>>>>> someone else does.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at
>>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in the body of a message to
>>>>>> majordomo@vger.kernel.org
>>>>>> More majordomo info at
>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)
2018-06-28 2:57 ` jdow
@ 2018-06-28 7:40 ` Martin Steigerwald
0 siblings, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28 7:40 UTC (permalink / raw)
To: jdow
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Changing subject, so that there is at least a chance for someone to find
this discussions with a search engine :)
Joanne,
jdow - 28.06.18, 04:57:
> The issue is what happens when one of those disks appears on a 3.1
> system. {^_^}
That is right, so I think the warning about 64 bit support in native OS
is okay, but that issue already exists *without* Linux.
Remember, I created that RDB with Media Toolbox on AmigaOS 4.0. I did
not even use Linux to create that beast :). If it booms in AmigaOS < 4
without NSD64, TD64 or SCSI direct, that would happen with or without
the warning in Linux, even without the disk ever have been seen by a
Linux kernel.
I´d say the warning about support in native OS does not harm, even when
it is about educating Amiga users who, in case they use that large
drives – and I pretty much bet, some of them do –, better know the
limitations beforehand.
I do think the extra kernel option does not make all that much sense,
but I accept it anyway.
Cause if you argue like that, what would need fixing is AmigaOS < 4
without NSD64, TD64 or SCSI direct, but then that is what NSD64 and TD64
was made for more than 10 years ago.
Of course, if a partitioning tool for Linux ever allows to create such
an RDB, it makes sense to add a big fat warning about that. As… I think
would make sense to have in Media Toolbox and AmigaOS partitioning
tools.
However Linux here just reads the RDB, so I´d personally go with the
warning about support in native OS, but spare myself the extra kernel
option stuff.
It is Michael´s call tough, as he submits the patch. And if he chooses
to be on a safer side than this, that is fine with me.
Thanks,
Martin
> On 20180627 01:03, Martin Steigerwald wrote:
> > Dear Joanne.
> >
> > jdow - 27.06.18, 08:24:
> >> You allergic to using a GPT solution? It will get away from some of
> >> the evils that RDB has inherent in it because they are also
> >> features?
> >> (Loading a filesystem or DriveInit code from RDBs is just asking
> >> for
> >> a nearly impossible to remove malware infection.) Furthermore, any
> >> 32
> >> bit system that sees an RDSK block is going to try to translate it.
> >> If you add a new RDB format you are going to get bizarre and
> >> probably
> >> quite destructive results from the mistake. Fail safe is a rather
> >> good notion, methinks.
> >>
> >> Personally I figure this is all rather surreal. 2TG of junk on an
> >> Amiga system seems utterly outlandish to me. You cited another
> >> overflow potential. There are at least three we've identified, I
> >> believe. Are you 100% sure there are no more? The specific one you
> >> mention of translating RDB to Linux has a proper solution in the
> >> RDB
> >> reader. It should recover such overflow errors in the RDB as it can
> >> with due care and polish. It should flag any other overflow error
> >> it
> >> detects within the RDBs and return an error such as to leave the
> >> disk
> >> unmounted or mounted read-only if you feel like messing up a poor
> >> sod's backups. The simple solution is to read each of the variables
> >> with the nominal RDB size and convert it to uint64_t before
> >> calculating byte indices.
> >>
> >> However, consider my inputs as advice from an adult who has seen
> >> the
> >> Amiga Elephant so to speak. I am not trying to assert any control.
> >> Do
> >> as you wish; but, I would plead with you to avoid ANY chance you
> >> can
> >> for the user to make a bonehead stupid move and lose all his
> >> treasured disk archives. Doing otherwise is very poor form.
> >
> > I am pretty confident that larger than 2 TiB disks are fully
> > supported within AmigaOS 4, as I outlined in my other mail.
> >
> > So with all due respect: I used a larger than 2 TiB disk in AmigaOS
> > 4 in 2012 already *just* fine. I even found I had the same
> > questions back then, and researched it. Which lead to this official
> > article back then:
> >
> > http://wiki.amigaos.net/wiki/RDB
> >
> > I am also pretty sure that AmigaOS still uses RDB as partitioning
> > format. They support MBR. I don´t think AmigaOS 4.1 supports GPT.
> > Whether to implement that of course is the decision of AmigaOS 4
> > development team. I am no longer a member of it since some time.
> >
> > Linux m68k should already be able to use disks in GPT format, but
> > you
> > likely won´t be able to read them in AmigaOS, unless there is some
> > third party support for it meanwhile.
> >
> > Thanks,
> > Martin
> >
> >> {o.o} Joanne "Said enough, she has" Dow
> >>
> >> On 20180626 18:07, Michael Schmitz wrote:
> >>> Joanne,
> >>>
> >>> As far as I have been able to test, the change is backwards
> >>> compatible (RDB partitions from an old disk 80 GB disk are still
> >>> recognized OK). That"s only been done on an emulator though.
> >>>
> >>> Your advice about the dangers of using RDB disks that would have
> >>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>> well
> >>> taken though. Maybe Martin can clarify that for me - was the 2 TB
> >>> disk in question ever used on a 32 bit Amiga system?
> >>>
> >>> RDB disk format is meant for legacy use only, so I think we can
> >>> get
> >>> away with printing a big fat warning during boot, advising the
> >>> user
> >>> that the oversize RDB partition(s) scanned are not compatible with
> >>> legacy 32 bit AmigaOS. With the proposed fix they will work under
> >>> both AmigaOS 4.1 and Linux instead of truncating the first
> >>> overflowing partition at disk end and trashing valid partitions
> >>> that overlap, which is what Martin was after.
> >>>
> >>> If that still seems too risky, we can make the default behaviour
> >>> to
> >>> bail out once a potential overflow is detected, and allow the user
> >>> to
> >>> override that through a boot parameter. I'd leave that decision up
> >>> for the code review on linux-block.
> >>>
> >>> Two more comments: Linux uses 512 byte block sizes for the
> >>> partition
> >>> start and size calculations, so a change of the RDB blocksize to
> >>> reduce the block counts stored in the RDB would still result in
> >>> the
> >>> same overflow. And amiga-fdisk is indeed utterly broken and needs
> >>> updating (along with probably most legacy m68k partitioners).
> >>> Adrian
> >>> has advertised parted as replacement for the old tools - maybe
> >>> this
> >>> would make a nice test case for parted?
> >>>
> >>> Cheers,
> >>>
> >>> Michael
> >>>
> >>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> >>>> If it is not backwards compatible I for one would refuse to use
> >>>> it.
> >>>> And if it still mattered that much to me I'd also generate a
> >>>> reasonable alternative. Modifying RDBs nay not be even an
> >>>> approximation of a good idea. You'd discover that as soon as an
> >>>> RDB uint64_t disk is tasted by a uint32_t only system. If it is
> >>>> for your personal use then you're entirely free to reject my
> >>>> advice and are probably smart enough to keep it working for
> >>>> yourself.
> >>>>
> >>>> GPT is probably the right way to go. Preserve the ability to read
> >>>> RDBs for legacy disks only.
> >>>>
> >>>> {^_^}
> >>>>
> >>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>>
> >>>>> I think we all agree that doing 32 bit calculations on 512-byte
> >>>>> block
> >>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>> causing
> >>>>> the issues Martin reported. Your patch addresses that by using
> >>>>> the correct data type for the calculations (as do other
> >>>>> partition
> >>>>> parsers that may have to deal with large disks) and fixes
> >>>>> Martin's bug, so appears to be the right thing to do.
> >>>>>
> >>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>> calculations don't currently overflow is not expected to cause
> >>>>> new issues, other than enabling use of disk and partitions
> >>>>> larger
> >>>>> than 2 TB (which may have ramifications with filesystems on
> >>>>> these
> >>>>> partitions). So comptibility is preserved.
> >>>>>
> >>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>> overflow
> >>>>> issues in filesystems as well, but I can't see how the block
> >>>>> size
> >>>>> stored in the RDB would enforce use of the same block size in
> >>>>> filesystems. We'll have to rely on the filesystem tools to get
> >>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >>>>> limitation) so this should allow partitions larger than 2 TB to
> >>>>> work already (but I suspect Al Viro may have found a few issues
> >>>>> when he looked at the AFFS code so I won't say more). Anyway
> >>>>> partitioning tools and filesystems are unrelated to the Linux
> >>>>> partition parser code which is all we aim to fix in this patch.
> >>>>>
> >>>>> If you feel strongly about unknown ramifications of any
> >>>>> filesystems on partitions larger than 2 TB, say so and I'll have
> >>>>> the kernel print a warning about these partitions.
> >>>>>
> >>>>> I'll get this patch tested on Martin's test case image as well
> >>>>> as
> >>>>> on a RDB image from a disk known to currently work under Linux
> >>>>> (thanks Geert for the losetup hint). Can't do much more without
> >>>>> procuring a working Amiga disk image to use with an emulator,
> >>>>> sorry. The Amiga I plan to use for tests is a long way away from
> >>>>> my home indeed.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>> suppose.
> >>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>> force
> >>>>>> the
> >>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>> systems
> >>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>> blocks
> >>>>>> list will have to go beyond a uint32_t size, for example. But a
> >>>>>> block
> >>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>> tad
> >>>>>> under 1% of the disk all by itself. A block bitmap is not quite
> >>>>>> so bad. {^_-}
> >>>>>>
> >>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>> make
> >>>>>> a
> >>>>>> change. I remember thinking about this for awhile and then
> >>>>>> determining I REALLY did not want to think about it as my brain
> >>>>>> was getting tied into a gordian knot.
> >>>>>>
> >>>>>> {^_^}
> >>>>>>
> >>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>> Joanne,
> >>>>>>>
> >>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>
> >>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK
> >>>>>>> (512)
> >>>>>>> sdb1
> >>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>> (DOS^C)(res
> >>>>>>> 2 spb
> >>>>>>> 4)
> >>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>> [sdb]
> >>>>>>> Attached SCSI disk
> >>>>>>>
> >>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>> means
> >>>>>>> 512
> >>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>> block
> >>>>>>> size.
> >>>>>>>
> >>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>> units.
> >>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>> others
> >>>>>>> yet.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Michael
> >>>>>>>
> >>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
> >
> > wrote:
> >>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>> system is
> >>>>>>>> a famn
> >>>>>>>> dool.
> >>>>>>>>
> >>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>> so
> >>>>>>>> it
> >>>>>>>> should
> >>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>> blocks
> >>>>>>>> is
> >>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
> >>>>>>>> in
> >>>>>>>> block maps.
> >>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>
> >>>>>>>> {^_^}
> >>>>>>>>
> >>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>> Hi.
> >>>>>>>>>
> >>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>> test results at
> >>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>> there, so if
> >>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>
> >>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>
> >>>>>>>>> I do not care enough about this, in order to motivate myself
> >>>>>>>>> preparing the a patch from Joanne Dow´s fix.
> >>>>>>>>>
> >>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>> Sam440ep
> >>>>>>>>> which
> >>>>>>>>> I still have in my apartment.
> >>>>>>>>>
> >>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>>>>> TB,
> >>>>>>>>> unless
> >>>>>>>>> someone else does.
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>>> linux-m68k" in
> >>>>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>>>> More majordomo info at
> >>>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>>>
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>> linux-m68k" in the body of a message to
> >>>>>> majordomo@vger.kernel.org
> >>>>>> More majordomo info at
> >>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe
> >>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
> >>>> More majordomo info at
> >>>> http://vger.kernel.org/majordomo-info.html
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-27 6:24 ` jdow
2018-06-27 8:03 ` Martin Steigerwald
@ 2018-06-27 9:00 ` Michael Schmitz
2018-06-28 3:44 ` jdow
1 sibling, 1 reply; 73+ messages in thread
From: Michael Schmitz @ 2018-06-27 9:00 UTC (permalink / raw)
To: jdow
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Joanne,
I'm not at all allergic to avoiding RDB at all cost for new disks. If
AmigaOS 4.1 supports more recent partition formats, all the better. This
is all about supporting use of legacy RDB disks on Linux (though 2 TB
does stretch the definition of 'legacy' a little). My interest in this
is to ensure we can continue to use RDB format disks on m68k Amiga
computers which have no other way to boot Linux from disk.
Not proposing to change the RDB format at all, either. Just trying to
make sure we translate RDB info into Linux 512-byte block offset and
size numbers correctly. The kernel won't modify the RDB at all
(intentionally, that is - with the correct choice of partition sizes,
Martin might well have wiped out his RDB with the current version of the
parser).
The choice of refusing to mount a disk (or mounting read-only) rests
with the VFS drivers alone - AFFS in that case. Not touching any of
that. At partition scan time, we only have the option of making the
partition available (with a warning printed), or refusing to make it
available to the kernel. Once it's made available, all bets are off.
From what Martin writes, his test case RDB was valid and worked as
expected on 32 bit AmigaOS (4.1). Apparently, that version has the
necessary extensions to handle the large offsets resulting from 2 TB
disks. Not sure what safeguards are in place when connecting such a disk
to older versions of AmigaOS, but that is a different matter entirely.
The overflows in partition offset and size are the only ones I can see
in the partition parser - there is no other overflow I've identified. I
just stated that in order to place a partition towards the end of a 2 TB
disk, the offset calculation will overflow regardless of what
combination of rdb->rdb_BlockBytes and sector addresses stored in the
RDB (in units of 512 byte blocks) we use:
blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
be32_to_cpu(pb->pb_Environment[9])) *
be32_to_cpu(pb->pb_Environment[3]) *
be32_to_cpu(pb->pb_Environment[5]) *
blksize;
if (!nr_sects)
continue;
start_sect = be32_to_cpu(pb->pb_Environment[9]) *
be32_to_cpu(pb->pb_Environment[3]) *
be32_to_cpu(pb->pb_Environment[5]) *
blksize;
But in the interest of avoiding any accidental use of a RDB partition
where calculations currently overflow, I'll make the default behaviour
to bail out (instead of using wrong offset or size as we currently do).
Given the 'eat_my_RDB_disk=1' boot option, the user may proceed at their
own risk (though I still can't see what harm should result from now
translating a well formed v4.1 2 TB disk RDB correctly for the first time).
Whether or not Linux correctly handles AFFS filesystems larger than 1 TB
is a matter for VFS experts. Bailing out on nr_sects overflowing ought
to prevent accidental use of AFFS filesystems on RDB disks which I
suppose is the majority of use cases.
Bugs in partitioning tools on Linux are entirely out of scope - the
partitioning tools bypass the partition structure discovered by the
kernel, and work straight on the raw device. No protecting against that.
If you can point out a way to cause data loss with these precautions,
for a disk 2 TB or larger that was partitioned and used on a recent
version or AmigaOS supporting such large disks, I'd consider omitting
the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
option.
Cheers,
Michael
Am 27.06.2018 um 18:24 schrieb jdow:
> You allergic to using a GPT solution? It will get away from some of the
> evils that RDB has inherent in it because they are also features?
> (Loading a filesystem or DriveInit code from RDBs is just asking for a
> nearly impossible to remove malware infection.) Furthermore, any 32 bit
> system that sees an RDSK block is going to try to translate it. If you
> add a new RDB format you are going to get bizarre and probably quite
> destructive results from the mistake. Fail safe is a rather good notion,
> methinks.
>
> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
> system seems utterly outlandish to me. You cited another overflow
> potential. There are at least three we've identified, I believe. Are you
> 100% sure there are no more? The specific one you mention of translating
> RDB to Linux has a proper solution in the RDB reader. It should recover
> such overflow errors in the RDB as it can with due care and polish. It
> should flag any other overflow error it detects within the RDBs and
> return an error such as to leave the disk unmounted or mounted read-only
> if you feel like messing up a poor sod's backups. The simple solution is
> to read each of the variables with the nominal RDB size and convert it
> to uint64_t before calculating byte indices.
>
> However, consider my inputs as advice from an adult who has seen the
> Amiga Elephant so to speak. I am not trying to assert any control. Do as
> you wish; but, I would plead with you to avoid ANY chance you can for
> the user to make a bonehead stupid move and lose all his treasured disk
> archives. Doing otherwise is very poor form.
>
> {o.o} Joanne "Said enough, she has" Dow
>
> On 20180626 18:07, Michael Schmitz wrote:
>> Joanne,
>>
>> As far as I have been able to test, the change is backwards compatible
>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>> That"s only been done on an emulator though.
>>
>> Your advice about the dangers of using RDB disks that would have
>> failed the current Linux RDB parser on legacy 32 bit systems is well
>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>> in question ever used on a 32 bit Amiga system?
>>
>> RDB disk format is meant for legacy use only, so I think we can get
>> away with printing a big fat warning during boot, advising the user
>> that the oversize RDB partition(s) scanned are not compatible with
>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>> partition at disk end and trashing valid partitions that overlap,
>> which is what Martin was after.
>>
>> If that still seems too risky, we can make the default behaviour to
>> bail out once a potential overflow is detected, and allow the user to
>> override that through a boot parameter. I'd leave that decision up for
>> the code review on linux-block.
>>
>> Two more comments: Linux uses 512 byte block sizes for the partition
>> start and size calculations, so a change of the RDB blocksize to
>> reduce the block counts stored in the RDB would still result in the
>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>> updating (along with probably most legacy m68k partitioners). Adrian
>> has advertised parted as replacement for the old tools - maybe this
>> would make a nice test case for parted?
>>
>> Cheers,
>>
>> Michael
>>
>>
>>
>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>> If it is not backwards compatible I for one would refuse to use it.
>>> And if
>>> it still mattered that much to me I'd also generate a reasonable
>>> alternative. Modifying RDBs nay not be even an approximation of a
>>> good idea.
>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>> uint32_t
>>> only system. If it is for your personal use then you're entirely free to
>>> reject my advice and are probably smart enough to keep it working for
>>> yourself.
>>>
>>> GPT is probably the right way to go. Preserve the ability to read
>>> RDBs for
>>> legacy disks only.
>>>
>>> {^_^}
>>>
>>>
>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>
>>>> Joanne,
>>>>
>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>>> issues Martin reported. Your patch addresses that by using the correct
>>>> data type for the calculations (as do other partition parsers that may
>>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>>> the right thing to do.
>>>>
>>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>>> don't currently overflow is not expected to cause new issues, other
>>>> than
>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>> ramifications with filesystems on these partitions). So comptibility is
>>>> preserved.
>>>>
>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>> issues in filesystems as well, but I can't see how the block size
>>>> stored
>>>> in the RDB would enforce use of the same block size in filesystems.
>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>> Linux
>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>> Viro
>>>> may have found a few issues when he looked at the AFFS code so I won't
>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>> the Linux partition parser code which is all we aim to fix in this
>>>> patch.
>>>>
>>>> If you feel strongly about unknown ramifications of any filesystems on
>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>> warning about these partitions.
>>>>
>>>> I'll get this patch tested on Martin's test case image as well as on a
>>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>>> for the losetup hint). Can't do much more without procuring a working
>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>> use
>>>> for tests is a long way away from my home indeed.
>>>>
>>>> Cheers,
>>>>
>>>> Michael
>>>>
>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>
>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>> {^_-}
>>>>>
>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>> change. I remember thinking about this for awhile and then determining
>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>> into a gordian knot.
>>>>>
>>>>> {^_^}
>>>>>
>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>
>>>>>> Joanne,
>>>>>>
>>>>>> Martin's boot log (including your patch) says:
>>>>>>
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>>> 4)
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>> Attached SCSI disk
>>>>>>
>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>>> byte blocks) and can be worked around by using a different block
>>>>>> size.
>>>>>>
>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>>
>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>>> a famn
>>>>>>> dool.
>>>>>>>
>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>> should
>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>> block maps.
>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>
>>>>>>> {^_^}
>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>> so if
>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>
>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>> preparing
>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>
>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>> which
>>>>>>>> I still have in my apartment.
>>>>>>>>
>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>> unless
>>>>>>>> someone else does.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-27 9:00 ` moving affs + RDB partition support to staging? Michael Schmitz
@ 2018-06-28 3:44 ` jdow
2018-06-28 5:43 ` Michael Schmitz
0 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-28 3:44 UTC (permalink / raw)
To: Michael Schmitz
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Michael, as long as m68k only supports int fseek( int ) 4 GB * block size is
your HARD limit. Versions that support __int64 fseek64( __int64 ) can work with
larger disks. RDBs could include RDSK and a new set of other blocks that replace
the last two characters with "64" and use __int64 where needed in the various
values. That way a clever disk partitioner could give allow normal (32 bit) RDB
definitions where possible. Then at least SOME of the disk could be supported
AND a very clever filesystem that abstracts very large disks properly could give
access to the whole disk. (Read the RDBs first 32 bits. Then if a filesystem or
driveinit was loaded re-read the RDBs to see if new 64 bit partitions are revealed.
I could be wrong but I do not think RDBs could be safely modified any other way
to work. And, trust me as I bet this is still true, you will need a SERIOUSLY
good bomb shelter on the Moon if you change RDBs. Suppose Joe Amigoid uses it,
and then Joe Amigoid loads Amigados 2.4 because he wants to run a game that
crashes on anything newer. Then Joe got far enough something writes to the disk
and data is corrupted. Note further that Amigoids do NOT, repeat NOT, listen to
facts in such cases. Hell, some of them never listened to facts about an
incident at Jerry Pournelle's place when a 1.1 DPaint session with Kelly Freas
hung and we lost a delightful drawing. Jerry reported it. Amigoids screamed. I
tried to tell them I was there, it was my machine, and 1.1 was, indeed, crap.
{o.o}
On 20180627 02:00, Michael Schmitz wrote:
> Joanne,
>
> I'm not at all allergic to avoiding RDB at all cost for new disks. If AmigaOS
> 4.1 supports more recent partition formats, all the better. This is all about
> supporting use of legacy RDB disks on Linux (though 2 TB does stretch the
> definition of 'legacy' a little). My interest in this is to ensure we can
> continue to use RDB format disks on m68k Amiga computers which have no other way
> to boot Linux from disk.
>
> Not proposing to change the RDB format at all, either. Just trying to make sure
> we translate RDB info into Linux 512-byte block offset and size numbers
> correctly. The kernel won't modify the RDB at all (intentionally, that is - with
> the correct choice of partition sizes, Martin might well have wiped out his RDB
> with the current version of the parser).
>
> The choice of refusing to mount a disk (or mounting read-only) rests with the
> VFS drivers alone - AFFS in that case. Not touching any of that. At partition
> scan time, we only have the option of making the partition available (with a
> warning printed), or refusing to make it available to the kernel. Once it's made
> available, all bets are off.
>
> From what Martin writes, his test case RDB was valid and worked as expected on
> 32 bit AmigaOS (4.1). Apparently, that version has the necessary extensions to
> handle the large offsets resulting from 2 TB disks. Not sure what safeguards are
> in place when connecting such a disk to older versions of AmigaOS, but that is a
> different matter entirely.
>
> The overflows in partition offset and size are the only ones I can see in the
> partition parser - there is no other overflow I've identified. I just stated
> that in order to place a partition towards the end of a 2 TB disk, the offset
> calculation will overflow regardless of what combination of rdb->rdb_BlockBytes
> and sector addresses stored in the RDB (in units of 512 byte blocks) we use:
>
> blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
>
>
> nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
> be32_to_cpu(pb->pb_Environment[9])) *
> be32_to_cpu(pb->pb_Environment[3]) *
> be32_to_cpu(pb->pb_Environment[5]) *
> blksize;
> if (!nr_sects)
> continue;
> start_sect = be32_to_cpu(pb->pb_Environment[9]) *
> be32_to_cpu(pb->pb_Environment[3]) *
> be32_to_cpu(pb->pb_Environment[5]) *
> blksize;
>
> But in the interest of avoiding any accidental use of a RDB partition where
> calculations currently overflow, I'll make the default behaviour to bail out
> (instead of using wrong offset or size as we currently do). Given the
> 'eat_my_RDB_disk=1' boot option, the user may proceed at their own risk (though
> I still can't see what harm should result from now translating a well formed
> v4.1 2 TB disk RDB correctly for the first time).
>
> Whether or not Linux correctly handles AFFS filesystems larger than 1 TB is a
> matter for VFS experts. Bailing out on nr_sects overflowing ought to prevent
> accidental use of AFFS filesystems on RDB disks which I suppose is the majority
> of use cases.
>
> Bugs in partitioning tools on Linux are entirely out of scope - the partitioning
> tools bypass the partition structure discovered by the kernel, and work straight
> on the raw device. No protecting against that.
>
> If you can point out a way to cause data loss with these precautions, for a disk
> 2 TB or larger that was partitioned and used on a recent version or AmigaOS
> supporting such large disks, I'd consider omitting the 'eat_my_RDB_disk' boot
> option, and just bail out as the only safe option.
>
> Cheers,
>
> Michael
>
>
> Am 27.06.2018 um 18:24 schrieb jdow:
>> You allergic to using a GPT solution? It will get away from some of the
>> evils that RDB has inherent in it because they are also features?
>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>> system that sees an RDSK block is going to try to translate it. If you
>> add a new RDB format you are going to get bizarre and probably quite
>> destructive results from the mistake. Fail safe is a rather good notion,
>> methinks.
>>
>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>> system seems utterly outlandish to me. You cited another overflow
>> potential. There are at least three we've identified, I believe. Are you
>> 100% sure there are no more? The specific one you mention of translating
>> RDB to Linux has a proper solution in the RDB reader. It should recover
>> such overflow errors in the RDB as it can with due care and polish. It
>> should flag any other overflow error it detects within the RDBs and
>> return an error such as to leave the disk unmounted or mounted read-only
>> if you feel like messing up a poor sod's backups. The simple solution is
>> to read each of the variables with the nominal RDB size and convert it
>> to uint64_t before calculating byte indices.
>>
>> However, consider my inputs as advice from an adult who has seen the
>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>> you wish; but, I would plead with you to avoid ANY chance you can for
>> the user to make a bonehead stupid move and lose all his treasured disk
>> archives. Doing otherwise is very poor form.
>>
>> {o.o} Joanne "Said enough, she has" Dow
>>
>> On 20180626 18:07, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> As far as I have been able to test, the change is backwards compatible
>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>> That"s only been done on an emulator though.
>>>
>>> Your advice about the dangers of using RDB disks that would have
>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>> in question ever used on a 32 bit Amiga system?
>>>
>>> RDB disk format is meant for legacy use only, so I think we can get
>>> away with printing a big fat warning during boot, advising the user
>>> that the oversize RDB partition(s) scanned are not compatible with
>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>> partition at disk end and trashing valid partitions that overlap,
>>> which is what Martin was after.
>>>
>>> If that still seems too risky, we can make the default behaviour to
>>> bail out once a potential overflow is detected, and allow the user to
>>> override that through a boot parameter. I'd leave that decision up for
>>> the code review on linux-block.
>>>
>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>> start and size calculations, so a change of the RDB blocksize to
>>> reduce the block counts stored in the RDB would still result in the
>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>> updating (along with probably most legacy m68k partitioners). Adrian
>>> has advertised parted as replacement for the old tools - maybe this
>>> would make a nice test case for parted?
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>>
>>>
>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>>> If it is not backwards compatible I for one would refuse to use it.
>>>> And if
>>>> it still mattered that much to me I'd also generate a reasonable
>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>> good idea.
>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>> uint32_t
>>>> only system. If it is for your personal use then you're entirely free to
>>>> reject my advice and are probably smart enough to keep it working for
>>>> yourself.
>>>>
>>>> GPT is probably the right way to go. Preserve the ability to read
>>>> RDBs for
>>>> legacy disks only.
>>>>
>>>> {^_^}
>>>>
>>>>
>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>
>>>>> Joanne,
>>>>>
>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>>>> issues Martin reported. Your patch addresses that by using the correct
>>>>> data type for the calculations (as do other partition parsers that may
>>>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>>>> the right thing to do.
>>>>>
>>>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>>>> don't currently overflow is not expected to cause new issues, other
>>>>> than
>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>> ramifications with filesystems on these partitions). So comptibility is
>>>>> preserved.
>>>>>
>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>> issues in filesystems as well, but I can't see how the block size
>>>>> stored
>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>> Linux
>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>> Viro
>>>>> may have found a few issues when he looked at the AFFS code so I won't
>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>> patch.
>>>>>
>>>>> If you feel strongly about unknown ramifications of any filesystems on
>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>> warning about these partitions.
>>>>>
>>>>> I'll get this patch tested on Martin's test case image as well as on a
>>>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>> use
>>>>> for tests is a long way away from my home indeed.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Michael
>>>>>
>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>
>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>> {^_-}
>>>>>>
>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>> change. I remember thinking about this for awhile and then determining
>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>> into a gordian knot.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>
>>>>>>> Joanne,
>>>>>>>
>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>>>> 4)
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>> Attached SCSI disk
>>>>>>>
>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>> size.
>>>>>>>
>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Michael
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>>>
>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>>>> a famn
>>>>>>>> dool.
>>>>>>>>
>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>> should
>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>> block maps.
>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>> so if
>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>
>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>> preparing
>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>
>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>> which
>>>>>>>>> I still have in my apartment.
>>>>>>>>>
>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>> unless
>>>>>>>>> someone else does.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-28 3:44 ` jdow
@ 2018-06-28 5:43 ` Michael Schmitz
2018-06-28 6:39 ` jdow
2018-06-28 8:07 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
0 siblings, 2 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-28 5:43 UTC (permalink / raw)
To: jdow
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Joanne,
Linux on m68k has supported lseek64 (or llseek) for a long time (from
glibc version 2.1 according to what I found). About the only area where
we are limited by 32 bits is the virtual memory size.
I'm not proposing to modify the RDB format definition, though an
extension to store 64 bit offsets separate from the 32 bit ones would be
one way to make certain such disks are safe to use on 3.1 and earlier
versions of AmigaOS. (Another one would be to modify the disk drivers on
older versions to do the offset calculation in 64 bit, and check for
overflow just as we do here. Not sure whether that's feasible. And as
you so eloquently describe, we can't rely on users listening.)
Either way, we need the cooperation of partitioning tool writers to
ensure partition information that is prone to overflows is never stored
in the 32 bit, classic RDB. That appears to have failed already, as
Martin's experience illustrates.
I'm only concerned with fixing the (dangerous) but in the Linux
partition format parser for RDB. Refusing to use any partitions that
will cause havoc on old AmigaOS versions is all I can do to try and get
the users' attention.
Your warning makes me wonder whether the log message should just say
'report this bug to linux-m68k@vger.kernel.org' to at least try and
educate any that respond about the dangers of their partitioning scheme
before telling them about the override option. Problem with that is, in
about three years no one will remember any of this ...
Cheers,
Michael
Am 28.06.2018 um 15:44 schrieb jdow:
> Michael, as long as m68k only supports int fseek( int ) 4 GB * block
> size is your HARD limit. Versions that support __int64 fseek64( __int64
> ) can work with larger disks. RDBs could include RDSK and a new set of
> other blocks that replace the last two characters with "64" and use
> __int64 where needed in the various values. That way a clever disk
> partitioner could give allow normal (32 bit) RDB definitions where
> possible. Then at least SOME of the disk could be supported AND a very
> clever filesystem that abstracts very large disks properly could give
> access to the whole disk. (Read the RDBs first 32 bits. Then if a
> filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit
> partitions are revealed.
>
> I could be wrong but I do not think RDBs could be safely modified any
> other way to work. And, trust me as I bet this is still true, you will
> need a SERIOUSLY good bomb shelter on the Moon if you change RDBs.
> Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4
> because he wants to run a game that crashes on anything newer. Then Joe
> got far enough something writes to the disk and data is corrupted. Note
> further that Amigoids do NOT, repeat NOT, listen to facts in such cases.
> Hell, some of them never listened to facts about an incident at Jerry
> Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we
> lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried
> to tell them I was there, it was my machine, and 1.1 was, indeed, crap.
>
> {o.o}
>
> On 20180627 02:00, Michael Schmitz wrote:
>> Joanne,
>>
>> I'm not at all allergic to avoiding RDB at all cost for new disks. If
>> AmigaOS 4.1 supports more recent partition formats, all the better.
>> This is all about supporting use of legacy RDB disks on Linux (though
>> 2 TB does stretch the definition of 'legacy' a little). My interest in
>> this is to ensure we can continue to use RDB format disks on m68k
>> Amiga computers which have no other way to boot Linux from disk.
>>
>> Not proposing to change the RDB format at all, either. Just trying to
>> make sure we translate RDB info into Linux 512-byte block offset and
>> size numbers correctly. The kernel won't modify the RDB at all
>> (intentionally, that is - with the correct choice of partition sizes,
>> Martin might well have wiped out his RDB with the current version of
>> the parser).
>>
>> The choice of refusing to mount a disk (or mounting read-only) rests
>> with the VFS drivers alone - AFFS in that case. Not touching any of
>> that. At partition scan time, we only have the option of making the
>> partition available (with a warning printed), or refusing to make it
>> available to the kernel. Once it's made available, all bets are off.
>>
>> From what Martin writes, his test case RDB was valid and worked as
>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
>> necessary extensions to handle the large offsets resulting from 2 TB
>> disks. Not sure what safeguards are in place when connecting such a
>> disk to older versions of AmigaOS, but that is a different matter
>> entirely.
>>
>> The overflows in partition offset and size are the only ones I can see
>> in the partition parser - there is no other overflow I've identified.
>> I just stated that in order to place a partition towards the end of a
>> 2 TB disk, the offset calculation will overflow regardless of what
>> combination of rdb->rdb_BlockBytes and sector addresses stored in the
>> RDB (in units of 512 byte blocks) we use:
>>
>> blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
>>
>>
>> nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
>> be32_to_cpu(pb->pb_Environment[9])) *
>> be32_to_cpu(pb->pb_Environment[3]) *
>> be32_to_cpu(pb->pb_Environment[5]) *
>> blksize;
>> if (!nr_sects)
>> continue;
>> start_sect = be32_to_cpu(pb->pb_Environment[9]) *
>> be32_to_cpu(pb->pb_Environment[3]) *
>> be32_to_cpu(pb->pb_Environment[5]) *
>> blksize;
>>
>> But in the interest of avoiding any accidental use of a RDB partition
>> where calculations currently overflow, I'll make the default behaviour
>> to bail out (instead of using wrong offset or size as we currently
>> do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed
>> at their own risk (though I still can't see what harm should result
>> from now translating a well formed v4.1 2 TB disk RDB correctly for
>> the first time).
>>
>> Whether or not Linux correctly handles AFFS filesystems larger than 1
>> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
>> ought to prevent accidental use of AFFS filesystems on RDB disks which
>> I suppose is the majority of use cases.
>>
>> Bugs in partitioning tools on Linux are entirely out of scope - the
>> partitioning tools bypass the partition structure discovered by the
>> kernel, and work straight on the raw device. No protecting against that.
>>
>> If you can point out a way to cause data loss with these precautions,
>> for a disk 2 TB or larger that was partitioned and used on a recent
>> version or AmigaOS supporting such large disks, I'd consider omitting
>> the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
>> option.
>>
>> Cheers,
>>
>> Michael
>>
>>
>> Am 27.06.2018 um 18:24 schrieb jdow:
>>> You allergic to using a GPT solution? It will get away from some of the
>>> evils that RDB has inherent in it because they are also features?
>>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>>> system that sees an RDSK block is going to try to translate it. If you
>>> add a new RDB format you are going to get bizarre and probably quite
>>> destructive results from the mistake. Fail safe is a rather good notion,
>>> methinks.
>>>
>>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>>> system seems utterly outlandish to me. You cited another overflow
>>> potential. There are at least three we've identified, I believe. Are you
>>> 100% sure there are no more? The specific one you mention of translating
>>> RDB to Linux has a proper solution in the RDB reader. It should recover
>>> such overflow errors in the RDB as it can with due care and polish. It
>>> should flag any other overflow error it detects within the RDBs and
>>> return an error such as to leave the disk unmounted or mounted read-only
>>> if you feel like messing up a poor sod's backups. The simple solution is
>>> to read each of the variables with the nominal RDB size and convert it
>>> to uint64_t before calculating byte indices.
>>>
>>> However, consider my inputs as advice from an adult who has seen the
>>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>>> you wish; but, I would plead with you to avoid ANY chance you can for
>>> the user to make a bonehead stupid move and lose all his treasured disk
>>> archives. Doing otherwise is very poor form.
>>>
>>> {o.o} Joanne "Said enough, she has" Dow
>>>
>>> On 20180626 18:07, Michael Schmitz wrote:
>>>> Joanne,
>>>>
>>>> As far as I have been able to test, the change is backwards compatible
>>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>>> That"s only been done on an emulator though.
>>>>
>>>> Your advice about the dangers of using RDB disks that would have
>>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>>> in question ever used on a 32 bit Amiga system?
>>>>
>>>> RDB disk format is meant for legacy use only, so I think we can get
>>>> away with printing a big fat warning during boot, advising the user
>>>> that the oversize RDB partition(s) scanned are not compatible with
>>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>>> partition at disk end and trashing valid partitions that overlap,
>>>> which is what Martin was after.
>>>>
>>>> If that still seems too risky, we can make the default behaviour to
>>>> bail out once a potential overflow is detected, and allow the user to
>>>> override that through a boot parameter. I'd leave that decision up for
>>>> the code review on linux-block.
>>>>
>>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>>> start and size calculations, so a change of the RDB blocksize to
>>>> reduce the block counts stored in the RDB would still result in the
>>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>>> updating (along with probably most legacy m68k partitioners). Adrian
>>>> has advertised parted as replacement for the old tools - maybe this
>>>> would make a nice test case for parted?
>>>>
>>>> Cheers,
>>>>
>>>> Michael
>>>>
>>>>
>>>>
>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>>>> If it is not backwards compatible I for one would refuse to use it.
>>>>> And if
>>>>> it still mattered that much to me I'd also generate a reasonable
>>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>>> good idea.
>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>>> uint32_t
>>>>> only system. If it is for your personal use then you're entirely
>>>>> free to
>>>>> reject my advice and are probably smart enough to keep it working for
>>>>> yourself.
>>>>>
>>>>> GPT is probably the right way to go. Preserve the ability to read
>>>>> RDBs for
>>>>> legacy disks only.
>>>>>
>>>>> {^_^}
>>>>>
>>>>>
>>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>>
>>>>>> Joanne,
>>>>>>
>>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>>> the
>>>>>> issues Martin reported. Your patch addresses that by using the
>>>>>> correct
>>>>>> data type for the calculations (as do other partition parsers that
>>>>>> may
>>>>>> have to deal with large disks) and fixes Martin's bug, so appears
>>>>>> to be
>>>>>> the right thing to do.
>>>>>>
>>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>>> calculations
>>>>>> don't currently overflow is not expected to cause new issues, other
>>>>>> than
>>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>>> ramifications with filesystems on these partitions). So
>>>>>> comptibility is
>>>>>> preserved.
>>>>>>
>>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>>> issues in filesystems as well, but I can't see how the block size
>>>>>> stored
>>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>>> Linux
>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>>> Viro
>>>>>> may have found a few issues when he looked at the AFFS code so I
>>>>>> won't
>>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>>> patch.
>>>>>>
>>>>>> If you feel strongly about unknown ramifications of any
>>>>>> filesystems on
>>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>>> warning about these partitions.
>>>>>>
>>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>>> on a
>>>>>> RDB image from a disk known to currently work under Linux (thanks
>>>>>> Geert
>>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>>> use
>>>>>> for tests is a long way away from my home indeed.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>>
>>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>>> block
>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>>> under
>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>>> {^_-}
>>>>>>>
>>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>>> change. I remember thinking about this for awhile and then
>>>>>>> determining
>>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>>> into a gordian knot.
>>>>>>>
>>>>>>> {^_^}
>>>>>>>
>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>>
>>>>>>>> Joanne,
>>>>>>>>
>>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>>
>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
>>>>>>>> sdb1
>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>>> 2 spb
>>>>>>>> 4)
>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>>> Attached SCSI disk
>>>>>>>>
>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>>> 512
>>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>>> size.
>>>>>>>>
>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>>> yet.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Michael
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>>>>
>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>>> system is
>>>>>>>>> a famn
>>>>>>>>> dool.
>>>>>>>>>
>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>>> should
>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>>> block maps.
>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>>
>>>>>>>>> {^_^}
>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>>
>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> test results at
>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>>> so if
>>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>>
>>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>>> preparing
>>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>>
>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>>> which
>>>>>>>>>> I still have in my apartment.
>>>>>>>>>>
>>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>>> unless
>>>>>>>>>> someone else does.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>> linux-m68k" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-28 5:43 ` Michael Schmitz
@ 2018-06-28 6:39 ` jdow
2018-06-28 8:16 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
2018-06-28 8:07 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
1 sibling, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-28 6:39 UTC (permalink / raw)
To: Michael Schmitz
Cc: Martin Steigerwald, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Anything done to RDBs for Linux must remain 100.000% compatible with existing
Amiga equipment. Otherwise, what's the point of bothering to use RDBs?
Note, as an experiment I had a CrossDOS partition on a hard disk briefly. So
some amazing nonsense is possible. I concluded that for me it was not something
I wanted to do. Alternative techniques worked well enough.
That brings to the fore an interesting question. Why bother with RDBs over 2TB
unless you want a disk with one single partition? This Win10 monster I am using
has a modest BIOS driver partition for the OS and a giant data partition. That
smaller partition would easily work with any RDB/Filesystem combination since
2.0. So there are some good workarounds that are probably "safer" and at least
as flexible as RDBs, one Linux has used for a very long time, too.
With that it is easy in concept to put together a dual boot system with all the
disks and partitions readable if you play the proper games at the filesystem
level and keep individual partitions small. The filesystem would have to take on
some of the attributes of a device driver, though. A 281 TB disk made up of 2 TB
partitions should be possible to create if you think it through. (Note that with
RDBs the CHS disk description is purely an abstraction that has no relationship
to anything inside the disk. The same tricks could exist at the filesystem level.)
As for existing tools screwing up, do you think that is an excuse to perpetuate
the failure?
If we are speaking Linux for m68k, what does it do when confronted with which is
described with a size larger than 32 bits? Before it had everything inside it
that refers to positions within a file by byte index with an __int64 type of
storage did it throw up it's hands, corrupt the disk, or what? If the m68k Linux
is a failure perhaps it doesn't matter what you do for Linuxoids. (I'll stick
with the x64 version in that case. And I never go back to earlier versions for
games since I consider a good compiler is the best game around. It's better than
a blank piece of paper for possibilities.)
As I have said, for the RDB parser fix the famndool thing. Do fix it right in
such a manner that if somebody compiles it against a version with no 64 bit
device code it will throw a proper overflow error and protect the user. Users
are dumb. We like to think of ourselves as smart. Let's try to be smart about
this where we can so fingers can't point back at us rather than the fool that
made some other error.
And do remember, I am merely (and vociferously) advising rather than dictating.
You don't need my approval to proceed. I may want my name noted as an early
contributor only. Meanwhile I spit out ideas as they come to me. One or more of
them might be good. And offering alternatives is better than simply saying "No"
most of the time.
If people are using RDBs for TB level disks I doubt they can remember which is
the left shoe when they are getting dressed in the morning before going out in
the yard to beat some dead horses. Or else maybe they just want to see how far
they can flog the m68k architecture as a mental challenge. In that case, taking
it too seriously could hurt. Note that I am mostly ignoring m68k Linux. People
using that are hard core. People using x86/x64 Linux aren't such hard core
folks. And I bet most of them want to read the disks so they can copy stuff to
Amiga Forever or WinUAE running on other architectures. So TB is not likely to
be an issue for them, either.
{^_^}
On 20180627 22:43, Michael Schmitz wrote:
> Joanne,
>
> Linux on m68k has supported lseek64 (or llseek) for a long time (from glibc
> version 2.1 according to what I found). About the only area where we are limited
> by 32 bits is the virtual memory size.
>
> I'm not proposing to modify the RDB format definition, though an extension to
> store 64 bit offsets separate from the 32 bit ones would be one way to make
> certain such disks are safe to use on 3.1 and earlier versions of AmigaOS.
> (Another one would be to modify the disk drivers on older versions to do the
> offset calculation in 64 bit, and check for overflow just as we do here. Not
> sure whether that's feasible. And as you so eloquently describe, we can't rely
> on users listening.)
>
> Either way, we need the cooperation of partitioning tool writers to ensure
> partition information that is prone to overflows is never stored in the 32 bit,
> classic RDB. That appears to have failed already, as Martin's experience
> illustrates.
>
> I'm only concerned with fixing the (dangerous) but in the Linux partition format
> parser for RDB. Refusing to use any partitions that will cause havoc on old
> AmigaOS versions is all I can do to try and get the users' attention.
>
> Your warning makes me wonder whether the log message should just say 'report
> this bug to linux-m68k@vger.kernel.org' to at least try and educate any that
> respond about the dangers of their partitioning scheme before telling them about
> the override option. Problem with that is, in about three years no one will
> remember any of this ...
>
> Cheers,
>
> Michael
>
>
> Am 28.06.2018 um 15:44 schrieb jdow:
>> Michael, as long as m68k only supports int fseek( int ) 4 GB * block
>> size is your HARD limit. Versions that support __int64 fseek64( __int64
>> ) can work with larger disks. RDBs could include RDSK and a new set of
>> other blocks that replace the last two characters with "64" and use
>> __int64 where needed in the various values. That way a clever disk
>> partitioner could give allow normal (32 bit) RDB definitions where
>> possible. Then at least SOME of the disk could be supported AND a very
>> clever filesystem that abstracts very large disks properly could give
>> access to the whole disk. (Read the RDBs first 32 bits. Then if a
>> filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit
>> partitions are revealed.
>>
>> I could be wrong but I do not think RDBs could be safely modified any
>> other way to work. And, trust me as I bet this is still true, you will
>> need a SERIOUSLY good bomb shelter on the Moon if you change RDBs.
>> Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4
>> because he wants to run a game that crashes on anything newer. Then Joe
>> got far enough something writes to the disk and data is corrupted. Note
>> further that Amigoids do NOT, repeat NOT, listen to facts in such cases.
>> Hell, some of them never listened to facts about an incident at Jerry
>> Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we
>> lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried
>> to tell them I was there, it was my machine, and 1.1 was, indeed, crap.
>>
>> {o.o}
>>
>> On 20180627 02:00, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> I'm not at all allergic to avoiding RDB at all cost for new disks. If
>>> AmigaOS 4.1 supports more recent partition formats, all the better.
>>> This is all about supporting use of legacy RDB disks on Linux (though
>>> 2 TB does stretch the definition of 'legacy' a little). My interest in
>>> this is to ensure we can continue to use RDB format disks on m68k
>>> Amiga computers which have no other way to boot Linux from disk.
>>>
>>> Not proposing to change the RDB format at all, either. Just trying to
>>> make sure we translate RDB info into Linux 512-byte block offset and
>>> size numbers correctly. The kernel won't modify the RDB at all
>>> (intentionally, that is - with the correct choice of partition sizes,
>>> Martin might well have wiped out his RDB with the current version of
>>> the parser).
>>>
>>> The choice of refusing to mount a disk (or mounting read-only) rests
>>> with the VFS drivers alone - AFFS in that case. Not touching any of
>>> that. At partition scan time, we only have the option of making the
>>> partition available (with a warning printed), or refusing to make it
>>> available to the kernel. Once it's made available, all bets are off.
>>>
>>> From what Martin writes, his test case RDB was valid and worked as
>>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
>>> necessary extensions to handle the large offsets resulting from 2 TB
>>> disks. Not sure what safeguards are in place when connecting such a
>>> disk to older versions of AmigaOS, but that is a different matter
>>> entirely.
>>>
>>> The overflows in partition offset and size are the only ones I can see
>>> in the partition parser - there is no other overflow I've identified.
>>> I just stated that in order to place a partition towards the end of a
>>> 2 TB disk, the offset calculation will overflow regardless of what
>>> combination of rdb->rdb_BlockBytes and sector addresses stored in the
>>> RDB (in units of 512 byte blocks) we use:
>>>
>>> blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
>>>
>>>
>>> nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
>>> be32_to_cpu(pb->pb_Environment[9])) *
>>> be32_to_cpu(pb->pb_Environment[3]) *
>>> be32_to_cpu(pb->pb_Environment[5]) *
>>> blksize;
>>> if (!nr_sects)
>>> continue;
>>> start_sect = be32_to_cpu(pb->pb_Environment[9]) *
>>> be32_to_cpu(pb->pb_Environment[3]) *
>>> be32_to_cpu(pb->pb_Environment[5]) *
>>> blksize;
>>>
>>> But in the interest of avoiding any accidental use of a RDB partition
>>> where calculations currently overflow, I'll make the default behaviour
>>> to bail out (instead of using wrong offset or size as we currently
>>> do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed
>>> at their own risk (though I still can't see what harm should result
>>> from now translating a well formed v4.1 2 TB disk RDB correctly for
>>> the first time).
>>>
>>> Whether or not Linux correctly handles AFFS filesystems larger than 1
>>> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
>>> ought to prevent accidental use of AFFS filesystems on RDB disks which
>>> I suppose is the majority of use cases.
>>>
>>> Bugs in partitioning tools on Linux are entirely out of scope - the
>>> partitioning tools bypass the partition structure discovered by the
>>> kernel, and work straight on the raw device. No protecting against that.
>>>
>>> If you can point out a way to cause data loss with these precautions,
>>> for a disk 2 TB or larger that was partitioned and used on a recent
>>> version or AmigaOS supporting such large disks, I'd consider omitting
>>> the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
>>> option.
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>>
>>> Am 27.06.2018 um 18:24 schrieb jdow:
>>>> You allergic to using a GPT solution? It will get away from some of the
>>>> evils that RDB has inherent in it because they are also features?
>>>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>>>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>>>> system that sees an RDSK block is going to try to translate it. If you
>>>> add a new RDB format you are going to get bizarre and probably quite
>>>> destructive results from the mistake. Fail safe is a rather good notion,
>>>> methinks.
>>>>
>>>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>>>> system seems utterly outlandish to me. You cited another overflow
>>>> potential. There are at least three we've identified, I believe. Are you
>>>> 100% sure there are no more? The specific one you mention of translating
>>>> RDB to Linux has a proper solution in the RDB reader. It should recover
>>>> such overflow errors in the RDB as it can with due care and polish. It
>>>> should flag any other overflow error it detects within the RDBs and
>>>> return an error such as to leave the disk unmounted or mounted read-only
>>>> if you feel like messing up a poor sod's backups. The simple solution is
>>>> to read each of the variables with the nominal RDB size and convert it
>>>> to uint64_t before calculating byte indices.
>>>>
>>>> However, consider my inputs as advice from an adult who has seen the
>>>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>>>> you wish; but, I would plead with you to avoid ANY chance you can for
>>>> the user to make a bonehead stupid move and lose all his treasured disk
>>>> archives. Doing otherwise is very poor form.
>>>>
>>>> {o.o} Joanne "Said enough, she has" Dow
>>>>
>>>> On 20180626 18:07, Michael Schmitz wrote:
>>>>> Joanne,
>>>>>
>>>>> As far as I have been able to test, the change is backwards compatible
>>>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>>>> That"s only been done on an emulator though.
>>>>>
>>>>> Your advice about the dangers of using RDB disks that would have
>>>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>>>> in question ever used on a 32 bit Amiga system?
>>>>>
>>>>> RDB disk format is meant for legacy use only, so I think we can get
>>>>> away with printing a big fat warning during boot, advising the user
>>>>> that the oversize RDB partition(s) scanned are not compatible with
>>>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>>>> partition at disk end and trashing valid partitions that overlap,
>>>>> which is what Martin was after.
>>>>>
>>>>> If that still seems too risky, we can make the default behaviour to
>>>>> bail out once a potential overflow is detected, and allow the user to
>>>>> override that through a boot parameter. I'd leave that decision up for
>>>>> the code review on linux-block.
>>>>>
>>>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>>>> start and size calculations, so a change of the RDB blocksize to
>>>>> reduce the block counts stored in the RDB would still result in the
>>>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>>>> updating (along with probably most legacy m68k partitioners). Adrian
>>>>> has advertised parted as replacement for the old tools - maybe this
>>>>> would make a nice test case for parted?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Michael
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>>>>> If it is not backwards compatible I for one would refuse to use it.
>>>>>> And if
>>>>>> it still mattered that much to me I'd also generate a reasonable
>>>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>>>> good idea.
>>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>>>> uint32_t
>>>>>> only system. If it is for your personal use then you're entirely
>>>>>> free to
>>>>>> reject my advice and are probably smart enough to keep it working for
>>>>>> yourself.
>>>>>>
>>>>>> GPT is probably the right way to go. Preserve the ability to read
>>>>>> RDBs for
>>>>>> legacy disks only.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>>
>>>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>>>
>>>>>>> Joanne,
>>>>>>>
>>>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>>>> the
>>>>>>> issues Martin reported. Your patch addresses that by using the
>>>>>>> correct
>>>>>>> data type for the calculations (as do other partition parsers that
>>>>>>> may
>>>>>>> have to deal with large disks) and fixes Martin's bug, so appears
>>>>>>> to be
>>>>>>> the right thing to do.
>>>>>>>
>>>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>>>> calculations
>>>>>>> don't currently overflow is not expected to cause new issues, other
>>>>>>> than
>>>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>>>> ramifications with filesystems on these partitions). So
>>>>>>> comptibility is
>>>>>>> preserved.
>>>>>>>
>>>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>>>> issues in filesystems as well, but I can't see how the block size
>>>>>>> stored
>>>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>>>> Linux
>>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>>>> Viro
>>>>>>> may have found a few issues when he looked at the AFFS code so I
>>>>>>> won't
>>>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>>>> patch.
>>>>>>>
>>>>>>> If you feel strongly about unknown ramifications of any
>>>>>>> filesystems on
>>>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>>>> warning about these partitions.
>>>>>>>
>>>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>>>> on a
>>>>>>> RDB image from a disk known to currently work under Linux (thanks
>>>>>>> Geert
>>>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>>>> use
>>>>>>> for tests is a long way away from my home indeed.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Michael
>>>>>>>
>>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>>>
>>>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>>>> block
>>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>>>> under
>>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>>>> {^_-}
>>>>>>>>
>>>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>>>> change. I remember thinking about this for awhile and then
>>>>>>>> determining
>>>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>>>> into a gordian knot.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>>
>>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>>>
>>>>>>>>> Joanne,
>>>>>>>>>
>>>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>>>
>>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
>>>>>>>>> sdb1
>>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>>>> 2 spb
>>>>>>>>> 4)
>>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>>>> Attached SCSI disk
>>>>>>>>>
>>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>>>> 512
>>>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>>>> size.
>>>>>>>>>
>>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>>>> yet.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Michael
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net> wrote:
>>>>>>>>>>
>>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>>>> system is
>>>>>>>>>> a famn
>>>>>>>>>> dool.
>>>>>>>>>>
>>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>>>> should
>>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>>>> block maps.
>>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>>>
>>>>>>>>>> {^_^}
>>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi.
>>>>>>>>>>>
>>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> test results at
>>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>>>> so if
>>>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>>>
>>>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>>>> preparing
>>>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>>>
>>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>>>> which
>>>>>>>>>>> I still have in my apartment.
>>>>>>>>>>>
>>>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>>>> unless
>>>>>>>>>>> someone else does.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>> linux-m68k" in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)
2018-06-28 6:39 ` jdow
@ 2018-06-28 8:16 ` Martin Steigerwald
2018-06-28 10:00 ` Amiga RDB partition support for disks >= 2 TB jdow
0 siblings, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28 8:16 UTC (permalink / raw)
To: jdow
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Dear Joanne.
jdow - 28.06.18, 08:39:
> Anything done to RDBs for Linux must remain 100.000% compatible with
> existing Amiga equipment. Otherwise, what's the point of bothering to
> use RDBs?
Done to, in the sense of written to: Yes. I completely agree. But that
is for amiga-fdisk and parted. And for partitioning tools on native OS.
[…]
> That brings to the fore an interesting question. Why bother with RDBs
> over 2TB unless you want a disk with one single partition? This Win10
> monster I am using has a modest BIOS driver partition for the OS and
> a giant data partition. That smaller partition would easily work with
> any RDB/Filesystem combination since 2.0. So there are some good
> workarounds that are probably "safer" and at least as flexible as
> RDBs, one Linux has used for a very long time, too.
Well, my use case was simple:
I had this 2 TB disk and I choose to share it as a backup disk for Linux
*and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.
As AmigaOS up to my knowledge does not have GPT support and MBR with 2
TB disk is asking for even more trouble and is not supported in any
sensible partitioning tool on AmigaOS and I choose to use Media Toolbox,
I went with RDB as I thought it is nicely supported in Linux, which it
is, apart from the overflowing issue.
So I did it this way.
> As I have said, for the RDB parser fix the famndool thing. Do fix it
> right in such a manner that if somebody compiles it against a version
> with no 64 bit device code it will throw a proper overflow error and
> protect the user. Users are dumb. We like to think of ourselves as
Sure, if the Linux kernel can´t handle it due to missing CONFIG_LBDAF or
so… by all means *bail* out. Without even a kernel option to parse this
anyway. No overflowing. Period. That is what I wrote from the beginning.
> smart. Let's try to be smart about this where we can so fingers can't
> point back at us rather than the fool that made some other error.
I completely agree.
> And do remember, I am merely (and vociferously) advising rather than
> dictating. You don't need my approval to proceed. I may want my name
> noted as an early contributor only. Meanwhile I spit out ideas as
> they come to me. One or more of them might be good. And offering
> alternatives is better than simply saying "No" most of the time.
>
> If people are using RDBs for TB level disks I doubt they can remember
> which is the left shoe when they are getting dressed in the morning
> before going out in the yard to beat some dead horses. Or else maybe
Heh, I remembered my shoes back then. And I informed myself about what I
was doing.
> they just want to see how far they can flog the m68k architecture as
> a mental challenge. In that case, taking it too seriously could hurt.
And well, yes… I wanted to see how far I can get away with it :)
> Note that I am mostly ignoring m68k Linux. People using that are hard
> core. People using x86/x64 Linux aren't such hard core folks. And I
> bet most of them want to read the disks so they can copy stuff to
> Amiga Forever or WinUAE running on other architectures. So TB is not
> likely to be an issue for them, either.
Yes.
Print a warning and be done with it in the RDB parser code.
Put a big fat warning into amiga-fdisk and parted!
Thanks,
Martin
> {^_^}
>
> On 20180627 22:43, Michael Schmitz wrote:
> > Joanne,
> >
> > Linux on m68k has supported lseek64 (or llseek) for a long time
> > (from glibc version 2.1 according to what I found). About the only
> > area where we are limited by 32 bits is the virtual memory size.
> >
> > I'm not proposing to modify the RDB format definition, though an
> > extension to store 64 bit offsets separate from the 32 bit ones
> > would be one way to make certain such disks are safe to use on 3.1
> > and earlier versions of AmigaOS. (Another one would be to modify
> > the disk drivers on older versions to do the offset calculation in
> > 64 bit, and check for overflow just as we do here. Not sure whether
> > that's feasible. And as you so eloquently describe, we can't rely
> > on users listening.)
> >
> > Either way, we need the cooperation of partitioning tool writers to
> > ensure partition information that is prone to overflows is never
> > stored in the 32 bit, classic RDB. That appears to have failed
> > already, as Martin's experience illustrates.
> >
> > I'm only concerned with fixing the (dangerous) but in the Linux
> > partition format parser for RDB. Refusing to use any partitions
> > that will cause havoc on old AmigaOS versions is all I can do to
> > try and get the users' attention.
> >
> > Your warning makes me wonder whether the log message should just say
> > 'report this bug to linux-m68k@vger.kernel.org' to at least try and
> > educate any that respond about the dangers of their partitioning
> > scheme before telling them about the override option. Problem with
> > that is, in about three years no one will remember any of this ...
> >
> > Cheers,
> >
> > Michael
> >
> > Am 28.06.2018 um 15:44 schrieb jdow:
> >> Michael, as long as m68k only supports int fseek( int ) 4 GB *
> >> block
> >> size is your HARD limit. Versions that support __int64 fseek64(
> >> __int64 ) can work with larger disks. RDBs could include RDSK and
> >> a new set of other blocks that replace the last two characters
> >> with "64" and use __int64 where needed in the various values. That
> >> way a clever disk partitioner could give allow normal (32 bit) RDB
> >> definitions where possible. Then at least SOME of the disk could
> >> be supported AND a very clever filesystem that abstracts very
> >> large disks properly could give access to the whole disk. (Read
> >> the RDBs first 32 bits. Then if a filesystem or driveinit was
> >> loaded re-read the RDBs to see if new 64 bit partitions are
> >> revealed.
> >>
> >> I could be wrong but I do not think RDBs could be safely modified
> >> any
> >> other way to work. And, trust me as I bet this is still true, you
> >> will need a SERIOUSLY good bomb shelter on the Moon if you change
> >> RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads
> >> Amigados 2.4 because he wants to run a game that crashes on
> >> anything newer. Then Joe got far enough something writes to the
> >> disk and data is corrupted. Note further that Amigoids do NOT,
> >> repeat NOT, listen to facts in such cases. Hell, some of them
> >> never listened to facts about an incident at Jerry Pournelle's
> >> place when a 1.1 DPaint session with Kelly Freas hung and we lost
> >> a delightful drawing. Jerry reported it. Amigoids screamed. I
> >> tried to tell them I was there, it was my machine, and 1.1 was,
> >> indeed, crap.
> >>
> >> {o.o}
> >>
> >> On 20180627 02:00, Michael Schmitz wrote:
> >>> Joanne,
> >>>
> >>> I'm not at all allergic to avoiding RDB at all cost for new disks.
> >>> If
> >>> AmigaOS 4.1 supports more recent partition formats, all the
> >>> better.
> >>> This is all about supporting use of legacy RDB disks on Linux
> >>> (though
> >>> 2 TB does stretch the definition of 'legacy' a little). My
> >>> interest in this is to ensure we can continue to use RDB format
> >>> disks on m68k Amiga computers which have no other way to boot
> >>> Linux from disk.
> >>>
> >>> Not proposing to change the RDB format at all, either. Just trying
> >>> to
> >>> make sure we translate RDB info into Linux 512-byte block offset
> >>> and
> >>> size numbers correctly. The kernel won't modify the RDB at all
> >>> (intentionally, that is - with the correct choice of partition
> >>> sizes,
> >>> Martin might well have wiped out his RDB with the current version
> >>> of
> >>> the parser).
> >>>
> >>> The choice of refusing to mount a disk (or mounting read-only)
> >>> rests
> >>> with the VFS drivers alone - AFFS in that case. Not touching any
> >>> of
> >>> that. At partition scan time, we only have the option of making
> >>> the
> >>> partition available (with a warning printed), or refusing to make
> >>> it
> >>> available to the kernel. Once it's made available, all bets are
> >>> off.
> >>>
> >>> From what Martin writes, his test case RDB was valid and worked
> >>> as
> >>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
> >>> necessary extensions to handle the large offsets resulting from 2
> >>> TB
> >>> disks. Not sure what safeguards are in place when connecting such
> >>> a
> >>> disk to older versions of AmigaOS, but that is a different matter
> >>> entirely.
> >>>
> >>> The overflows in partition offset and size are the only ones I can
> >>> see in the partition parser - there is no other overflow I've
> >>> identified. I just stated that in order to place a partition
> >>> towards the end of a 2 TB disk, the offset calculation will
> >>> overflow regardless of what combination of rdb->rdb_BlockBytes
> >>> and sector addresses stored in the RDB (in units of 512 byte
> >>> blocks) we use:
> >>>
> >>> blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
> >>>
> >>>
> >>> nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
> >>> 1 - be32_to_cpu(pb->pb_Environment[9])) *
> >>> be32_to_cpu(pb->pb_Environment[3]) *
> >>> be32_to_cpu(pb->pb_Environment[5]) * blksize;
> >>> if (!nr_sects)
> >>> continue;
> >>> start_sect = be32_to_cpu(pb->pb_Environment[9]) *
> >>> be32_to_cpu(pb->pb_Environment[3]) *
> >>> be32_to_cpu(pb->pb_Environment[5]) *
> >>> blksize;
> >>>
> >>> But in the interest of avoiding any accidental use of a RDB
> >>> partition
> >>> where calculations currently overflow, I'll make the default
> >>> behaviour to bail out (instead of using wrong offset or size as
> >>> we currently do). Given the 'eat_my_RDB_disk=1' boot option, the
> >>> user may proceed at their own risk (though I still can't see what
> >>> harm should result from now translating a well formed v4.1 2 TB
> >>> disk RDB correctly for the first time).
> >>>
> >>> Whether or not Linux correctly handles AFFS filesystems larger
> >>> than 1
> >>> TB is a matter for VFS experts. Bailing out on nr_sects
> >>> overflowing
> >>> ought to prevent accidental use of AFFS filesystems on RDB disks
> >>> which I suppose is the majority of use cases.
> >>>
> >>> Bugs in partitioning tools on Linux are entirely out of scope -
> >>> the
> >>> partitioning tools bypass the partition structure discovered by
> >>> the
> >>> kernel, and work straight on the raw device. No protecting against
> >>> that.
> >>>
> >>> If you can point out a way to cause data loss with these
> >>> precautions,
> >>> for a disk 2 TB or larger that was partitioned and used on a
> >>> recent
> >>> version or AmigaOS supporting such large disks, I'd consider
> >>> omitting
> >>> the 'eat_my_RDB_disk' boot option, and just bail out as the only
> >>> safe
> >>> option.
> >>>
> >>> Cheers,
> >>>
> >>> Michael
> >>>
> >>> Am 27.06.2018 um 18:24 schrieb jdow:
> >>>> You allergic to using a GPT solution? It will get away from some
> >>>> of the evils that RDB has inherent in it because they are also
> >>>> features? (Loading a filesystem or DriveInit code from RDBs is
> >>>> just asking for a nearly impossible to remove malware
> >>>> infection.) Furthermore, any 32 bit system that sees an RDSK
> >>>> block is going to try to translate it. If you add a new RDB
> >>>> format you are going to get bizarre and probably quite
> >>>> destructive results from the mistake. Fail safe is a rather good
> >>>> notion, methinks.
> >>>>
> >>>> Personally I figure this is all rather surreal. 2TG of junk on an
> >>>> Amiga system seems utterly outlandish to me. You cited another
> >>>> overflow potential. There are at least three we've identified, I
> >>>> believe. Are you 100% sure there are no more? The specific one
> >>>> you mention of translating RDB to Linux has a proper solution in
> >>>> the RDB reader. It should recover such overflow errors in the
> >>>> RDB as it can with due care and polish. It should flag any other
> >>>> overflow error it detects within the RDBs and return an error
> >>>> such as to leave the disk unmounted or mounted read-only if you
> >>>> feel like messing up a poor sod's backups. The simple solution
> >>>> is to read each of the variables with the nominal RDB size and
> >>>> convert it to uint64_t before calculating byte indices.
> >>>>
> >>>> However, consider my inputs as advice from an adult who has seen
> >>>> the
> >>>> Amiga Elephant so to speak. I am not trying to assert any
> >>>> control. Do as you wish; but, I would plead with you to avoid
> >>>> ANY chance you can for the user to make a bonehead stupid move
> >>>> and lose all his treasured disk archives. Doing otherwise is
> >>>> very poor form.
> >>>>
> >>>> {o.o} Joanne "Said enough, she has" Dow
> >>>>
> >>>> On 20180626 18:07, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>>
> >>>>> As far as I have been able to test, the change is backwards
> >>>>> compatible (RDB partitions from an old disk 80 GB disk are
> >>>>> still recognized OK). That"s only been done on an emulator
> >>>>> though.
> >>>>>
> >>>>> Your advice about the dangers of using RDB disks that would have
> >>>>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>>>> well
> >>>>> taken though. Maybe Martin can clarify that for me - was the 2
> >>>>> TB disk in question ever used on a 32 bit Amiga system?
> >>>>>
> >>>>> RDB disk format is meant for legacy use only, so I think we can
> >>>>> get
> >>>>> away with printing a big fat warning during boot, advising the
> >>>>> user
> >>>>> that the oversize RDB partition(s) scanned are not compatible
> >>>>> with
> >>>>> legacy 32 bit AmigaOS. With the proposed fix they will work
> >>>>> under both AmigaOS 4.1 and Linux instead of truncating the
> >>>>> first overflowing partition at disk end and trashing valid
> >>>>> partitions that overlap, which is what Martin was after.
> >>>>>
> >>>>> If that still seems too risky, we can make the default behaviour
> >>>>> to
> >>>>> bail out once a potential overflow is detected, and allow the
> >>>>> user to
> >>>>> override that through a boot parameter. I'd leave that decision
> >>>>> up for the code review on linux-block.
> >>>>>
> >>>>> Two more comments: Linux uses 512 byte block sizes for the
> >>>>> partition
> >>>>> start and size calculations, so a change of the RDB blocksize to
> >>>>> reduce the block counts stored in the RDB would still result in
> >>>>> the
> >>>>> same overflow. And amiga-fdisk is indeed utterly broken and
> >>>>> needs
> >>>>> updating (along with probably most legacy m68k partitioners).
> >>>>> Adrian
> >>>>> has advertised parted as replacement for the old tools - maybe
> >>>>> this
> >>>>> would make a nice test case for parted?
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net>
wrote:
> >>>>>> If it is not backwards compatible I for one would refuse to use
> >>>>>> it.
> >>>>>> And if
> >>>>>> it still mattered that much to me I'd also generate a
> >>>>>> reasonable
> >>>>>> alternative. Modifying RDBs nay not be even an approximation of
> >>>>>> a
> >>>>>> good idea.
> >>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted
> >>>>>> by a
> >>>>>> uint32_t
> >>>>>> only system. If it is for your personal use then you're
> >>>>>> entirely
> >>>>>> free to
> >>>>>> reject my advice and are probably smart enough to keep it
> >>>>>> working for
> >>>>>> yourself.
> >>>>>>
> >>>>>> GPT is probably the right way to go. Preserve the ability to
> >>>>>> read
> >>>>>> RDBs for
> >>>>>> legacy disks only.
> >>>>>>
> >>>>>> {^_^}
> >>>>>>
> >>>>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>>>> Joanne,
> >>>>>>>
> >>>>>>> I think we all agree that doing 32 bit calculations on
> >>>>>>> 512-byte block
> >>>>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>>>> causing
> >>>>>>> the
> >>>>>>> issues Martin reported. Your patch addresses that by using the
> >>>>>>> correct
> >>>>>>> data type for the calculations (as do other partition parsers
> >>>>>>> that
> >>>>>>> may
> >>>>>>> have to deal with large disks) and fixes Martin's bug, so
> >>>>>>> appears
> >>>>>>> to be
> >>>>>>> the right thing to do.
> >>>>>>>
> >>>>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>>>> calculations
> >>>>>>> don't currently overflow is not expected to cause new issues,
> >>>>>>> other
> >>>>>>> than
> >>>>>>> enabling use of disk and partitions larger than 2 TB (which
> >>>>>>> may have
> >>>>>>> ramifications with filesystems on these partitions). So
> >>>>>>> comptibility is
> >>>>>>> preserved.
> >>>>>>>
> >>>>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>>>> overflow
> >>>>>>> issues in filesystems as well, but I can't see how the block
> >>>>>>> size
> >>>>>>> stored
> >>>>>>> in the RDB would enforce use of the same block size in
> >>>>>>> filesystems.
> >>>>>>> We'll have to rely on the filesystem tools to get that right,
> >>>>>>> too.
> >>>>>>> Linux
> >>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this
> >>>>>>> should
> >>>>>>> allow partitions larger than 2 TB to work already (but I
> >>>>>>> suspect Al
> >>>>>>> Viro
> >>>>>>> may have found a few issues when he looked at the AFFS code so
> >>>>>>> I
> >>>>>>> won't
> >>>>>>> say more). Anyway partitioning tools and filesystems are
> >>>>>>> unrelated to
> >>>>>>> the Linux partition parser code which is all we aim to fix in
> >>>>>>> this
> >>>>>>> patch.
> >>>>>>>
> >>>>>>> If you feel strongly about unknown ramifications of any
> >>>>>>> filesystems on
> >>>>>>> partitions larger than 2 TB, say so and I'll have the kernel
> >>>>>>> print a
> >>>>>>> warning about these partitions.
> >>>>>>>
> >>>>>>> I'll get this patch tested on Martin's test case image as well
> >>>>>>> as
> >>>>>>> on a
> >>>>>>> RDB image from a disk known to currently work under Linux
> >>>>>>> (thanks
> >>>>>>> Geert
> >>>>>>> for the losetup hint). Can't do much more without procuring a
> >>>>>>> working
> >>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I
> >>>>>>> plan to
> >>>>>>> use
> >>>>>>> for tests is a long way away from my home indeed.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Michael
> >>>>>>>
> >>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>>>> suppose.
> >>>>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>>>> force the
> >>>>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>>>> systems
> >>>>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>>>> blocks
> >>>>>>>> list will have to go beyond a uint32_t size, for example. But
> >>>>>>>> a
> >>>>>>>> block
> >>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>>>> tad
> >>>>>>>> under
> >>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so
> >>>>>>>> bad.
> >>>>>>>> {^_-}
> >>>>>>>>
> >>>>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>>>> make a
> >>>>>>>> change. I remember thinking about this for awhile and then
> >>>>>>>> determining
> >>>>>>>> I REALLY did not want to think about it as my brain was
> >>>>>>>> getting tied
> >>>>>>>> into a gordian knot.
> >>>>>>>>
> >>>>>>>> {^_^}
> >>>>>>>>
> >>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>>>> Joanne,
> >>>>>>>>>
> >>>>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>>>
> >>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK
> >>>>>>>>> (512)
> >>>>>>>>> sdb1
> >>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>>>> (DOS^C)(res
> >>>>>>>>> 2 spb
> >>>>>>>>> 4)
> >>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>>>> [sdb]
> >>>>>>>>> Attached SCSI disk
> >>>>>>>>>
> >>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>>>> means
> >>>>>>>>> 512
> >>>>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>>>> block
> >>>>>>>>> size.
> >>>>>>>>>
> >>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>>>> units.
> >>>>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>>>> others
> >>>>>>>>> yet.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>
> >>>>>>>>> Michael
> >>>>>>>>>
> >>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
wrote:
> >>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>>>> system is
> >>>>>>>>>> a famn
> >>>>>>>>>> dool.
> >>>>>>>>>>
> >>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>>>> so it
> >>>>>>>>>> should
> >>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>>>> blocks is
> >>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >>>>>>>>>> disk in
> >>>>>>>>>> block maps.
> >>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>>>
> >>>>>>>>>> {^_^}
> >>>>>>>>>>
> >>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>>>> Hi.
> >>>>>>>>>>>
> >>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>>>> test results at
> >>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>>>> there,
> >>>>>>>>>>>> so if
> >>>>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>>>
> >>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>>>
> >>>>>>>>>>> I do not care enough about this, in order to motivate
> >>>>>>>>>>> myself
> >>>>>>>>>>> preparing
> >>>>>>>>>>> the a patch from Joanne Dow´s fix.
> >>>>>>>>>>>
> >>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>>>> Sam440ep
> >>>>>>>>>>> which
> >>>>>>>>>>> I still have in my apartment.
> >>>>>>>>>>>
> >>>>>>>>>>> So RDB support in Linux it remains broken for disks larger
> >>>>>>>>>>> 2 TB,
> >>>>>>>>>>> unless
> >>>>>>>>>>> someone else does.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: Amiga RDB partition support for disks >= 2 TB
2018-06-28 8:16 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
@ 2018-06-28 10:00 ` jdow
2018-06-28 11:30 ` Martin Steigerwald
0 siblings, 1 reply; 73+ messages in thread
From: jdow @ 2018-06-28 10:00 UTC (permalink / raw)
To: Martin Steigerwald
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
On 20180628 01:16, Martin Steigerwald wrote:
> Dear Joanne.
>
> jdow - 28.06.18, 08:39:
>> Anything done to RDBs for Linux must remain 100.000% compatible with
>> existing Amiga equipment. Otherwise, what's the point of bothering to
>> use RDBs?
>
> Done to, in the sense of written to: Yes. I completely agree. But that
> is for amiga-fdisk and parted. And for partitioning tools on native OS.
Design changes, too.
> […]
>> That brings to the fore an interesting question. Why bother with RDBs
>> over 2TB unless you want a disk with one single partition? This Win10
>> monster I am using has a modest BIOS driver partition for the OS and
>> a giant data partition. That smaller partition would easily work with
>> any RDB/Filesystem combination since 2.0. So there are some good
>> workarounds that are probably "safer" and at least as flexible as
>> RDBs, one Linux has used for a very long time, too.
>
> Well, my use case was simple:
>
> I had this 2 TB disk and I choose to share it as a backup disk for Linux
> *and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.
EEEEEEK! The hair on my neck is standing up straight! Have you heard of SAMBA?
The linux mail server firewall etc machine has an extra 4TB disk on it as a
backup for the other systems, although a piddly 4TB is small when I save the
entire 3G RAID system I have. It's a proof of concept so.... A full backup on a
1gig Ethernet still takes a looooong time. But backing up even an 18GB disk on
an Amiga via 100Base-t isn't too bad. And disk speeds of the era being what they
were it's about all you can do anyway.
{o.o}
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: Amiga RDB partition support for disks >= 2 TB
2018-06-28 10:00 ` Amiga RDB partition support for disks >= 2 TB jdow
@ 2018-06-28 11:30 ` Martin Steigerwald
2018-06-28 11:38 ` Martin Steigerwald
0 siblings, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28 11:30 UTC (permalink / raw)
To: jdow
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
jdow - 28.06.18, 12:00:
> On 20180628 01:16, Martin Steigerwald wrote:
[…]
> >> That brings to the fore an interesting question. Why bother with
> >> RDBs
> >> over 2TB unless you want a disk with one single partition? This
> >> Win10
> >> monster I am using has a modest BIOS driver partition for the OS
> >> and
> >> a giant data partition. That smaller partition would easily work
> >> with
> >> any RDB/Filesystem combination since 2.0. So there are some good
> >> workarounds that are probably "safer" and at least as flexible as
> >> RDBs, one Linux has used for a very long time, too.
> >
> > Well, my use case was simple:
> >
> > I had this 2 TB disk and I choose to share it as a backup disk for
> > Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me
> > desk here.
> EEEEEEK! The hair on my neck is standing up straight! Have you heard
> of SAMBA? The linux mail server firewall etc machine has an extra 4TB
> disk on it as a backup for the other systems, although a piddly 4TB
> is small when I save the entire 3G RAID system I have. It's a proof
> of concept so.... A full backup on a 1gig Ethernet still takes a
> looooong time. But backing up even an 18GB disk on an Amiga via
> 100Base-t isn't too bad. And disk speeds of the era being what they
> were it's about all you can do anyway.
Heh, the thing worked just fine in Amiga OS 4. I got away with it
without an issue, until I plugged the disk to my Linux laptop and wrote
data onto the Linux file system. Mind you, I think in that partition
marked LNX\0 I even created a Linux LVM with pvcreate. Do you call that
insane? Well it probably is. :)
And as an Amiga user I could just return to you: I clicked it, it did
not warn, so all is good :)
But yeah, as mentioned I researched the topic before. And I think there
has not even been an overflow within the RDB:
> The raw, theoretical limit on the maximum device capacity is about
> 2^105 bytes:
>
> 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> bytes/sector for the HD size in struct RigidDiskBlock\0
http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)
Confirmed by:
The .ADF (Amiga Disk File) format FAQ:
http://lclevy.free.fr/adflib/adf_info.html#p6
But what do I write, you know the RDB format :)
So just do the calculation in 96 Bit and you all are set :)
Now that is a reason for 128 Bit CPUs :).
Muuhahaha.
Ciao,
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: Amiga RDB partition support for disks >= 2 TB
2018-06-28 11:30 ` Martin Steigerwald
@ 2018-06-28 11:38 ` Martin Steigerwald
2018-06-28 12:31 ` jdow
0 siblings, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28 11:38 UTC (permalink / raw)
To: Martin Steigerwald
Cc: jdow, Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
Martin Steigerwald - 28.06.18, 13:30:
> jdow - 28.06.18, 12:00:
> > On 20180628 01:16, Martin Steigerwald wrote:
> […]
>
> > >> That brings to the fore an interesting question. Why bother with
> > >> RDBs
> > >> over 2TB unless you want a disk with one single partition? This
> > >> Win10
> > >> monster I am using has a modest BIOS driver partition for the OS
> > >> and
> > >> a giant data partition. That smaller partition would easily work
> > >> with
> > >> any RDB/Filesystem combination since 2.0. So there are some good
> > >> workarounds that are probably "safer" and at least as flexible as
> > >> RDBs, one Linux has used for a very long time, too.
> > >
> > > Well, my use case was simple:
> > >
> > > I had this 2 TB disk and I choose to share it as a backup disk for
> > > Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me
> > > desk here.
> >
> > EEEEEEK! The hair on my neck is standing up straight! Have you heard
> > of SAMBA? The linux mail server firewall etc machine has an extra
> > 4TB
> > disk on it as a backup for the other systems, although a piddly 4TB
> > is small when I save the entire 3G RAID system I have. It's a proof
> > of concept so.... A full backup on a 1gig Ethernet still takes a
> > looooong time. But backing up even an 18GB disk on an Amiga via
> > 100Base-t isn't too bad. And disk speeds of the era being what they
> > were it's about all you can do anyway.
>
> Heh, the thing worked just fine in Amiga OS 4. I got away with it
> without an issue, until I plugged the disk to my Linux laptop and
> wrote data onto the Linux file system. Mind you, I think in that
> partition marked LNX\0 I even created a Linux LVM with pvcreate. Do
> you call that insane? Well it probably is. :)
>
> And as an Amiga user I could just return to you: I clicked it, it did
> not warn, so all is good :)
>
> But yeah, as mentioned I researched the topic before. And I think
> there
> has not even been an overflow within the RDB:
> > The raw, theoretical limit on the maximum device capacity is about
> > 2^105 bytes:
> >
> > 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> > bytes/sector for the HD size in struct RigidDiskBlock\0
>
> http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)
>
> Confirmed by:
>
> The .ADF (Amiga Disk File) format FAQ:
> http://lclevy.free.fr/adflib/adf_info.html#p6
>
> But what do I write, you know the RDB format :)
>
> So just do the calculation in 96 Bit and you all are set :)
For sectors.
Or
3*32+9 (for 512 bytes per sector) = 105 bits
for bytes.
> Now that is a reason for 128 Bit CPUs :).
>
> Muuhahaha.
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: Amiga RDB partition support for disks >= 2 TB
2018-06-28 11:38 ` Martin Steigerwald
@ 2018-06-28 12:31 ` jdow
0 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-28 12:31 UTC (permalink / raw)
To: Martin Steigerwald
Cc: Michael Schmitz, Geert Uytterhoeven, Matthew Wilcox,
David Sterba, Linux FS Devel, Linux Kernel Mailing List,
Jens Axboe, linux-m68k
On 20180628 04:38, Martin Steigerwald wrote:
> Martin Steigerwald - 28.06.18, 13:30:
>> jdow - 28.06.18, 12:00:
>>> On 20180628 01:16, Martin Steigerwald wrote:
>> […]
>>
>>>>> That brings to the fore an interesting question. Why bother with
>>>>> RDBs
>>>>> over 2TB unless you want a disk with one single partition? This
>>>>> Win10
>>>>> monster I am using has a modest BIOS driver partition for the OS
>>>>> and
>>>>> a giant data partition. That smaller partition would easily work
>>>>> with
>>>>> any RDB/Filesystem combination since 2.0. So there are some good
>>>>> workarounds that are probably "safer" and at least as flexible as
>>>>> RDBs, one Linux has used for a very long time, too.
>>>>
>>>> Well, my use case was simple:
>>>>
>>>> I had this 2 TB disk and I choose to share it as a backup disk for
>>>> Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me
>>>> desk here.
>>>
>>> EEEEEEK! The hair on my neck is standing up straight! Have you heard
>>> of SAMBA? The linux mail server firewall etc machine has an extra
>>> 4TB
>>> disk on it as a backup for the other systems, although a piddly 4TB
>>> is small when I save the entire 3G RAID system I have. It's a proof
>>> of concept so.... A full backup on a 1gig Ethernet still takes a
>>> looooong time. But backing up even an 18GB disk on an Amiga via
>>> 100Base-t isn't too bad. And disk speeds of the era being what they
>>> were it's about all you can do anyway.
>>
>> Heh, the thing worked just fine in Amiga OS 4. I got away with it
>> without an issue, until I plugged the disk to my Linux laptop and
>> wrote data onto the Linux file system. Mind you, I think in that
>> partition marked LNX\0 I even created a Linux LVM with pvcreate. Do
>> you call that insane? Well it probably is. :)
>>
>> And as an Amiga user I could just return to you: I clicked it, it did
>> not warn, so all is good :)
>>
>> But yeah, as mentioned I researched the topic before. And I think
>> there
>> has not even been an overflow within the RDB:
>>> The raw, theoretical limit on the maximum device capacity is about
>>> 2^105 bytes:
>>>
>>> 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
>>> bytes/sector for the HD size in struct RigidDiskBlock
>>
>> http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)
>>
>> Confirmed by:
>>
>> The .ADF (Amiga Disk File) format FAQ:
>> http://lclevy.free.fr/adflib/adf_info.html#p6
>>
>> But what do I write, you know the RDB format :)
>>
>> So just do the calculation in 96 Bit and you all are set :)
>
> For sectors.
>
> Or
>
> 3*32+9 (for 512 bytes per sector) = 105 bits
>
> for bytes.
>
>> Now that is a reason for 128 Bit CPUs :).
>>
>> Muuhahaha.
And if you make the logical block size 65536 you need 128 bits or 10^something
big = 2^128.
{^_-}
^ permalink raw reply [flat|nested] 73+ messages in thread
* Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)
2018-06-28 5:43 ` Michael Schmitz
2018-06-28 6:39 ` jdow
@ 2018-06-28 8:07 ` Martin Steigerwald
1 sibling, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-28 8:07 UTC (permalink / raw)
To: Michael Schmitz
Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
linux-m68k
Michael Schmitz - 28.06.18, 07:43:
> Joanne,
>
> Linux on m68k has supported lseek64 (or llseek) for a long time (from
> glibc version 2.1 according to what I found). About the only area
> where we are limited by 32 bits is the virtual memory size.
>
> I'm not proposing to modify the RDB format definition, though an
> extension to store 64 bit offsets separate from the 32 bit ones would
> be one way to make certain such disks are safe to use on 3.1 and
> earlier versions of AmigaOS. (Another one would be to modify the disk
> drivers on older versions to do the offset calculation in 64 bit, and
> check for overflow just as we do here. Not sure whether that's
> feasible. And as you so eloquently describe, we can't rely on users
> listening.)
I think that would be up to upstream developers to decide. That would be
AmigaOS developers and/or MorphOS, AROS developers. I could try to at
least point them to this discussion here. Whether they choose to take
the time to look at it? I don´t know. They are developing that stuff in
their spare time.
> Either way, we need the cooperation of partitioning tool writers to
> ensure partition information that is prone to overflows is never
> stored in the 32 bit, classic RDB. That appears to have failed
> already, as Martin's experience illustrates.
Heh :)
But yeah, I´d say the damage is done already.
> The raw, theoretical limit on the maximum device capacity is about
> 2^105 bytes:
>
> 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> bytes/sector for the HD size in struct RigidDiskBlock\0
http://wiki.amigaos.net/wiki/RDB
Of course that only holds true for as long as >32 bit math is used :).
So yeah.
I wonder how many Amiga users may try to use such a large disk with a
native operating system that does not support this. NSD64 and TD64 are
at least 10 years old, if not older. Just see the dates in:
http://aminet.net/search?query=nsdpatch
http://aminet.net/package/disk/misc/td64patch
Forget about 10 years. 20 years are more accurate. This stuff is
*ancient*. Officially support is in AmigaOS since version 3.5, which has
been released more than 20 years ago as well:
https://en.wikipedia.org/wiki/AmigaOS_version_history#AmigaOS_3.5,_3.9
Even AmigaOS 4.0 is older than 10 years already.
In any case, it is the responsibility of the tool that creates or change
the RDB to take care of warning the user or using a new format.
Here the kernel just reads that which already exists in the wild.
> I'm only concerned with fixing the (dangerous) but in the Linux
> partition format parser for RDB. Refusing to use any partitions that
> will cause havoc on old AmigaOS versions is all I can do to try and
> get the users' attention.
Exactly.
And I do think, that no tool on Linux can create these kind of RDBs, and
if they can… a big fat warning belongs into that tool.
Not even Media Toolbox warns about that currently, so yes… adding an
additional warning in the kernel may be helpful.
However without refusing to parse this, the user may never notice the
warning. So that is an argument for refusing to parse this without a
kernel option.
I still think that is too much hand-holding for the few native OS users
out there. Cause in order to see the warning, the user would have to
stuff the disk into a Linux machine anyway. So the user is still
perfectly able to create such an RDB on AmigaOS 4.x or even AmigaOS
3.5/3.9 or even earlier, and then stuff the disk into a machine with
AmigaOS 1.3 and probably let it go boom there. Now, how many users who
do not know about these limits will ever put the disk into a Linux
machine, receive the warning and then be saved from data destruction? I
bet: None. It is just as unlikely as it can get. :) We are talking about
maybe a few thousand Amiga users here. And there would be even the
question how many of them will try to use disks larger than 2 TiB. I bet
there will be some, but I bet there won´t be many.
So it would be way more important to warn in Media Toolbox, amiga-fdisk,
parted and whatever other partitioning tool users may use.
I´d still leave in the warning anyway, but I think the kernel option… is
over-protecting users.
Anyway your call and I perfectly get it, in case you choose to be on a
100% safe side. You are submitting the patch.
> Your warning makes me wonder whether the log message should just say
> 'report this bug to linux-m68k@vger.kernel.org' to at least try and
> educate any that respond about the dangers of their partitioning
> scheme before telling them about the override option. Problem with
> that is, in about three years no one will remember any of this ...
>
> Cheers,
>
> Michael
>
> Am 28.06.2018 um 15:44 schrieb jdow:
> > Michael, as long as m68k only supports int fseek( int ) 4 GB * block
> > size is your HARD limit. Versions that support __int64 fseek64(
> > __int64 ) can work with larger disks. RDBs could include RDSK and a
> > new set of other blocks that replace the last two characters with
> > "64" and use __int64 where needed in the various values. That way a
> > clever disk partitioner could give allow normal (32 bit) RDB
> > definitions where possible. Then at least SOME of the disk could be
> > supported AND a very clever filesystem that abstracts very large
> > disks properly could give access to the whole disk. (Read the RDBs
> > first 32 bits. Then if a filesystem or driveinit was loaded re-read
> > the RDBs to see if new 64 bit partitions are revealed.
> >
> > I could be wrong but I do not think RDBs could be safely modified
> > any
> > other way to work. And, trust me as I bet this is still true, you
> > will need a SERIOUSLY good bomb shelter on the Moon if you change
> > RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads
> > Amigados 2.4 because he wants to run a game that crashes on
> > anything newer. Then Joe got far enough something writes to the
> > disk and data is corrupted. Note further that Amigoids do NOT,
> > repeat NOT, listen to facts in such cases. Hell, some of them never
> > listened to facts about an incident at Jerry Pournelle's place when
> > a 1.1 DPaint session with Kelly Freas hung and we lost a delightful
> > drawing. Jerry reported it. Amigoids screamed. I tried to tell them
> > I was there, it was my machine, and 1.1 was, indeed, crap.
> >
> > {o.o}
> >
> > On 20180627 02:00, Michael Schmitz wrote:
> >> Joanne,
> >>
> >> I'm not at all allergic to avoiding RDB at all cost for new disks.
> >> If
> >> AmigaOS 4.1 supports more recent partition formats, all the better.
> >> This is all about supporting use of legacy RDB disks on Linux
> >> (though
> >> 2 TB does stretch the definition of 'legacy' a little). My interest
> >> in this is to ensure we can continue to use RDB format disks on
> >> m68k Amiga computers which have no other way to boot Linux from
> >> disk.
> >>
> >> Not proposing to change the RDB format at all, either. Just trying
> >> to
> >> make sure we translate RDB info into Linux 512-byte block offset
> >> and
> >> size numbers correctly. The kernel won't modify the RDB at all
> >> (intentionally, that is - with the correct choice of partition
> >> sizes,
> >> Martin might well have wiped out his RDB with the current version
> >> of
> >> the parser).
> >>
> >> The choice of refusing to mount a disk (or mounting read-only)
> >> rests
> >> with the VFS drivers alone - AFFS in that case. Not touching any of
> >> that. At partition scan time, we only have the option of making the
> >> partition available (with a warning printed), or refusing to make
> >> it
> >> available to the kernel. Once it's made available, all bets are
> >> off.
> >>
> >> From what Martin writes, his test case RDB was valid and worked as
> >>
> >> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
> >> necessary extensions to handle the large offsets resulting from 2
> >> TB
> >> disks. Not sure what safeguards are in place when connecting such a
> >> disk to older versions of AmigaOS, but that is a different matter
> >> entirely.
> >>
> >> The overflows in partition offset and size are the only ones I can
> >> see in the partition parser - there is no other overflow I've
> >> identified. I just stated that in order to place a partition
> >> towards the end of a 2 TB disk, the offset calculation will
> >> overflow regardless of what combination of rdb->rdb_BlockBytes and
> >> sector addresses stored in the>>
> >> RDB (in units of 512 byte blocks) we use:
> >> blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
> >>
> >> nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
> >> 1 -
> >>
> >> be32_to_cpu(pb->pb_Environment[9])) *
> >>
> >> be32_to_cpu(pb->pb_Environment[3]) *
> >> be32_to_cpu(pb->pb_Environment[5]) *
> >> blksize;
> >>
> >> if (!nr_sects)
> >>
> >> continue;
> >>
> >> start_sect = be32_to_cpu(pb->pb_Environment[9]) *
> >>
> >> be32_to_cpu(pb->pb_Environment[3]) *
> >> be32_to_cpu(pb->pb_Environment[5]) *
> >> blksize;
> >>
> >> But in the interest of avoiding any accidental use of a RDB
> >> partition
> >> where calculations currently overflow, I'll make the default
> >> behaviour to bail out (instead of using wrong offset or size as we
> >> currently do). Given the 'eat_my_RDB_disk=1' boot option, the user
> >> may proceed at their own risk (though I still can't see what harm
> >> should result from now translating a well formed v4.1 2 TB disk
> >> RDB correctly for the first time).
> >>
> >> Whether or not Linux correctly handles AFFS filesystems larger than
> >> 1
> >> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
> >> ought to prevent accidental use of AFFS filesystems on RDB disks
> >> which I suppose is the majority of use cases.
> >>
> >> Bugs in partitioning tools on Linux are entirely out of scope - the
> >> partitioning tools bypass the partition structure discovered by the
> >> kernel, and work straight on the raw device. No protecting against
> >> that.
> >>
> >> If you can point out a way to cause data loss with these
> >> precautions,
> >> for a disk 2 TB or larger that was partitioned and used on a recent
> >> version or AmigaOS supporting such large disks, I'd consider
> >> omitting
> >> the 'eat_my_RDB_disk' boot option, and just bail out as the only
> >> safe
> >> option.
> >>
> >> Cheers,
> >>
> >> Michael
> >>
> >> Am 27.06.2018 um 18:24 schrieb jdow:
> >>> You allergic to using a GPT solution? It will get away from some
> >>> of the evils that RDB has inherent in it because they are also
> >>> features? (Loading a filesystem or DriveInit code from RDBs is
> >>> just asking for a nearly impossible to remove malware infection.)
> >>> Furthermore, any 32 bit system that sees an RDSK block is going
> >>> to try to translate it. If you add a new RDB format you are going
> >>> to get bizarre and probably quite destructive results from the
> >>> mistake. Fail safe is a rather good notion, methinks.
> >>>
> >>> Personally I figure this is all rather surreal. 2TG of junk on an
> >>> Amiga system seems utterly outlandish to me. You cited another
> >>> overflow potential. There are at least three we've identified, I
> >>> believe. Are you 100% sure there are no more? The specific one
> >>> you mention of translating RDB to Linux has a proper solution in
> >>> the RDB reader. It should recover such overflow errors in the RDB
> >>> as it can with due care and polish. It should flag any other
> >>> overflow error it detects within the RDBs and return an error
> >>> such as to leave the disk unmounted or mounted read-only if you
> >>> feel like messing up a poor sod's backups. The simple solution is
> >>> to read each of the variables with the nominal RDB size and
> >>> convert it to uint64_t before calculating byte indices.
> >>>
> >>> However, consider my inputs as advice from an adult who has seen
> >>> the
> >>> Amiga Elephant so to speak. I am not trying to assert any control.
> >>> Do as you wish; but, I would plead with you to avoid ANY chance
> >>> you can for the user to make a bonehead stupid move and lose all
> >>> his treasured disk archives. Doing otherwise is very poor form.
> >>>
> >>> {o.o} Joanne "Said enough, she has" Dow
> >>>
> >>> On 20180626 18:07, Michael Schmitz wrote:
> >>>> Joanne,
> >>>>
> >>>> As far as I have been able to test, the change is backwards
> >>>> compatible (RDB partitions from an old disk 80 GB disk are still
> >>>> recognized OK). That"s only been done on an emulator though.
> >>>>
> >>>> Your advice about the dangers of using RDB disks that would have
> >>>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>>> well
> >>>> taken though. Maybe Martin can clarify that for me - was the 2 TB
> >>>> disk in question ever used on a 32 bit Amiga system?
> >>>>
> >>>> RDB disk format is meant for legacy use only, so I think we can
> >>>> get
> >>>> away with printing a big fat warning during boot, advising the
> >>>> user
> >>>> that the oversize RDB partition(s) scanned are not compatible
> >>>> with
> >>>> legacy 32 bit AmigaOS. With the proposed fix they will work under
> >>>> both AmigaOS 4.1 and Linux instead of truncating the first
> >>>> overflowing partition at disk end and trashing valid partitions
> >>>> that overlap, which is what Martin was after.
> >>>>
> >>>> If that still seems too risky, we can make the default behaviour
> >>>> to
> >>>> bail out once a potential overflow is detected, and allow the
> >>>> user to
> >>>> override that through a boot parameter. I'd leave that decision
> >>>> up for the code review on linux-block.
> >>>>
> >>>> Two more comments: Linux uses 512 byte block sizes for the
> >>>> partition
> >>>> start and size calculations, so a change of the RDB blocksize to
> >>>> reduce the block counts stored in the RDB would still result in
> >>>> the
> >>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
> >>>> updating (along with probably most legacy m68k partitioners).
> >>>> Adrian
> >>>> has advertised parted as replacement for the old tools - maybe
> >>>> this
> >>>> would make a nice test case for parted?
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Michael
> >>>>
> >>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> >>>>> If it is not backwards compatible I for one would refuse to use
> >>>>> it.
> >>>>> And if
> >>>>> it still mattered that much to me I'd also generate a reasonable
> >>>>> alternative. Modifying RDBs nay not be even an approximation of
> >>>>> a
> >>>>> good idea.
> >>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by
> >>>>> a
> >>>>> uint32_t
> >>>>> only system. If it is for your personal use then you're entirely
> >>>>> free to
> >>>>> reject my advice and are probably smart enough to keep it
> >>>>> working for
> >>>>> yourself.
> >>>>>
> >>>>> GPT is probably the right way to go. Preserve the ability to
> >>>>> read
> >>>>> RDBs for
> >>>>> legacy disks only.
> >>>>>
> >>>>> {^_^}
> >>>>>
> >>>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>>> Joanne,
> >>>>>>
> >>>>>> I think we all agree that doing 32 bit calculations on 512-byte
> >>>>>> block
> >>>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>>> causing
> >>>>>> the
> >>>>>> issues Martin reported. Your patch addresses that by using the
> >>>>>> correct
> >>>>>> data type for the calculations (as do other partition parsers
> >>>>>> that
> >>>>>> may
> >>>>>> have to deal with large disks) and fixes Martin's bug, so
> >>>>>> appears
> >>>>>> to be
> >>>>>> the right thing to do.
> >>>>>>
> >>>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>>> calculations
> >>>>>> don't currently overflow is not expected to cause new issues,
> >>>>>> other
> >>>>>> than
> >>>>>> enabling use of disk and partitions larger than 2 TB (which may
> >>>>>> have
> >>>>>> ramifications with filesystems on these partitions). So
> >>>>>> comptibility is
> >>>>>> preserved.
> >>>>>>
> >>>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>>> overflow
> >>>>>> issues in filesystems as well, but I can't see how the block
> >>>>>> size
> >>>>>> stored
> >>>>>> in the RDB would enforce use of the same block size in
> >>>>>> filesystems.
> >>>>>> We'll have to rely on the filesystem tools to get that right,
> >>>>>> too.
> >>>>>> Linux
> >>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this
> >>>>>> should
> >>>>>> allow partitions larger than 2 TB to work already (but I
> >>>>>> suspect Al
> >>>>>> Viro
> >>>>>> may have found a few issues when he looked at the AFFS code so
> >>>>>> I
> >>>>>> won't
> >>>>>> say more). Anyway partitioning tools and filesystems are
> >>>>>> unrelated to
> >>>>>> the Linux partition parser code which is all we aim to fix in
> >>>>>> this
> >>>>>> patch.
> >>>>>>
> >>>>>> If you feel strongly about unknown ramifications of any
> >>>>>> filesystems on
> >>>>>> partitions larger than 2 TB, say so and I'll have the kernel
> >>>>>> print a
> >>>>>> warning about these partitions.
> >>>>>>
> >>>>>> I'll get this patch tested on Martin's test case image as well
> >>>>>> as
> >>>>>> on a
> >>>>>> RDB image from a disk known to currently work under Linux
> >>>>>> (thanks
> >>>>>> Geert
> >>>>>> for the losetup hint). Can't do much more without procuring a
> >>>>>> working
> >>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I
> >>>>>> plan to
> >>>>>> use
> >>>>>> for tests is a long way away from my home indeed.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Michael
> >>>>>>
> >>>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>>> suppose.
> >>>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>>> force the
> >>>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>>> systems
> >>>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>>> blocks
> >>>>>>> list will have to go beyond a uint32_t size, for example. But
> >>>>>>> a
> >>>>>>> block
> >>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>>> tad
> >>>>>>> under
> >>>>>>> 1% of the disk all by itself. A block bitmap is not quite so
> >>>>>>> bad.
> >>>>>>> {^_-}
> >>>>>>>
> >>>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>>> make a
> >>>>>>> change. I remember thinking about this for awhile and then
> >>>>>>> determining
> >>>>>>> I REALLY did not want to think about it as my brain was
> >>>>>>> getting tied
> >>>>>>> into a gordian knot.
> >>>>>>>
> >>>>>>> {^_^}
> >>>>>>>
> >>>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>>> Joanne,
> >>>>>>>>
> >>>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>>
> >>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK
> >>>>>>>> (512)
> >>>>>>>> sdb1
> >>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>>> (DOS^C)(res
> >>>>>>>> 2 spb
> >>>>>>>> 4)
> >>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>>> [sdb]
> >>>>>>>> Attached SCSI disk
> >>>>>>>>
> >>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>>> means
> >>>>>>>> 512
> >>>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>>> block
> >>>>>>>> size.
> >>>>>>>>
> >>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>>> units.
> >>>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>>> others
> >>>>>>>> yet.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>>
> >>>>>>>> Michael
> >>>>>>>>
> >>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
wrote:
> >>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>>> system is
> >>>>>>>>> a famn
> >>>>>>>>> dool.
> >>>>>>>>>
> >>>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>>> so it
> >>>>>>>>> should
> >>>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>>> blocks is
> >>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >>>>>>>>> disk in
> >>>>>>>>> block maps.
> >>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>>
> >>>>>>>>> {^_^}
> >>>>>>>>>
> >>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>>> Hi.
> >>>>>>>>>>
> >>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>>> test results at
> >>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>>> there,
> >>>>>>>>>>> so if
> >>>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>>
> >>>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>>
> >>>>>>>>>> I do not care enough about this, in order to motivate
> >>>>>>>>>> myself
> >>>>>>>>>> preparing
> >>>>>>>>>> the a patch from Joanne Dow´s fix.
> >>>>>>>>>>
> >>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>>> Sam440ep
> >>>>>>>>>> which
> >>>>>>>>>> I still have in my apartment.
> >>>>>>>>>>
> >>>>>>>>>> So RDB support in Linux it remains broken for disks larger
> >>>>>>>>>> 2 TB,
> >>>>>>>>>> unless
> >>>>>>>>>> someone else does.
> >>>>>>>>>>
> >>>>>>>>>> Thanks.
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>>>> linux-m68k" in
> >>>>>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>>>>> More majordomo info at
> >>>>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>>>>
> >>>>>>> --
> >>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>> linux-m68k" in
> >>>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>>> More majordomo info at
> >>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>>
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>> linux-m68k" in
> >>>>> the body of a message to majordomo@vger.kernel.org
> >>>>> More majordomo info at
> >>>>> http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-27 1:07 ` Michael Schmitz
2018-06-27 6:24 ` jdow
@ 2018-06-27 7:57 ` Martin Steigerwald
2018-06-28 2:56 ` jdow
1 sibling, 1 reply; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-27 7:57 UTC (permalink / raw)
To: Michael Schmitz
Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
linux-m68k
Michael Schmitz - 27.06.18, 03:07:
> Joanne,
>
> As far as I have been able to test, the change is backwards compatible
> (RDB partitions from an old disk 80 GB disk are still recognized OK).
> That"s only been done on an emulator though.
>
> Your advice about the dangers of using RDB disks that would have
> failed the current Linux RDB parser on legacy 32 bit systems is well
> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
> in question ever used on a 32 bit Amiga system?
Sure! Are there actually *any* 64 bit Amiga systems? I am not completely
sure architecture-wise, but the operating system is still just 32-Bit.
However, on AmigaOS officially since at least 3.5/3.9 there are various
mechanisms to handle 64-bit block numbers for disk devices, called
NSD64, TD64 and scsi direct¹. One of them, NSD 64 in AmigaOS 3.5/3.9, is
part of the operating system. First via SetPatch based update to
Kickstart ROM, but with AmigaOS 4.x its just included.
[1] http://www.amigawiki.de/doku.php?id=de:system:filesystems_limits
(I do not agree with maximum filesystem size limits stated there for
Amiga Fast Filesystem (FFS), but well, for early FFS versions they could
have been true, I am pretty sure that at least with higher block sizes
you can have FFS > 2 GiB in size. And I remember having had JFXS and I
think also SFS2 partitions in AmigaOS 4 with more than 100 GiB.)
> RDB disk format is meant for legacy use only, so I think we can get
> away with printing a big fat warning during boot, advising the user
> that the oversize RDB partition(s) scanned are not compatible with
> legacy 32 bit AmigaOS. With the proposed fix they will work under both
> AmigaOS 4.1 and Linux instead of truncating the first overflowing
> partition at disk end and trashing valid partitions that overlap,
> which is what Martin was after.
They are compatible. For AmigaOS 4.x I am completely sure, cause I
installed and used the disk there. That was the original motivation for
my Linux bug report back then: I used the disk in AmigaOS 4 *just fine*
and it broke in Linux. I used Media Toolbox in order to create the RDB I
posted back then.
Its difficult to find proof for my claims online, but here is an
official one:
http://wiki.amigaos.net/wiki/RDB
Limits of filesystem sizes may of course be different, but here we are
talking about RDB.
Also this confirms that RDB only uses 512 byte block size on AmigaOS 4.
Hmmm, I even created the page back then. Interestingly at about the same
time I made the bug report about the Linux issue. The calculations in
there are not really from me. I bet I copied them over from what other
AmigaOS developers wrote on development mailing lists, at that time.
With their agreement. I think I started my research back then due by my
own question on: Will such a large disk actually work in AmigaOS?
It is all a long time ago…
> If that still seems too risky, we can make the default behaviour to
> bail out once a potential overflow is detected, and allow the user to
> override that through a boot parameter. I'd leave that decision up for
> the code review on linux-block.
>
> Two more comments: Linux uses 512 byte block sizes for the partition
> start and size calculations, so a change of the RDB blocksize to
> reduce the block counts stored in the RDB would still result in the
> same overflow. And amiga-fdisk is indeed utterly broken and needs
> updating (along with probably most legacy m68k partitioners). Adrian
> has advertised parted as replacement for the old tools - maybe this
> would make a nice test case for parted?
I always used AmigaOS based tools to partition in RDB format, cause I
never trusted amiga-fdisk :).
> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
> > If it is not backwards compatible I for one would refuse to use it.
> > And if it still mattered that much to me I'd also generate a
> > reasonable alternative. Modifying RDBs nay not be even an
> > approximation of a good idea. You'd discover that as soon as an RDB
> > uint64_t disk is tasted by a uint32_t only system. If it is for
> > your personal use then you're entirely free to reject my advice and
> > are probably smart enough to keep it working for yourself.
> >
> > GPT is probably the right way to go. Preserve the ability to read
> > RDBs for legacy disks only.
> >
> > {^_^}
> >
> > On 20180626 01:31, Michael Schmitz wrote:
> >> Joanne,
> >>
> >> I think we all agree that doing 32 bit calculations on 512-byte
> >> block
> >> addresses that overflow on disks 2 TB and larger is a bug, causing
> >> the issues Martin reported. Your patch addresses that by using the
> >> correct data type for the calculations (as do other partition
> >> parsers that may have to deal with large disks) and fixes Martin's
> >> bug, so appears to be the right thing to do.
> >>
> >> Using 64 bit data types for disks smaller than 2 TB where
> >> calculations don't currently overflow is not expected to cause new
> >> issues, other than enabling use of disk and partitions larger than
> >> 2 TB (which may have ramifications with filesystems on these
> >> partitions). So comptibility is preserved.
> >>
> >> Forcing larger block sizes might be a good strategy to avoid
> >> overflow
> >> issues in filesystems as well, but I can't see how the block size
> >> stored in the RDB would enforce use of the same block size in
> >> filesystems. We'll have to rely on the filesystem tools to get
> >> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >> limitation) so this should allow partitions larger than 2 TB to
> >> work already (but I suspect Al Viro may have found a few issues
> >> when he looked at the AFFS code so I won't say more). Anyway
> >> partitioning tools and filesystems are unrelated to the Linux
> >> partition parser code which is all we aim to fix in this patch.
> >>
> >> If you feel strongly about unknown ramifications of any filesystems
> >> on partitions larger than 2 TB, say so and I'll have the kernel
> >> print a warning about these partitions.
> >>
> >> I'll get this patch tested on Martin's test case image as well as
> >> on a RDB image from a disk known to currently work under Linux
> >> (thanks Geert for the losetup hint). Can't do much more without
> >> procuring a working Amiga disk image to use with an emulator,
> >> sorry. The Amiga I plan to use for tests is a long way away from
> >> my home indeed.
> >>
> >> Cheers,
> >>
> >> Michael
> >>
> >> Am 26.06.18 um 17:17 schrieb jdow:
> >>> As long as it preserves compatibility it should be OK, I suppose.
> >>> Personally I'd make any partitioning tool front end gently force
> >>> the
> >>> block size towards 8k as the disk size gets larger. The file
> >>> systems
> >>> may also run into 2TB issues that are not obvious. An unused
> >>> blocks
> >>> list will have to go beyond a uint32_t size, for example. But a
> >>> block
> >>> list (OFS for sure, don't remember for the newer AFS) uses a tad
> >>> under 1% of the disk all by itself. A block bitmap is not quite
> >>> so bad. {^_-}
> >>>
> >>> Just be sure you are aware of all the ramifications when you make
> >>> a
> >>> change. I remember thinking about this for awhile and then
> >>> determining I REALLY did not want to think about it as my brain
> >>> was getting tied into a gordian knot.
> >>>
> >>> {^_^}
> >>>
> >>> On 20180625 19:23, Michael Schmitz wrote:
> >>>> Joanne,
> >>>>
> >>>> Martin's boot log (including your patch) says:
> >>>>
> >>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
> >>>> sdb1
> >>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
> >>>> spb
> >>>> 4)
> >>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> >>>> Attached SCSI disk
> >>>>
> >>>> so it's indeed a case of self inflicted damage (RDSK (512) means
> >>>> 512
> >>>> byte blocks) and can be worked around by using a different block
> >>>> size.
> >>>>
> >>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>> units.
> >>>> I'll still submit a patch to Jens anyway as this may bite others
> >>>> yet.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Michael
> >>>>
> >>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
wrote:
> >>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system
> >>>>> is
> >>>>> a famn
> >>>>> dool.
> >>>>>
> >>>>> If memory serves the RDBs think in blocks rather than bytes so
> >>>>> it
> >>>>> should
> >>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
> >>>>> is
> >>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
> >>>>> block maps.
> >>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>
> >>>>> {^_^}
> >>>>>
> >>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>> Hi.
> >>>>>>
> >>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>> test results at
> >>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>> indicate the RDB parser bug is fixed by the patch given there,
> >>>>>>> so if
> >>>>>>> Martin now submits the patch, all should be well?
> >>>>>>
> >>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>
> >>>>>> I do not care enough about this, in order to motivate myself
> >>>>>> preparing the a patch from Joanne Dow´s fix.
> >>>>>>
> >>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>> Sam440ep
> >>>>>> which
> >>>>>> I still have in my apartment.
> >>>>>>
> >>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>> TB,
> >>>>>> unless
> >>>>>> someone else does.
> >>>>>>
> >>>>>> Thanks.
> >>>>>
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>> linux-m68k" in
> >>>>> the body of a message to majordomo@vger.kernel.org
> >>>>> More majordomo info at
> >>>>> http://vger.kernel.org/majordomo-info.html
> >>>
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe
> >>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> > linux-m68k" in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-27 7:57 ` moving affs + RDB partition support to staging? Martin Steigerwald
@ 2018-06-28 2:56 ` jdow
0 siblings, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-28 2:56 UTC (permalink / raw)
To: Martin Steigerwald, Michael Schmitz
Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k
FFS is limited to 2 GHz file size if you don't want any corruption using
fseek(). Otherwise it can go to 4 GHz sort of safely.
{^_^}
On 20180627 00:57, Martin Steigerwald wrote:
> Michael Schmitz - 27.06.18, 03:07:
>> Joanne,
>>
>> As far as I have been able to test, the change is backwards compatible
>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>> That"s only been done on an emulator though.
>>
>> Your advice about the dangers of using RDB disks that would have
>> failed the current Linux RDB parser on legacy 32 bit systems is well
>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>> in question ever used on a 32 bit Amiga system?
>
> Sure! Are there actually *any* 64 bit Amiga systems? I am not completely
> sure architecture-wise, but the operating system is still just 32-Bit.
>
> However, on AmigaOS officially since at least 3.5/3.9 there are various
> mechanisms to handle 64-bit block numbers for disk devices, called
> NSD64, TD64 and scsi direct¹. One of them, NSD 64 in AmigaOS 3.5/3.9, is
> part of the operating system. First via SetPatch based update to
> Kickstart ROM, but with AmigaOS 4.x its just included.
>
> [1] http://www.amigawiki.de/doku.php?id=de:system:filesystems_limits
>
> (I do not agree with maximum filesystem size limits stated there for
> Amiga Fast Filesystem (FFS), but well, for early FFS versions they could
> have been true, I am pretty sure that at least with higher block sizes
> you can have FFS > 2 GiB in size. And I remember having had JFXS and I
> think also SFS2 partitions in AmigaOS 4 with more than 100 GiB.)
>
>> RDB disk format is meant for legacy use only, so I think we can get
>> away with printing a big fat warning during boot, advising the user
>> that the oversize RDB partition(s) scanned are not compatible with
>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>> partition at disk end and trashing valid partitions that overlap,
>> which is what Martin was after.
>
> They are compatible. For AmigaOS 4.x I am completely sure, cause I
> installed and used the disk there. That was the original motivation for
> my Linux bug report back then: I used the disk in AmigaOS 4 *just fine*
> and it broke in Linux. I used Media Toolbox in order to create the RDB I
> posted back then.
>
> Its difficult to find proof for my claims online, but here is an
> official one:
>
> http://wiki.amigaos.net/wiki/RDB
>
> Limits of filesystem sizes may of course be different, but here we are
> talking about RDB.
>
> Also this confirms that RDB only uses 512 byte block size on AmigaOS 4.
>
> Hmmm, I even created the page back then. Interestingly at about the same
> time I made the bug report about the Linux issue. The calculations in
> there are not really from me. I bet I copied them over from what other
> AmigaOS developers wrote on development mailing lists, at that time.
> With their agreement. I think I started my research back then due by my
> own question on: Will such a large disk actually work in AmigaOS?
>
> It is all a long time ago…
>
>> If that still seems too risky, we can make the default behaviour to
>> bail out once a potential overflow is detected, and allow the user to
>> override that through a boot parameter. I'd leave that decision up for
>> the code review on linux-block.
>>
>> Two more comments: Linux uses 512 byte block sizes for the partition
>> start and size calculations, so a change of the RDB blocksize to
>> reduce the block counts stored in the RDB would still result in the
>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>> updating (along with probably most legacy m68k partitioners). Adrian
>> has advertised parted as replacement for the old tools - maybe this
>> would make a nice test case for parted?
>
> I always used AmigaOS based tools to partition in RDB format, cause I
> never trusted amiga-fdisk :).
>
>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>> If it is not backwards compatible I for one would refuse to use it.
>>> And if it still mattered that much to me I'd also generate a
>>> reasonable alternative. Modifying RDBs nay not be even an
>>> approximation of a good idea. You'd discover that as soon as an RDB
>>> uint64_t disk is tasted by a uint32_t only system. If it is for
>>> your personal use then you're entirely free to reject my advice and
>>> are probably smart enough to keep it working for yourself.
>>>
>>> GPT is probably the right way to go. Preserve the ability to read
>>> RDBs for legacy disks only.
>>>
>>> {^_^}
>>>
>>> On 20180626 01:31, Michael Schmitz wrote:
>>>> Joanne,
>>>>
>>>> I think we all agree that doing 32 bit calculations on 512-byte
>>>> block
>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>> the issues Martin reported. Your patch addresses that by using the
>>>> correct data type for the calculations (as do other partition
>>>> parsers that may have to deal with large disks) and fixes Martin's
>>>> bug, so appears to be the right thing to do.
>>>>
>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>> calculations don't currently overflow is not expected to cause new
>>>> issues, other than enabling use of disk and partitions larger than
>>>> 2 TB (which may have ramifications with filesystems on these
>>>> partitions). So comptibility is preserved.
>>>>
>>>> Forcing larger block sizes might be a good strategy to avoid
>>>> overflow
>>>> issues in filesystems as well, but I can't see how the block size
>>>> stored in the RDB would enforce use of the same block size in
>>>> filesystems. We'll have to rely on the filesystem tools to get
>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
>>>> limitation) so this should allow partitions larger than 2 TB to
>>>> work already (but I suspect Al Viro may have found a few issues
>>>> when he looked at the AFFS code so I won't say more). Anyway
>>>> partitioning tools and filesystems are unrelated to the Linux
>>>> partition parser code which is all we aim to fix in this patch.
>>>>
>>>> If you feel strongly about unknown ramifications of any filesystems
>>>> on partitions larger than 2 TB, say so and I'll have the kernel
>>>> print a warning about these partitions.
>>>>
>>>> I'll get this patch tested on Martin's test case image as well as
>>>> on a RDB image from a disk known to currently work under Linux
>>>> (thanks Geert for the losetup hint). Can't do much more without
>>>> procuring a working Amiga disk image to use with an emulator,
>>>> sorry. The Amiga I plan to use for tests is a long way away from
>>>> my home indeed.
>>>>
>>>> Cheers,
>>>>
>>>> Michael
>>>>
>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>> Personally I'd make any partitioning tool front end gently force
>>>>> the
>>>>> block size towards 8k as the disk size gets larger. The file
>>>>> systems
>>>>> may also run into 2TB issues that are not obvious. An unused
>>>>> blocks
>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>> block
>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>> under 1% of the disk all by itself. A block bitmap is not quite
>>>>> so bad. {^_-}
>>>>>
>>>>> Just be sure you are aware of all the ramifications when you make
>>>>> a
>>>>> change. I remember thinking about this for awhile and then
>>>>> determining I REALLY did not want to think about it as my brain
>>>>> was getting tied into a gordian knot.
>>>>>
>>>>> {^_^}
>>>>>
>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>> Joanne,
>>>>>>
>>>>>> Martin's boot log (including your patch) says:
>>>>>>
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
>>>>>> sdb1
>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
>>>>>> spb
>>>>>> 4)
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>> Attached SCSI disk
>>>>>>
>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>> 512
>>>>>> byte blocks) and can be worked around by using a different block
>>>>>> size.
>>>>>>
>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
>>>>>> units.
>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>> yet.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
> wrote:
>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system
>>>>>>> is
>>>>>>> a famn
>>>>>>> dool.
>>>>>>>
>>>>>>> If memory serves the RDBs think in blocks rather than bytes so
>>>>>>> it
>>>>>>> should
>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
>>>>>>> is
>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>> block maps.
>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>
>>>>>>> {^_^}
>>>>>>>
>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>> test results at
>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>> so if
>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>
>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>
>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>>>>
>>>>>>>> I am not even using my Amiga boxes anymore, not even the
>>>>>>>> Sam440ep
>>>>>>>> which
>>>>>>>> I still have in my apartment.
>>>>>>>>
>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
>>>>>>>> TB,
>>>>>>>> unless
>>>>>>>> someone else does.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at
>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 2:23 ` Michael Schmitz
2018-06-26 5:17 ` jdow
@ 2018-06-26 8:02 ` Martin Steigerwald
2018-06-26 8:40 ` Michael Schmitz
2018-06-26 9:31 ` jdow
1 sibling, 2 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-26 8:02 UTC (permalink / raw)
To: Michael Schmitz
Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
linux-m68k
Michael.
Michael Schmitz - 26.06.18, 04:23:
> Joanne,
>
> Martin's boot log (including your patch) says:
>
> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
> 4)
> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> Attached SCSI disk
>
> so it's indeed a case of self inflicted damage (RDSK (512) means 512
> byte blocks) and can be worked around by using a different block size.
Well I pretty much believe that this was the standard block size of the
tool I used in order to create the RDB. I think it has been Media
Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has
been deprecated by AmigaOS upstream.
Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would
have set it up differently.
Anyway: It works like this in AmigaOS 4 without any issues. With 512
Byte Blocks. I think it is good that Linux does not create data
corruption when writing to disks that work just fine in AmigaOS.
Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or
Acube Sam machines may dual boot AmigaOS and Linux on their machines.
Thanks again for putting together a patch.
Thanks,
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 8:02 ` Martin Steigerwald
@ 2018-06-26 8:40 ` Michael Schmitz
2018-06-26 9:31 ` jdow
1 sibling, 0 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-26 8:40 UTC (permalink / raw)
To: Martin Steigerwald
Cc: jdow, Geert Uytterhoeven, Matthew Wilcox, David Sterba,
Linux FS Devel, Linux Kernel Mailing List, Jens Axboe,
linux-m68k
Hi Martin,
Am 26.06.18 um 20:02 schrieb Martin Steigerwald:
> Michael.
>
> Michael Schmitz - 26.06.18, 04:23:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
> Well I pretty much believe that this was the standard block size of the
> tool I used in order to create the RDB. I think it has been Media
No offense meant - 512 bytes per block would certainly have been the
default and it certainly wasn't obvious that this needed changing.
> Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
> Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has
> been deprecated by AmigaOS upstream.
>
> Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would
> have set it up differently.
>
> Anyway: It works like this in AmigaOS 4 without any issues. With 512
> Byte Blocks. I think it is good that Linux does not create data
> corruption when writing to disks that work just fine in AmigaOS.
> Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or
> Acube Sam machines may dual boot AmigaOS and Linux on their machines.
That's the goal.
> Thanks again for putting together a patch.
The hard work had been done by Joanne and you six years ago, so no
matter at all. If Jens and the crowd at linux-block point out errors in
review, I take that as a learning experience...
Cheers,
Michael
>
> Thanks,
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging?
2018-06-26 8:02 ` Martin Steigerwald
2018-06-26 8:40 ` Michael Schmitz
@ 2018-06-26 9:31 ` jdow
1 sibling, 0 replies; 73+ messages in thread
From: jdow @ 2018-06-26 9:31 UTC (permalink / raw)
To: Martin Steigerwald, Michael Schmitz
Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k
HDToolBox is not mine. That is where the intelligence is needed.
{^_^}
On 20180626 01:02, Martin Steigerwald wrote:
> Michael.
>
> Michael Schmitz - 26.06.18, 04:23:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
>
> Well I pretty much believe that this was the standard block size of the
> tool I used in order to create the RDB. I think it has been Media
> Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
> Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has
> been deprecated by AmigaOS upstream.
>
> Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would
> have set it up differently.
>
> Anyway: It works like this in AmigaOS 4 without any issues. With 512
> Byte Blocks. I think it is good that Linux does not create data
> corruption when writing to disks that work just fine in AmigaOS.
> Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or
> Acube Sam machines may dual boot AmigaOS and Linux on their machines.
>
> Thanks again for putting together a patch.
>
> Thanks,
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
2018-06-24 9:06 ` Martin Steigerwald
2018-06-24 11:33 ` moving affs + RDB partition support to staging? jdow
2018-06-24 11:40 ` jdow
@ 2018-06-25 7:53 ` Michael Schmitz
2018-06-25 8:26 ` Martin Steigerwald
2018-06-25 8:40 ` Geert Uytterhoeven
2 siblings, 2 replies; 73+ messages in thread
From: Michael Schmitz @ 2018-06-25 7:53 UTC (permalink / raw)
To: Martin Steigerwald
Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow,
John Paul Adrian Glaubitz
Hi Martin,
OK, I'll prepare a patch and submit it to linux-block for review. I'll
have to refer to your testing back in 2012 since all I can test is
whether the patch still allows partition tables on small disks to be
recognized at this time (unless Adrian has a 2 TB disk and a SATA-SCSI
bridge to test this properly on).
Cheers,
Michael
Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> Hi.
>
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
> Ok, better be honest than having anyone waiting for it:
>
> I do not care enough about this, in order to motivate myself preparing
> the a patch from Joanne Dow´s fix.
>
> I am not even using my Amiga boxes anymore, not even the Sam440ep which
> I still have in my apartment.
>
> So RDB support in Linux it remains broken for disks larger 2 TB, unless
> someone else does.
>
> Thanks.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
2018-06-25 7:53 ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
@ 2018-06-25 8:26 ` Martin Steigerwald
2018-06-25 8:40 ` Geert Uytterhoeven
1 sibling, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-06-25 8:26 UTC (permalink / raw)
To: Michael Schmitz
Cc: Geert Uytterhoeven, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow,
John Paul Adrian Glaubitz
Hi Michael.
Michael Schmitz - 25.06.18, 09:53:
> OK, I'll prepare a patch and submit it to linux-block for review. I'll
> have to refer to your testing back in 2012 since all I can test is
> whether the patch still allows partition tables on small disks to be
> recognized at this time (unless Adrian has a 2 TB disk and a
> SATA-SCSI bridge to test this properly on).
Thank you very much.
I believe the testing I did is still valid.
Feel free to include any parts of the description of the test I made
back then into your patch description as you see it as relevant for it.
Also feel free to include my Tested-By: (probably with a hint the test
was in 2012).
I am not sure I am ready to permanently let go of (some of) the hardware
I still have collected, but at some time I may. I do have SATA-SCSI
bridges. That A4000T I have here probably could be a nice Debian m68k
build host.
Thanks,
Martin
>
> Cheers,
>
> Michael
>
> Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> > Hi.
> >
> > Michael Schmitz - 27.04.18, 04:11:
> >> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >> indicate the RDB parser bug is fixed by the patch given there, so
> >> if
> >> Martin now submits the patch, all should be well?
> >
> > Ok, better be honest than having anyone waiting for it:
> >
> > I do not care enough about this, in order to motivate myself
> > preparing the a patch from Joanne Dow�s fix.
> >
> > I am not even using my Amiga boxes anymore, not even the Sam440ep
> > which I still have in my apartment.
> >
> > So RDB support in Linux it remains broken for disks larger 2 TB,
> > unless someone else does.
> >
> > Thanks.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
2018-06-25 7:53 ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
2018-06-25 8:26 ` Martin Steigerwald
@ 2018-06-25 8:40 ` Geert Uytterhoeven
1 sibling, 0 replies; 73+ messages in thread
From: Geert Uytterhoeven @ 2018-06-25 8:40 UTC (permalink / raw)
To: Michael Schmitz
Cc: Martin Steigerwald, Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, jdow,
John Paul Adrian Glaubitz
Hi Michael,
On Mon, Jun 25, 2018 at 9:53 AM Michael Schmitz <schmitzmic@gmail.com> wrote:
> OK, I'll prepare a patch and submit it to linux-block for review. I'll
Thanks a lot!
> have to refer to your testing back in 2012 since all I can test is
> whether the patch still allows partition tables on small disks to be
> recognized at this time (unless Adrian has a 2 TB disk and a SATA-SCSI
> bridge to test this properly on).
Sparse file and loopback mounting with losetup --partscan?
> Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> > Michael Schmitz - 27.04.18, 04:11:
> >> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >> indicate the RDB parser bug is fixed by the patch given there, so if
> >> Martin now submits the patch, all should be well?
> > Ok, better be honest than having anyone waiting for it:
> >
> > I do not care enough about this, in order to motivate myself preparing
> > the a patch from Joanne Dow´s fix.
> >
> > I am not even using my Amiga boxes anymore, not even the Sam440ep which
> > I still have in my apartment.
> >
> > So RDB support in Linux it remains broken for disks larger 2 TB, unless
> > someone else does.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)
2018-04-26 11:08 ` Geert Uytterhoeven
` (2 preceding siblings ...)
2018-04-27 2:11 ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
@ 2018-04-27 8:01 ` Martin Steigerwald
3 siblings, 0 replies; 73+ messages in thread
From: Martin Steigerwald @ 2018-04-27 8:01 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Matthew Wilcox, David Sterba, Linux FS Devel,
Linux Kernel Mailing List, Jens Axboe, linux-m68k, Joanne Dow
Geert Uytterhoeven - 26.04.18, 13:08:
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
>
> <martin@lichtvoll.de> wrote:
> > You probably put your stick into a cave with ancient sleeping
> > dragons
> >
> > Added in linux-m68k mailing list, as they likely have an opinion on
> > how to treat affs + RDB partition support. Also added in Jens Axboe
> > about patching that RDB support broken with 2 TB or larger
> > harddisks issue which had been in Linux kernel for 6 years while a
> > patch exists that to my testing back then solves the issue.
[…]
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's
> still not a reason to move it to staging (hi msdos partitioning!).
What I ran into was *not* a limitation in the RDB format, but a bug in
the Linux implementation on it. After Joanne Dow´s change the 2 TB disk
was detected and handled properly by Linux. Also AmigaOS 4.x handles
those disks just well and I think also AmigaOS 3.1/3.5/3.9 supported
them, but I am not sure on the details on that, it has been a long time
since I last booted one of my Amiga systems.
Many classic Amiga users may not deal with such large disks, but AmigaOS
4 users probably still, and some of them may run Linux on their PowerPC
motherboards as well.
So I think the issue is worth fixing and am looking into submitting the
fix, which looks pretty straight-forward to me upstream unless someone
bets me to it.
Thanks,
--
Martin
^ permalink raw reply [flat|nested] 73+ messages in thread