Linux-Fsdevel Archive on lore.kernel.org
 help / color / Atom feed
From: jdow <jdow@earthlink.net>
To: Martin Steigerwald <martin@lichtvoll.de>,
	Michael Schmitz <schmitzmic@gmail.com>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>,
	Matthew Wilcox <willy@infradead.org>,
	David Sterba <dsterba@suse.cz>,
	Linux FS Devel <linux-fsdevel@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Jens Axboe <axboe@kernel.dk>,
	linux-m68k <linux-m68k@lists.linux-m68k.org>
Subject: Re: moving affs + RDB partition support to staging?
Date: Wed, 27 Jun 2018 19:56:38 -0700
Message-ID: <37f25a45-c9b4-da5b-e6ca-054a3b931175@earthlink.net> (raw)
In-Reply-To: <6247961.nY8s5dWVsg@merkaba>

FFS is limited to 2 GHz file size if you don't want any corruption using 
fseek(). Otherwise it can go to 4 GHz sort of safely.
{^_^}

On 20180627 00:57, Martin Steigerwald wrote:
> Michael Schmitz - 27.06.18, 03:07:
>> Joanne,
>>
>> As far as I have been able to test, the change is backwards compatible
>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>> That"s only been done on an emulator though.
>>
>> Your advice about the dangers of using RDB disks that would have
>> failed the current Linux RDB parser on legacy 32 bit systems is well
>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>> in question ever used on a 32 bit Amiga system?
> 
> Sure! Are there actually *any* 64 bit Amiga systems? I am not completely
> sure architecture-wise, but the operating system is still just 32-Bit.
> 
> However, on AmigaOS officially since at least 3.5/3.9 there are various
> mechanisms to handle 64-bit block numbers for disk devices, called
> NSD64, TD64 and scsi direct¹. One of them, NSD 64 in AmigaOS 3.5/3.9, is
> part of the operating system. First via SetPatch based update to
> Kickstart ROM, but with AmigaOS 4.x its just included.
> 
> [1] http://www.amigawiki.de/doku.php?id=de:system:filesystems_limits
> 
> (I do not agree with maximum filesystem size limits stated there for
> Amiga Fast Filesystem (FFS), but well, for early FFS versions they could
> have been true, I am pretty sure that at least with higher block sizes
> you can have FFS > 2 GiB in size. And I remember having had JFXS and I
> think also SFS2 partitions in AmigaOS 4 with more than 100 GiB.)
> 
>> RDB disk format is meant for legacy use only, so I think we can get
>> away with printing a big fat warning during boot, advising the user
>> that the oversize RDB partition(s) scanned are not compatible with
>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>> partition at disk end and trashing valid partitions that overlap,
>> which is what Martin was after.
> 
> They are compatible. For AmigaOS 4.x I am completely sure, cause I
> installed and used the disk there. That was the original motivation for
> my Linux bug report back then: I used the disk in AmigaOS 4 *just fine*
> and it broke in Linux. I used Media Toolbox in order to create the RDB I
> posted back then.
> 
> Its difficult to find proof for my claims online, but here is an
> official one:
> 
> http://wiki.amigaos.net/wiki/RDB
> 
> Limits of filesystem sizes may of course be different, but here we are
> talking about RDB.
> 
> Also this confirms that RDB only uses 512 byte block size on AmigaOS 4.
> 
> Hmmm, I even created the page back then. Interestingly at about the same
> time I made the bug report about the Linux issue. The calculations in
> there are not really from me. I bet I copied them over from what other
> AmigaOS developers wrote on development mailing lists, at that time.
> With their agreement. I think I started my research back then due by my
> own question on: Will such a large disk actually work in AmigaOS?
> 
> It is all a long time ago…
> 
>> If that still seems too risky, we can make the default behaviour to
>> bail out once a potential overflow is detected, and allow the user to
>> override that through a boot parameter. I'd leave that decision up for
>> the code review on linux-block.
>>
>> Two more comments: Linux uses 512 byte block sizes for the partition
>> start and size calculations, so a change of the RDB blocksize to
>> reduce the block counts stored in the RDB would still result in the
>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>> updating (along with probably most legacy m68k partitioners). Adrian
>> has advertised parted as replacement for the old tools - maybe this
>> would make a nice test case for parted?
> 
> I always used AmigaOS based tools to partition in RDB format, cause I
> never trusted amiga-fdisk :).
> 
>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@earthlink.net> wrote:
>>> If it is not backwards compatible I for one would refuse to use it.
>>> And if it still mattered that much to me I'd also generate a
>>> reasonable alternative. Modifying RDBs nay not be even an
>>> approximation of a good idea. You'd discover that as soon as an RDB
>>> uint64_t disk is tasted by a uint32_t only system. If it is for
>>> your personal use then you're entirely free to reject my advice and
>>> are probably smart enough to keep it working for yourself.
>>>
>>> GPT is probably the right way to go. Preserve the ability to read
>>> RDBs for legacy disks only.
>>>
>>> {^_^}
>>>
>>> On 20180626 01:31, Michael Schmitz wrote:
>>>> Joanne,
>>>>
>>>> I think we all agree that doing 32 bit calculations on 512-byte
>>>> block
>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>> the issues Martin reported. Your patch addresses that by using the
>>>> correct data type for the calculations (as do other partition
>>>> parsers that may have to deal with large disks) and fixes Martin's
>>>> bug, so appears to be the right thing to do.
>>>>
>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>> calculations don't currently overflow is not expected to cause new
>>>> issues, other than enabling use of disk and partitions larger than
>>>> 2 TB (which may have ramifications with filesystems on these
>>>> partitions). So comptibility is preserved.
>>>>
>>>> Forcing larger block sizes might be a good strategy to avoid
>>>> overflow
>>>> issues in filesystems as well, but I can't see how the block size
>>>> stored in the RDB would enforce use of the same block size in
>>>> filesystems. We'll have to rely on the filesystem tools to get
>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
>>>> limitation) so this should allow partitions larger than 2 TB to
>>>> work already (but I suspect Al Viro may have found a few issues
>>>> when he looked at the AFFS code so I won't say more). Anyway
>>>> partitioning tools and filesystems are unrelated to the Linux
>>>> partition parser code which is all we aim to fix in this patch.
>>>>
>>>> If you feel strongly about unknown ramifications of any filesystems
>>>> on partitions larger than 2 TB, say so and I'll have the kernel
>>>> print a warning about these partitions.
>>>>
>>>> I'll get this patch tested on Martin's test case image as well as
>>>> on a RDB image from a disk known to currently work under Linux
>>>> (thanks Geert for the losetup hint). Can't do much more without
>>>> procuring a working Amiga disk image to use with an emulator,
>>>> sorry. The Amiga I plan to use for tests is a long way away from
>>>> my home indeed.
>>>>
>>>> Cheers,
>>>>
>>>>       Michael
>>>>
>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>> Personally I'd make any partitioning tool front end gently force
>>>>> the
>>>>> block size towards 8k as the disk size gets larger. The file
>>>>> systems
>>>>> may also run into 2TB issues that are not obvious. An unused
>>>>> blocks
>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>> block
>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>> under 1% of the disk all by itself. A block bitmap is not quite
>>>>> so bad. {^_-}
>>>>>
>>>>> Just be sure you are aware of all the ramifications when you make
>>>>> a
>>>>> change. I remember thinking about this for awhile and then
>>>>> determining I REALLY did not want to think about it as my brain
>>>>> was getting tied into a gordian knot.
>>>>>
>>>>> {^_^}
>>>>>
>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>> Joanne,
>>>>>>
>>>>>> Martin's boot log (including your patch) says:
>>>>>>
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)
>>>>>> sdb1
>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
>>>>>> spb
>>>>>> 4)
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>> Attached SCSI disk
>>>>>>
>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>> 512
>>>>>> byte blocks) and can be worked around by using a different block
>>>>>> size.
>>>>>>
>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
>>>>>> units.
>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>> yet.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>      Michael
>>>>>>
>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@earthlink.net>
> wrote:
>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system
>>>>>>> is
>>>>>>> a famn
>>>>>>> dool.
>>>>>>>
>>>>>>> If memory serves the RDBs think in blocks rather than bytes so
>>>>>>> it
>>>>>>> should
>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
>>>>>>> is
>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>> block maps.
>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>
>>>>>>> {^_^}
>>>>>>>
>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>> test results at
>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>> so if
>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>
>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>
>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>>>>
>>>>>>>> I am not even using my Amiga boxes anymore, not even the
>>>>>>>> Sam440ep
>>>>>>>> which
>>>>>>>> I still have in my apartment.
>>>>>>>>
>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
>>>>>>>> TB,
>>>>>>>> unless
>>>>>>>> someone else does.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at
>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-m68k" in the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

  reply index

Thread overview: 73+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-25 15:46 Moving unmaintained filesystems to staging Matthew Wilcox
2018-04-25 15:47 ` Christoph Hellwig
2018-04-25 20:30 ` David Sterba
2018-04-26  2:57   ` Matthew Wilcox
2018-04-26 10:28     ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Martin Steigerwald
2018-04-26 10:45       ` moving affs + RDB partition support to staging? John Paul Adrian Glaubitz
2018-04-26 10:59         ` David Sterba
2018-04-26 11:06           ` John Paul Adrian Glaubitz
2018-05-06  0:59         ` Al Viro
2018-05-06  7:40           ` Al Viro
2018-05-06 20:46             ` Al Viro
2018-05-06 20:49               ` John Paul Adrian Glaubitz
2018-05-06 21:32               ` Al Viro
2018-05-07  2:15                 ` Al Viro
2018-05-07  2:40                   ` Michael Schmitz
2018-05-07  7:08                     ` Martin Steigerwald
2018-05-07 20:50                       ` Michael Schmitz
2018-05-07 20:56                         ` Ingo Jürgensmann
2018-05-07 20:58                           ` John Paul Adrian Glaubitz
2018-05-06  8:40           ` John Paul Adrian Glaubitz
2018-05-06 10:12           ` Martin Steigerwald
2018-04-26 11:00       ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Christoph Hellwig
2018-04-26 11:08       ` Geert Uytterhoeven
2018-04-26 23:56         ` Finn Thain
2018-04-27  1:43           ` moving affs + RDB partition support to staging? jdow
2018-04-27  1:26         ` jdow
2018-05-06  8:52           ` John Paul Adrian Glaubitz
2018-05-06 10:10             ` Martin Steigerwald
2018-05-07  4:54             ` jdow
2018-04-27  2:11         ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
2018-06-24  9:06           ` Martin Steigerwald
2018-06-24 11:33             ` moving affs + RDB partition support to staging? jdow
2018-06-24 11:40             ` jdow
2018-06-26  2:23               ` Michael Schmitz
2018-06-26  5:17                 ` jdow
2018-06-26  8:12                   ` Martin Steigerwald
2018-06-26  9:46                     ` jdow
2018-06-26  8:31                   ` Michael Schmitz
2018-06-26  9:45                     ` jdow
2018-06-27  1:07                       ` Michael Schmitz
2018-06-27  6:24                         ` jdow
2018-06-27  8:03                           ` Martin Steigerwald
2018-06-28  2:57                             ` jdow
2018-06-28  7:40                               ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
2018-06-27  9:00                           ` moving affs + RDB partition support to staging? Michael Schmitz
2018-06-28  3:44                             ` jdow
2018-06-28  5:43                               ` Michael Schmitz
2018-06-28  6:39                                 ` jdow
2018-06-28  8:16                                   ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
2018-06-28 10:00                                     ` Amiga RDB partition support for disks >= 2 TB jdow
2018-06-28 11:30                                       ` Martin Steigerwald
2018-06-28 11:38                                         ` Martin Steigerwald
2018-06-28 12:31                                           ` jdow
2018-06-28  8:07                                 ` Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?) Martin Steigerwald
2018-06-27  7:57                         ` moving affs + RDB partition support to staging? Martin Steigerwald
2018-06-28  2:56                           ` jdow [this message]
2018-06-26  8:02                 ` Martin Steigerwald
2018-06-26  8:40                   ` Michael Schmitz
2018-06-26  9:31                   ` jdow
2018-06-25  7:53             ` moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging) Michael Schmitz
2018-06-25  8:26               ` Martin Steigerwald
2018-06-25  8:40               ` Geert Uytterhoeven
2018-04-27  8:01         ` Martin Steigerwald
2018-04-26  4:58   ` Moving unmaintained filesystems to staging Nikolay Borisov
2018-04-26  5:30     ` Willy Tarreau
2018-04-26  6:11 ` Pavel Machek
2018-04-26 10:36   ` Martin Steigerwald
2018-05-03  9:18     ` Pavel Machek
2018-04-27  1:10   ` Luis R. Rodriguez
2018-04-29 12:07   ` Greg KH
2018-04-29 20:07     ` Ondrej Zary
2018-04-29 23:37       ` Greg KH
2018-05-01 10:14         ` Pavel Machek

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=37f25a45-c9b4-da5b-e6ca-054a3b931175@earthlink.net \
    --to=jdow@earthlink.net \
    --cc=axboe@kernel.dk \
    --cc=dsterba@suse.cz \
    --cc=geert@linux-m68k.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-m68k@lists.linux-m68k.org \
    --cc=martin@lichtvoll.de \
    --cc=schmitzmic@gmail.com \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Fsdevel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-fsdevel/0 linux-fsdevel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-fsdevel linux-fsdevel/ https://lore.kernel.org/linux-fsdevel \
		linux-fsdevel@vger.kernel.org
	public-inbox-index linux-fsdevel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-fsdevel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git