Re: page->index limitation on 32bit system?

From: Matthew Wilcox <willy@infradead.org>
To: Erik Jensen <erikjensen@rkjnsn.net>
Cc: Qu Wenruo <quwenruo.btrfs@gmx.com>,
	Linux FS Devel <linux-fsdevel@vger.kernel.org>,
	"linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>
Subject: Re: page->index limitation on 32bit system?
Date: Fri, 19 Feb 2021 14:22:01 +0000	[thread overview]
Message-ID: <20210219142201.GU2858050@casper.infradead.org> (raw)
In-Reply-To: <927c018f-c951-c44c-698b-cb76d15d67bb@rkjnsn.net>

On Thu, Feb 18, 2021 at 01:27:09PM -0800, Erik Jensen wrote:
> On 2/18/21 4:15 AM, Matthew Wilcox wrote:
> 
> > On Thu, Feb 18, 2021 at 04:54:46PM +0800, Qu Wenruo wrote:
> > > Recently we got a strange bug report that, one 32bit systems like armv6
> > > or non-64bit x86, certain large btrfs can't be mounted.
> > > 
> > > It turns out that, since page->index is just unsigned long, and on 32bit
> > > systemts, that can just be 32bit.
> > > 
> > > And when filesystems is utilizing any page offset over 4T, page->index
> > > get truncated, causing various problems.
> > 4TB?  I think you mean 16TB (4kB * 4GB)
> > 
> > Yes, this is a known limitation.  Some vendors have gone to the trouble
> > of introducing a new page_index_t.  I'm not convinced this is a problem
> > worth solving.  There are very few 32-bit systems with this much storage
> > on a single partition (everything should work fine if you take a 20TB
> > drive and partition it into two 10TB partitions).
> For what it's worth, I'm the reporter of the original bug. My use case is a
> custom NAS system. It runs on a 32-bit ARM processor, and has 5 8TB drives,
> which I'd like to use as a single, unified storage array. I chose btrfs for
> this project due to the filesystem-integrated snapshots and checksums.
> Currently, I'm working around this issue by exporting the raw drives using
> nbd and mounting them on a 64-bit system to access the filesystem, but this
> is very inconvenient, only allows one machine to access the filesystem at a
> time, and prevents running any tools that need access to the filesystem
> (such as backup and file sync utilities) on the NAS itself.
> 
> It sounds like this limitation would also prevent me from trying to use a
> different filesystem on top of software RAID, since in that case the logical
> filesystem would still be over 16TB.
> 
> > As usual, the best solution is for people to stop buying 32-bit systems.
> I purchased this device in 2018, so it's not exactly ancient. At the time,
> it was the only SBC I could find that was low power, used ECC RAM, had a
> crypto accelerator, and had multiple sata ports with port-multiplier
> support.

I'm sorry you bought unsupported hardware.

This limitation has been known since at least 2009:
https://lore.kernel.org/lkml/19041.4714.686158.130252@notabene.brown/

In the last decade, nobody's tried to fix it in mainline that I know of.
As I said, some vendors have tried to fix it in their NAS products,
but I don't know where to find that patch any more.

https://bootlin.com/blog/large-page-support-for-nas-systems-on-32-bit-arm/
might help you, but btrfs might still contain assumptions that will trip
you up.