* memory requirements for a 400TB fs with reflinks @ 2021-03-20 18:01 Ralf Groß 2021-03-22 16:50 ` Ralf Groß 0 siblings, 1 reply; 6+ messages in thread From: Ralf Groß @ 2021-03-20 18:01 UTC (permalink / raw) To: linux-xfs Hi, I plan to deploy a couple of Linux (RHEL 8.x) server as Veeam backup repositories. Base for this might be high density server with 58 x 16TB disks, 2x RAID 60, each with its own raid controller and 28 disks. So each RAID 6 has 14 disks, + 2 globale spare. I wonder what memory requirement such a server would have, is there any special requirement regarding reflinks? I remember that xfs_repair has been a problem in the past, but my experience with this is from 10 years ago. Currently I plan to use 192GB RAM, this would be perfect as it utilizes 6 memory channels and 16GB DIMMs are not so expensive. Thanks - Ralf ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory requirements for a 400TB fs with reflinks 2021-03-20 18:01 memory requirements for a 400TB fs with reflinks Ralf Groß @ 2021-03-22 16:50 ` Ralf Groß 2021-03-22 21:50 ` Dave Chinner 2021-03-23 9:31 ` Lucas Stach 0 siblings, 2 replies; 6+ messages in thread From: Ralf Groß @ 2021-03-22 16:50 UTC (permalink / raw) To: linux-xfs No advice or rule of thumb regarding needed memory for xfs_repair? Ralf Am Sa., 20. März 2021 um 19:01 Uhr schrieb Ralf Groß <ralf.gross+xfs@gmail.com>: > > Hi, > > I plan to deploy a couple of Linux (RHEL 8.x) server as Veeam backup > repositories. Base for this might be high density server with 58 x > 16TB disks, 2x RAID 60, each with its own raid controller and 28 > disks. So each RAID 6 has 14 disks, + 2 globale spare. > > I wonder what memory requirement such a server would have, is there > any special requirement regarding reflinks? I remember that xfs_repair > has been a problem in the past, but my experience with this is from 10 > years ago. Currently I plan to use 192GB RAM, this would be perfect as > it utilizes 6 memory channels and 16GB DIMMs are not so expensive. > > Thanks - Ralf ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory requirements for a 400TB fs with reflinks 2021-03-22 16:50 ` Ralf Groß @ 2021-03-22 21:50 ` Dave Chinner 2021-03-23 9:39 ` Ralf Groß 2021-03-23 9:31 ` Lucas Stach 1 sibling, 1 reply; 6+ messages in thread From: Dave Chinner @ 2021-03-22 21:50 UTC (permalink / raw) To: Ralf Groß; +Cc: linux-xfs On Mon, Mar 22, 2021 at 05:50:55PM +0100, Ralf Groß wrote: > No advice or rule of thumb regarding needed memory for xfs_repair? People are busy, and you posted on a weekend. Have some patience, please. > Am Sa., 20. März 2021 um 19:01 Uhr schrieb Ralf Groß <ralf.gross+xfs@gmail.com>: > > > > Hi, > > > > I plan to deploy a couple of Linux (RHEL 8.x) server as Veeam backup > > repositories. Base for this might be high density server with 58 x > > 16TB disks, 2x RAID 60, each with its own raid controller and 28 > > disks. So each RAID 6 has 14 disks, + 2 globale spare. > > > > I wonder what memory requirement such a server would have, is there > > any special requirement regarding reflinks? I remember that xfs_repair > > has been a problem in the past, but my experience with this is from 10 > > years ago. Currently I plan to use 192GB RAM, this would be perfect as > > it utilizes 6 memory channels and 16GB DIMMs are not so expensive. Filesystem capacity doesn't massively affect repair memory usage these days. The amount of metadata and the type of it certainly does, though. I recently saw a 14TB filesystem require 240GB of RAM to repair because, as a hardlink based backup farm, it had hundreds of millions of directories, inodes and hardlinks in it. Resolving all those directories and hardlinks took 3 weeks and 240GB of RAM.... I've seen other broken backup server filesystems of similar size that have had close on 500GB of metadata in them, and repair needs to index and cross-reference all that metadata. Hence memory demands can be massive, even in today's terms.... Unfortunately, I haven't seen a broken filesystem containing extensive production use of reflink at that scale, so I can't really say what difference that will make to memory usage at this point in time. So there's no one answer - the amount of RAM xfs_repair might need largely depends on what you are storing in the filesystem. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory requirements for a 400TB fs with reflinks 2021-03-22 21:50 ` Dave Chinner @ 2021-03-23 9:39 ` Ralf Groß 2021-03-23 22:02 ` Dave Chinner 0 siblings, 1 reply; 6+ messages in thread From: Ralf Groß @ 2021-03-23 9:39 UTC (permalink / raw) To: linux-xfs Hi Dave, > People are busy, and you posted on a weekend. Have some patience, > please. I know, sorry for that. > .... > > So there's no one answer - the amount of RAM xfs_repair might need > largely depends on what you are storing in the filesystem. I just checked our existing backups repositories. The backup files are VMware image backup files, daily ones are smaller, weekly/GFS larger. But the are not millions of smaller files. For primary backup there are ~25.000 files in 68 TB of a 100 TB share, for a new repository with a 400 TB fs this would result in ~150.000 files. For the the secondary copy repository I see 3000 files in a 100 TB share. This would there result in ~200.000 files in a 700 TB repository. Is there any formula to calculate the memory requirement for a number of files? Ralf ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory requirements for a 400TB fs with reflinks 2021-03-23 9:39 ` Ralf Groß @ 2021-03-23 22:02 ` Dave Chinner 0 siblings, 0 replies; 6+ messages in thread From: Dave Chinner @ 2021-03-23 22:02 UTC (permalink / raw) To: Ralf Groß; +Cc: linux-xfs On Tue, Mar 23, 2021 at 10:39:27AM +0100, Ralf Groß wrote: > Hi Dave, > > > People are busy, and you posted on a weekend. Have some patience, > > please. > > I know, sorry for that. > > > .... > > > > So there's no one answer - the amount of RAM xfs_repair might need > > largely depends on what you are storing in the filesystem. > > I just checked our existing backups repositories. The backup files are > VMware image backup files, daily ones are smaller, weekly/GFS larger. > But the are not millions of smaller files. For primary backup there > are ~25.000 files in 68 TB of a 100 TB share, for a new repository > with a 400 TB fs this would result in ~150.000 files. For the the > secondary copy repository I see 3000 files in a 100 TB share. This > would there result in ~200.000 files in a 700 TB repository. Is there > any formula to calculate the memory requirement for a number of files? Worse case static data indexing memory usage can be reported by xfs_repair itself by abusing verbose reporting and memory limiting. A 500TB filesystem with 50 million zero length files in it: # xfs_repair -n -vvv -m 1 /dev/vdc Phase 1 - find and verify superblock... - reporting progress in intervals of 15 minutes - max_mem = 1024, icount = 51221120, imem = 200082, dblock = 134217727500, dmem = 65535999 Required memory for repair is greater that the maximum specified with the -m option. Please increase it to at least 64244. # Says that worst case it is going to need "dmem = 65535999" to index the space usage. That's 64GB of RAM. Inode based requirements are "imem = 200082" another 200MB for indexing 50 million inodes. Of course, there is the inodes themselves and all the other metadata that need to be brought into RAM, but that is typically paged in and out of the buffer cache that is not actually included in these memory usage counts. So for a 500GB filesystem with minimal metadata and large contiguous files as you describe you're probably only going to need a few GB of RAM to repair it. OF course, if things get broken, then you should plan for worst case minimums as described by xfs_repair above... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory requirements for a 400TB fs with reflinks 2021-03-22 16:50 ` Ralf Groß 2021-03-22 21:50 ` Dave Chinner @ 2021-03-23 9:31 ` Lucas Stach 1 sibling, 0 replies; 6+ messages in thread From: Lucas Stach @ 2021-03-23 9:31 UTC (permalink / raw) To: Ralf Groß, linux-xfs Hi Ralf, Am Montag, dem 22.03.2021 um 17:50 +0100 schrieb Ralf Groß: > No advice or rule of thumb regarding needed memory for xfs_repair? xfs_repair can be quite a memory hog, however the memory requirements are mostly related to the amount of metadata in the FS, not so much with the overall size of the FS. So a small FS with a ton of small files will require much more RAM on a repair run than a big FS with only a few big files. However, xfs_repair makes linear passes over its workingset, so it works really well with swap. Our backupservers are handling filesystems with ~400GB of metadata (size of the metadump) and are only equipped with 64GB RAM. For the worst-case where a xfs_repair run might be needed they simply have a 1TB SSD to be used as swap for the repair run. Regards, Lucas > Ralf > > > Am Sa., 20. März 2021 um 19:01 Uhr schrieb Ralf Groß <ralf.gross+xfs@gmail.com>: > > > > Hi, > > > > I plan to deploy a couple of Linux (RHEL 8.x) server as Veeam backup > > repositories. Base for this might be high density server with 58 x > > 16TB disks, 2x RAID 60, each with its own raid controller and 28 > > disks. So each RAID 6 has 14 disks, + 2 globale spare. > > > > I wonder what memory requirement such a server would have, is there > > any special requirement regarding reflinks? I remember that xfs_repair > > has been a problem in the past, but my experience with this is from 10 > > years ago. Currently I plan to use 192GB RAM, this would be perfect as > > it utilizes 6 memory channels and 16GB DIMMs are not so expensive. > > > > Thanks - Ralf ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-03-23 22:03 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-20 18:01 memory requirements for a 400TB fs with reflinks Ralf Groß 2021-03-22 16:50 ` Ralf Groß 2021-03-22 21:50 ` Dave Chinner 2021-03-23 9:39 ` Ralf Groß 2021-03-23 22:02 ` Dave Chinner 2021-03-23 9:31 ` Lucas Stach
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.