"Too many links (31)" issue

"Too many links (31)" issue

[Andy Smith]

Hi,

I have a host with an xfs filesystem on it, with about 25 million
files. It contains an rsnapshot backup that has been aggressively
deduplicated by means of hardlinks and there's probably only about 7
million unique files on there.

I'm trying to rsync it to a different host into a btrfs filesystem,
but part way through the rsync I get a "Too many links (31)" error:

rsync: [generator] link "/data/backup/rsnapshot/daily.0/chacha/var/lib/dpkg/info/.apt-utils.postrm.0" => daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm failed: Too many links (31)
Hlink node data for 219191 already has path=daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm (daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm)
rsync error: errors with program diagnostics (code 13) at hlink.c(539) [generator=3.2.3]

I searched around on this topic and found hits from 10 years ago
about maximum hardlinks per directory and being dependent upon
length of file path. Is that still relevant today?

This particular file does indeed have a huge number of hardlinks:

$ stat daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm
  File: daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm
  Size: 132             Blocks: 8          IO Block: 4096   regular file
Device: fd05h/64773d    Inode: 1342355538  Links: 7565
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-02-03 04:51:29.978915067 +0000
Modify: 2011-03-01 21:59:29.000000000 +0000
Change: 2022-01-01 17:35:06.598921506 +0000
 Birth: -
$ sudo find . -mount -samefile daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm | wc -l
7565

I guess it is some sort of template file that is littered all over
Debian systems.

Is there anything I can do to get this working?

The receiving host with the btrfs filesystem is Debian 11
(bullseye), stock Debian kernel 5.10.0-11-amd64. The btrfs
filesystem is mounted with options:

/dev/mapper/backupenc on /data/backup type btrfs (rw,noatime,compress=zstd:15,space_cache,subvolid=5,subvol=/)

As an aside, when the file is as small as 132 bytes is there
actually any advantage in hardlinking copies of it together rather
than just having multiple copies of it? Is there some minimum
file size where it's just not worth it?

(Yes I am aware of offline deuplication which works in XFS as well,
it's just that where the files are known to be entirely identical
I've found that simply hardlinking them together was faster and
easier.)

Cheers,
Andy

Re: "Too many links (31)" issue

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]

On Thu, 3 Feb 2022 16:31:08 +0000
Andy Smith <andy@strugglers.net> wrote:

> Hi,
> 
> I have a host with an xfs filesystem on it, with about 25 million
> files. It contains an rsnapshot backup that has been aggressively
> deduplicated by means of hardlinks and there's probably only about 7
> million unique files on there.
> 
> I'm trying to rsync it to a different host into a btrfs filesystem,
> but part way through the rsync I get a "Too many links (31)" error:
> 
> rsync: [generator] link "/data/backup/rsnapshot/daily.0/chacha/var/lib/dpkg/info/.apt-utils.postrm.0" => daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm failed: Too many links (31)
> Hlink node data for 219191 already has path=daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm (daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm)
> rsync error: errors with program diagnostics (code 13) at hlink.c(539) [generator=3.2.3]
> 
> I searched around on this topic and found hits from 10 years ago
> about maximum hardlinks per directory and being dependent upon
> length of file path. Is that still relevant today?
> 
> [...]
> 
> Is there anything I can do to get this working?

Hello, Have you tried the "extended_iref" mount option? 

-- 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: "Too many links (31)" issue

[Andy Smith]

Hi Lukas,

On Thu, Feb 03, 2022 at 10:15:06PM +0000, Lukas Straub wrote:
> On Thu, 3 Feb 2022 16:31:08 +0000
> Andy Smith <andy@strugglers.net> wrote:
> > I searched around on this topic and found hits from 10 years ago
> > about maximum hardlinks per directory and being dependent upon
> > length of file path. Is that still relevant today?
> > 
> > [...]
> > 
> > Is there anything I can do to get this working?
> 
> Hello, Have you tried the "extended_iref" mount option? 

During my searching I did see mention of it, but as far as I could
see this is a mkfs feature not a mount option, and it's default on
now anyway. So I already have it don't I? (how to check enabled
filesystem features?)

Anyway, it is not recognised as a mount option.

Cheers,
Andy

Re: "Too many links (31)" issue

[Filipe Manana]

On Thu, Feb 03, 2022 at 10:27:52PM +0000, Andy Smith wrote:
> Hi Lukas,
> 
> On Thu, Feb 03, 2022 at 10:15:06PM +0000, Lukas Straub wrote:
> > On Thu, 3 Feb 2022 16:31:08 +0000
> > Andy Smith <andy@strugglers.net> wrote:
> > > I searched around on this topic and found hits from 10 years ago
> > > about maximum hardlinks per directory and being dependent upon
> > > length of file path. Is that still relevant today?
> > > 
> > > [...]
> > > 
> > > Is there anything I can do to get this working?
> > 
> > Hello, Have you tried the "extended_iref" mount option? 
> 
> During my searching I did see mention of it, but as far as I could
> see this is a mkfs feature not a mount option, and it's default on
> now anyway. So I already have it don't I? (how to check enabled
> filesystem features?)

Yes, it's a feature enabled by mkfs, and it has been enabled by
default for many years now.

You can confirm if it's enabled like this:

$ cat /sys/fs/btrfs/80353f52-8425-45c2-a9a2-103c3e25ebb9/features/extended_iref 
1

Where the 8035(...) is the UUID of the mounted filesystem.
You can get the UUID using blkid:

$ blkid
(...)
/dev/sdj: UUID="80353f52-8425-45c2-a9a2-103c3e25ebb9" UUID_SUB="0d1e7dfe-6ec8-494c-8467-bc4eabca56f2" BLOCK_SIZE="4096" TYPE="btrfs"
(...)

With extent irefs, the hard link limit is 65535 (64K - 1), I just confirmed
with a quick test that it is indeed being respected:

$ mkfs.btrfs -f /dev/sdj
$ mount /dev/sdj /mnt/sdj
$ cd /mnt/sdj
$ touch foo
$ for ((i = 1; i < 65536; i++)); do echo "Iteration $i"; ln foo foo_$i; done
(...)
Iteration 65534
Iteration 65535
ln: failed to create hard link to 'foo': Too many links

$ stat foo
  File: foo
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: 2ch/44d	Inode: 257         Links: 65535
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-02-04 14:18:49.788113075 +0000
Modify: 2022-02-04 14:18:49.788113075 +0000
Change: 2022-02-04 14:21:58.432166650 +0000
 Birth: 2022-02-04 14:18:49.788113075 +0000

$ ls -1 | wc -l
65535


> 
> Anyway, it is not recognised as a mount option.
> 
> Cheers,
> Andy