Corrupted files on "network" filesystem in 5.x kernels

Corrupted files on "network" filesystem in 5.x kernels

[Jan Marek]

[-- Attachment #1: Type: text/plain, Size: 4121 bytes --]

Hello lkml,

I have problem with two "network" filesystems in 5.x kernels.

I've Debian Buster server with Postfix and Dovecot, users have
Maildir mailboxes on CEPHfs. Backup of these mailboxes are making
by the borgbackup to the remote machine, using CIFS.

Debian Buster have "normal" distro kernel v. 4.19 (in my case
4.19.194-3), which works fine. I've tried to use kernel from
buster-backports, linux-image-5.10.0-0.bpo.8-amd64, which is
5.10.46-2~bpo10+1.

From time of booting this kernel, I've problems with massive corruption
of dovecot index files on CEPHfs. I've tried to set up dovecot in
the similar manner, as for NFS filesystem, although mailboxes are not
sharing by multiple machines, but it not works.

As the second problem, I've cannot do borg backup of mailboxes to
CIFS directory - borgbackup had problem with corrupted archives.

When I reboot back to 4.19.194-3, problems disappeared.

I've tried to reproduce from my work computer, where I have
Debian Buster and kernel 5.12.0-19.2-liquorix-amd64, which is
5.12-28.1~buster, and there is the same problem with CIFS
filesystem and borg backup. You can easily reproduce it:

1) mount CIFS filesystem
2) make directory on it
3) try to init borg archive direcory from this directory

In my case:

mount /mnt/smb
cd /mnt/smb/tmp
mkdir tmp
borg init -e none tmp

I've got this (the same problem, as on email machine):
Inconsistency detected. Please run "borg check /mnt/smb/tmp/tmp" - although likely this is "beyond repair".
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4455, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4387, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 154, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 276, in do_init
    with Cache(repository, key, manifest, warn_if_unencrypted=False):
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 380, in __new__
    return local()
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 374, in local
    lock_wait=lock_wait, cache_mode=cache_mode)
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 467, in __init__
    self.sync()
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 851, in sync
    self.chunks = create_master_idx(self.chunks)
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 786, in create_master_idx
    master_index_capacity = int(len(self.repository) / ChunkIndex.MAX_LOAD_FACTOR)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1009, in __len__
    self.index = self.open_index(self.get_transaction_id())
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 376, in get_transaction_id
    self.check_transaction()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 365, in check_transaction
    raise self.CheckNeeded(msg)
borg.repository.Repository.CheckNeeded: Inconsistency detected. Please run "borg check /mnt/smb/tmp/tmp" - although likely this is "beyond repair".

Platform: Linux hazard 5.12.0-19.2-liquorix-amd64 #1 ZEN SMP PREEMPT liquorix 5.12-28.1~buster (2021-07-22) x86_64
Linux: debian 10.10
Borg: 1.1.9  Python: CPython 3.7.3
PID: 3320  CWD: /mnt/smb/tmp
sys.argv: ['/usr/bin/borg', 'init', '-e', 'none', 'tmp']
SSH_ORIGINAL_COMMAND: None

You can try repair this archive directory, but it didn't happend:

borg check tmp

Data integrity error: Invalid segment entry size 0 - too small [segment 1, offset 17]
Completed repository check, errors found.

On local filesystems, there is no problem, everythink works fine
- tested on ext4, JFS and XFS.

It seems, there is problem with caching of this filesystems?

Questions are welcome.

Sincerely
Jan Marek
-- 
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[REGRESSION][BISECTED] Corrupted files on CIFS in 5.x kernels

[Jan Marek]

[-- Attachment #1: Type: text/plain, Size: 5207 bytes --]

Hello lkml,

I've did a bisect to find 'bad' commit, and I've found, that problem
commit is:

# first bad commit: [e8506d25f740fd058791cc12a6dfa9386ada6b96] smb3: make default i/o size for smb3 mounts larger

It seems, that problem is somewhere in cifs?

Detailed information about mounting this cifs filesystem:

fstab:

//some.machine/jmarek       /mnt/smb   cifs    _netdev,credentials=/etc/cred_file,forceuid,forcegid,uid=root,gid=root,vers=1.0 3 0

some.machine runs Samba (sorry, I don't know, which version,
server administrator is just now on vacancy).

Questions are welcome.

I can test patches, if you will need.

Sincerely
Jan Marek

Dne Čt, srp 12, 2021 at 08:45:42 CEST napsal Jan Marek:
> Hello lkml,
> 
> I have problem with two "network" filesystems in 5.x kernels.
> 
> I've Debian Buster server with Postfix and Dovecot, users have
> Maildir mailboxes on CEPHfs. Backup of these mailboxes are making
> by the borgbackup to the remote machine, using CIFS.
> 
> Debian Buster have "normal" distro kernel v. 4.19 (in my case
> 4.19.194-3), which works fine. I've tried to use kernel from
> buster-backports, linux-image-5.10.0-0.bpo.8-amd64, which is
> 5.10.46-2~bpo10+1.
> 
> From time of booting this kernel, I've problems with massive corruption
> of dovecot index files on CEPHfs. I've tried to set up dovecot in
> the similar manner, as for NFS filesystem, although mailboxes are not
> sharing by multiple machines, but it not works.
> 
> As the second problem, I've cannot do borg backup of mailboxes to
> CIFS directory - borgbackup had problem with corrupted archives.
> 
> When I reboot back to 4.19.194-3, problems disappeared.
> 
> I've tried to reproduce from my work computer, where I have
> Debian Buster and kernel 5.12.0-19.2-liquorix-amd64, which is
> 5.12-28.1~buster, and there is the same problem with CIFS
> filesystem and borg backup. You can easily reproduce it:
> 
> 1) mount CIFS filesystem
> 2) make directory on it
> 3) try to init borg archive direcory from this directory
> 
> In my case:
> 
> mount /mnt/smb
> cd /mnt/smb/tmp
> mkdir tmp
> borg init -e none tmp
> 
> I've got this (the same problem, as on email machine):
> Inconsistency detected. Please run "borg check /mnt/smb/tmp/tmp" - although likely this is "beyond repair".
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4455, in main
>     exit_code = archiver.run(args)
>   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4387, in run
>     return set_ec(func(args))
>   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 154, in wrapper
>     return method(self, args, repository=repository, **kwargs)
>   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 276, in do_init
>     with Cache(repository, key, manifest, warn_if_unencrypted=False):
>   File "/usr/lib/python3/dist-packages/borg/cache.py", line 380, in __new__
>     return local()
>   File "/usr/lib/python3/dist-packages/borg/cache.py", line 374, in local
>     lock_wait=lock_wait, cache_mode=cache_mode)
>   File "/usr/lib/python3/dist-packages/borg/cache.py", line 467, in __init__
>     self.sync()
>   File "/usr/lib/python3/dist-packages/borg/cache.py", line 851, in sync
>     self.chunks = create_master_idx(self.chunks)
>   File "/usr/lib/python3/dist-packages/borg/cache.py", line 786, in create_master_idx
>     master_index_capacity = int(len(self.repository) / ChunkIndex.MAX_LOAD_FACTOR)
>   File "/usr/lib/python3/dist-packages/borg/repository.py", line 1009, in __len__
>     self.index = self.open_index(self.get_transaction_id())
>   File "/usr/lib/python3/dist-packages/borg/repository.py", line 376, in get_transaction_id
>     self.check_transaction()
>   File "/usr/lib/python3/dist-packages/borg/repository.py", line 365, in check_transaction
>     raise self.CheckNeeded(msg)
> borg.repository.Repository.CheckNeeded: Inconsistency detected. Please run "borg check /mnt/smb/tmp/tmp" - although likely this is "beyond repair".
> 
> Platform: Linux hazard 5.12.0-19.2-liquorix-amd64 #1 ZEN SMP PREEMPT liquorix 5.12-28.1~buster (2021-07-22) x86_64
> Linux: debian 10.10
> Borg: 1.1.9  Python: CPython 3.7.3
> PID: 3320  CWD: /mnt/smb/tmp
> sys.argv: ['/usr/bin/borg', 'init', '-e', 'none', 'tmp']
> SSH_ORIGINAL_COMMAND: None
> 
> You can try repair this archive directory, but it didn't happend:
> 
> borg check tmp
> 
> Data integrity error: Invalid segment entry size 0 - too small [segment 1, offset 17]
> Completed repository check, errors found.
> 
> On local filesystems, there is no problem, everythink works fine
> - tested on ext4, JFS and XFS.
> 
> It seems, there is problem with caching of this filesystems?
> 
> Questions are welcome.
> 
> Sincerely
> Jan Marek
> -- 
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
> http://www.gnu.org/philosophy/no-word-attachments.cs.html



-- 
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]