All of lore.kernel.org
 help / color / mirror / Atom feed
* btrfs send and receive showing errors
@ 2023-01-04 12:54 Randy Nürnberger
  2023-01-04 13:41 ` Filipe Manana
  0 siblings, 1 reply; 13+ messages in thread
From: Randy Nürnberger @ 2023-01-04 12:54 UTC (permalink / raw)
  To: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 2279 bytes --]

Hello,

I’m in the process of copying a btrfs filesystem (a couple years old) 
from one disk to another by using btrfs send and receive on all 
snapshots. The snapshots were created by a tool every hour on the hour 
as one backup measure and automatically removed as they became older.

I got errors like the following and when I compare the copied snapshots 
with the originals, some files are missing:

ERROR: unlink █████ failed: No such file or directory
ERROR: link █████ -> █████ failed: No such file or directory
ERROR: utimes █████ failed: No such file or directory
ERROR: rmdir █████ failed: No such file or directory

Is this a known bug and how can I help diagnosing and fixing this?


# uname -a  # this is the kernel on which this originally happend
Linux arktos 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) 
x86_64 GNU/Linux


# uname -a  # I already retried everything on the latest Debian 
backports kernel with the same results
Linux arktos 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian 
6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux

# btrfs --version
btrfs-progs v5.10.1

# btrfs fi sh /mnt/randy  # this is the source
Label: none  uuid: 84bba855-578d-48db-80eb-49f1029c7543
         Total devices 2 FS bytes used 2.04TiB
         devid    1 size 4.00TiB used 2.05TiB path /dev/mapper/randy_1_crypt
         devid    2 size 4.00TiB used 2.05TiB path /dev/mapper/randy_2_crypt

# btrfs fi df /mnt/randy
Data, RAID1: total=2.02TiB, used=2.02TiB
System, RAID1: total=8.00MiB, used=320.00KiB
Metadata, RAID1: total=25.00GiB, used=22.99GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


# btrfs fi sh /mnt/intern  # this is the target
Label: none  uuid: ebb94498-644c-42cd-892f-37886173523c
         Total devices 2 FS bytes used 1.91TiB
         devid    1 size 8.00TiB used 1.91TiB path 
/dev/mapper/vg_arktos_hdd_b-lv_randy_1
         devid    2 size 8.00TiB used 1.91TiB path 
/dev/mapper/vg_arktos_hdd_b-lv_randy_2

# btrfs fi df /mnt/intern
Data, RAID1: total=1.89TiB, used=1.89TiB
System, RAID1: total=8.00MiB, used=288.00KiB
Metadata, RAID1: total=21.00GiB, used=20.76GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

[-- Attachment #2: dmesg.log --]
[-- Type: text/x-log, Size: 90215 bytes --]

[    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-15
[    0.000000] Linux version 6.0.0-0.deb11.6-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT_DYNAMIC Debian 6.0.12-1~bpo11+1 (2022-12-19)
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.0.0-0.deb11.6-amd64 root=/dev/mapper/vg_arktos_ssd-lv_system ro rootflags=subvol=system quiet ip=192.168.20.10::192.168.20.1:255.255.255.0::enp1s0f0
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
[    0.000000] signal: max sigframe size: 2032
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000005efff] usable
[    0.000000] BIOS-e820: [mem 0x000000000005f000-0x000000000005ffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000060000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000008723efff] usable
[    0.000000] BIOS-e820: [mem 0x000000008723f000-0x000000008ae40fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000008ae41000-0x000000008afb5fff] usable
[    0.000000] BIOS-e820: [mem 0x000000008afb6000-0x000000008b396fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008b397000-0x000000008bb25fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000008bb26000-0x000000008bc0efff] type 20
[    0.000000] BIOS-e820: [mem 0x000000008bc0f000-0x000000008bc0ffff] usable
[    0.000000] BIOS-e820: [mem 0x000000008bc10000-0x000000008f7fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000086c7fffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] efi: EFI v2.70 by American Megatrends
[    0.000000] efi: ACPI 2.0=0x8b2c9000 ACPI=0x8b2c9000 SMBIOS=0x8b93a000 SMBIOS 3.0=0x8b939000 MEMATTR=0x830c4018 MOKvar=0x8104c000 
[    0.000000] secureboot: Secure boot disabled
[    0.000000] SMBIOS 3.2.0 present.
[    0.000000] DMI: Supermicro Super Server/X11SCA, BIOS 1.5 10/05/2020
[    0.000000] tsc: Detected 3500.000 MHz processor
[    0.000000] tsc: Detected 3499.912 MHz TSC
[    0.000007] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000009] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000016] last_pfn = 0x86c800 max_arch_pfn = 0x400000000
[    0.000139] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.000734] last_pfn = 0x8bc10 max_arch_pfn = 0x400000000
[    0.007103] found SMP MP-table at [mem 0x000fcec0-0x000fcecf]
[    0.007117] e820: update [mem 0x8104c000-0x8104efff] usable ==> reserved
[    0.007129] Using GB pages for direct mapping
[    0.007503] RAMDISK: [mem 0x31411000-0x349fffff]
[    0.007506] ACPI: Early table checksum verification disabled
[    0.007508] ACPI: RSDP 0x000000008B2C9000 000024 (v02 SUPERM)
[    0.007511] ACPI: XSDT 0x000000008B2C90C0 000104 (v01 SUPERM SUPERM   01072009 AMI  00010013)
[    0.007515] ACPI: FACP 0x000000008B307220 000114 (v06                 01072009 AMI  00010013)
[    0.007520] ACPI: DSDT 0x000000008B2C9260 03DFBA (v02 SUPERM SMCI--MB 01072009 INTL 20160527)
[    0.007522] ACPI: FACS 0x000000008B396F80 000040
[    0.007524] ACPI: APIC 0x000000008B307338 0000F4 (v04                 01072009 AMI  00010013)
[    0.007527] ACPI: FPDT 0x000000008B307430 000044 (v01                 01072009 AMI  00010013)
[    0.007529] ACPI: FIDT 0x000000008B307478 00009C (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
[    0.007531] ACPI: MCFG 0x000000008B307518 00003C (v01 SUPERM SMCI--MB 01072009 MSFT 00000097)
[    0.007534] ACPI: SSDT 0x000000008B307558 001B26 (v02 CpuRef CpuSsdt  00003000 INTL 20160527)
[    0.007536] ACPI: SSDT 0x000000008B309080 0031C6 (v02 SaSsdt SaSsdt   00003000 INTL 20160527)
[    0.007538] ACPI: SSDT 0x000000008B30C248 0026A5 (v02 PegSsd PegSsdt  00001000 INTL 20160527)
[    0.007541] ACPI: HPET 0x000000008B30E8F0 000038 (v01 SUPERM SMCI--MB 00000002      01000013)
[    0.007543] ACPI: SSDT 0x000000008B30E928 001CD5 (v02 SUPERM CflS_Rvp 00001000 INTL 20160527)
[    0.007545] ACPI: SSDT 0x000000008B310600 000FAE (v02 SUPERM Ther_Rvp 00001000 INTL 20160527)
[    0.007548] ACPI: SSDT 0x000000008B3115B0 002FCF (v02 INTEL  xh_cfsd4 00000000 INTL 20160527)
[    0.007550] ACPI: UEFI 0x000000008B314580 000042 (v01 SUPERM SMCI--MB 00000002      01000013)
[    0.007552] ACPI: LPIT 0x000000008B3145C8 000094 (v01 SUPERM SMCI--MB 00000002      01000013)
[    0.007555] ACPI: SSDT 0x000000008B314660 0027DE (v02 SUPERM PtidDevc 00001000 INTL 20160527)
[    0.007557] ACPI: SSDT 0x000000008B316E40 0014E2 (v02 SUPERM TbtTypeC 00000000 INTL 20160527)
[    0.007559] ACPI: DBGP 0x000000008B318328 000034 (v01 SUPERM SMCI--MB 00000002      01000013)
[    0.007561] ACPI: DBG2 0x000000008B318360 000054 (v00 SUPERM SMCI--MB 00000002      01000013)
[    0.007564] ACPI: SSDT 0x000000008B3183B8 001B67 (v02 SUPERM UsbCTabl 00001000 INTL 20160527)
[    0.007566] ACPI: SSDT 0x000000008B319F20 000144 (v02 Intel  ADebTabl 00001000 INTL 20160527)
[    0.007568] ACPI: BGRT 0x000000008B31A068 000038 (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
[    0.007571] ACPI: ASF! 0x000000008B31A0A0 000074 (v32 SUPERM SMCI--MB 00000002      01000013)
[    0.007573] ACPI: DMAR 0x000000008B31A118 0000C8 (v01 INTEL  EDK2     00000002      01000013)
[    0.007575] ACPI: WSMT 0x000000008B31A1E0 000028 (v01 &}              01072009 AMI  00010013)
[    0.007578] ACPI: EINJ 0x000000008B31A208 000130 (v01 AMI    AMI.EINJ 00000000 AMI. 00000000)
[    0.007580] ACPI: ERST 0x000000008B31A338 000230 (v01 AMIER  AMI.ERST 00000000 AMI. 00000000)
[    0.007582] ACPI: BERT 0x000000008B31A568 000030 (v01 AMI    AMI.BERT 00000000 AMI. 00000000)
[    0.007585] ACPI: HEST 0x000000008B31A598 00027C (v01 AMI    AMI.HEST 00000000 AMI. 00000000)
[    0.007586] ACPI: Reserving FACP table memory at [mem 0x8b307220-0x8b307333]
[    0.007588] ACPI: Reserving DSDT table memory at [mem 0x8b2c9260-0x8b307219]
[    0.007588] ACPI: Reserving FACS table memory at [mem 0x8b396f80-0x8b396fbf]
[    0.007589] ACPI: Reserving APIC table memory at [mem 0x8b307338-0x8b30742b]
[    0.007590] ACPI: Reserving FPDT table memory at [mem 0x8b307430-0x8b307473]
[    0.007590] ACPI: Reserving FIDT table memory at [mem 0x8b307478-0x8b307513]
[    0.007591] ACPI: Reserving MCFG table memory at [mem 0x8b307518-0x8b307553]
[    0.007592] ACPI: Reserving SSDT table memory at [mem 0x8b307558-0x8b30907d]
[    0.007592] ACPI: Reserving SSDT table memory at [mem 0x8b309080-0x8b30c245]
[    0.007593] ACPI: Reserving SSDT table memory at [mem 0x8b30c248-0x8b30e8ec]
[    0.007593] ACPI: Reserving HPET table memory at [mem 0x8b30e8f0-0x8b30e927]
[    0.007594] ACPI: Reserving SSDT table memory at [mem 0x8b30e928-0x8b3105fc]
[    0.007595] ACPI: Reserving SSDT table memory at [mem 0x8b310600-0x8b3115ad]
[    0.007595] ACPI: Reserving SSDT table memory at [mem 0x8b3115b0-0x8b31457e]
[    0.007596] ACPI: Reserving UEFI table memory at [mem 0x8b314580-0x8b3145c1]
[    0.007597] ACPI: Reserving LPIT table memory at [mem 0x8b3145c8-0x8b31465b]
[    0.007597] ACPI: Reserving SSDT table memory at [mem 0x8b314660-0x8b316e3d]
[    0.007598] ACPI: Reserving SSDT table memory at [mem 0x8b316e40-0x8b318321]
[    0.007599] ACPI: Reserving DBGP table memory at [mem 0x8b318328-0x8b31835b]
[    0.007599] ACPI: Reserving DBG2 table memory at [mem 0x8b318360-0x8b3183b3]
[    0.007600] ACPI: Reserving SSDT table memory at [mem 0x8b3183b8-0x8b319f1e]
[    0.007601] ACPI: Reserving SSDT table memory at [mem 0x8b319f20-0x8b31a063]
[    0.007601] ACPI: Reserving BGRT table memory at [mem 0x8b31a068-0x8b31a09f]
[    0.007602] ACPI: Reserving ASF! table memory at [mem 0x8b31a0a0-0x8b31a113]
[    0.007603] ACPI: Reserving DMAR table memory at [mem 0x8b31a118-0x8b31a1df]
[    0.007603] ACPI: Reserving WSMT table memory at [mem 0x8b31a1e0-0x8b31a207]
[    0.007604] ACPI: Reserving EINJ table memory at [mem 0x8b31a208-0x8b31a337]
[    0.007605] ACPI: Reserving ERST table memory at [mem 0x8b31a338-0x8b31a567]
[    0.007605] ACPI: Reserving BERT table memory at [mem 0x8b31a568-0x8b31a597]
[    0.007606] ACPI: Reserving HEST table memory at [mem 0x8b31a598-0x8b31a813]
[    0.007931] No NUMA configuration found
[    0.007931] Faking a node at [mem 0x0000000000000000-0x000000086c7fffff]
[    0.007939] NODE_DATA(0) allocated [mem 0x86c7d5000-0x86c7fffff]
[    0.008207] Zone ranges:
[    0.008207]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.008209]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.008210]   Normal   [mem 0x0000000100000000-0x000000086c7fffff]
[    0.008212]   Device   empty
[    0.008212] Movable zone start for each node
[    0.008215] Early memory node ranges
[    0.008215]   node   0: [mem 0x0000000000001000-0x000000000005efff]
[    0.008216]   node   0: [mem 0x0000000000060000-0x000000000009ffff]
[    0.008217]   node   0: [mem 0x0000000000100000-0x000000008723efff]
[    0.008218]   node   0: [mem 0x000000008ae41000-0x000000008afb5fff]
[    0.008219]   node   0: [mem 0x000000008bc0f000-0x000000008bc0ffff]
[    0.008219]   node   0: [mem 0x0000000100000000-0x000000086c7fffff]
[    0.008222] Initmem setup node 0 [mem 0x0000000000001000-0x000000086c7fffff]
[    0.008225] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.008227] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.008250] On node 0, zone DMA: 96 pages in unavailable ranges
[    0.011518] On node 0, zone DMA32: 15362 pages in unavailable ranges
[    0.011542] On node 0, zone DMA32: 3161 pages in unavailable ranges
[    0.011883] On node 0, zone Normal: 17392 pages in unavailable ranges
[    0.011987] On node 0, zone Normal: 14336 pages in unavailable ranges
[    0.012021] Reserving Intel graphics memory at [mem 0x8d800000-0x8f7fffff]
[    0.012975] ACPI: PM-Timer IO Port: 0x1808
[    0.012979] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.012981] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.012981] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.012982] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.012982] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.012983] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.012984] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.012984] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
[    0.012985] ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
[    0.012985] ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
[    0.012986] ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
[    0.012986] ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
[    0.013048] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
[    0.013051] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.013052] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.013055] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.013056] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.013061] e820: update [mem 0x82633000-0x82873fff] usable ==> reserved
[    0.013071] TSC deadline timer available
[    0.013072] smpboot: Allowing 12 CPUs, 0 hotplug CPUs
[    0.013085] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.013087] PM: hibernation: Registered nosave memory: [mem 0x0005f000-0x0005ffff]
[    0.013088] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[    0.013090] PM: hibernation: Registered nosave memory: [mem 0x8104c000-0x8104efff]
[    0.013091] PM: hibernation: Registered nosave memory: [mem 0x82633000-0x82873fff]
[    0.013092] PM: hibernation: Registered nosave memory: [mem 0x8723f000-0x8ae40fff]
[    0.013093] PM: hibernation: Registered nosave memory: [mem 0x8afb6000-0x8b396fff]
[    0.013094] PM: hibernation: Registered nosave memory: [mem 0x8b397000-0x8bb25fff]
[    0.013094] PM: hibernation: Registered nosave memory: [mem 0x8bb26000-0x8bc0efff]
[    0.013096] PM: hibernation: Registered nosave memory: [mem 0x8bc10000-0x8f7fffff]
[    0.013096] PM: hibernation: Registered nosave memory: [mem 0x8f800000-0xdfffffff]
[    0.013097] PM: hibernation: Registered nosave memory: [mem 0xe0000000-0xefffffff]
[    0.013097] PM: hibernation: Registered nosave memory: [mem 0xf0000000-0xfdffffff]
[    0.013098] PM: hibernation: Registered nosave memory: [mem 0xfe000000-0xfe010fff]
[    0.013098] PM: hibernation: Registered nosave memory: [mem 0xfe011000-0xfebfffff]
[    0.013099] PM: hibernation: Registered nosave memory: [mem 0xfec00000-0xfec00fff]
[    0.013099] PM: hibernation: Registered nosave memory: [mem 0xfec01000-0xfecfffff]
[    0.013100] PM: hibernation: Registered nosave memory: [mem 0xfed00000-0xfed03fff]
[    0.013100] PM: hibernation: Registered nosave memory: [mem 0xfed04000-0xfedfffff]
[    0.013101] PM: hibernation: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[    0.013101] PM: hibernation: Registered nosave memory: [mem 0xfee01000-0xfeffffff]
[    0.013102] PM: hibernation: Registered nosave memory: [mem 0xff000000-0xffffffff]
[    0.013103] [mem 0x8f800000-0xdfffffff] available for PCI devices
[    0.013104] Booting paravirtualized kernel on bare hardware
[    0.013106] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.016684] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:12 nr_cpu_ids:12 nr_node_ids:1
[    0.016920] percpu: Embedded 61 pages/cpu s212992 r8192 d28672 u262144
[    0.016925] pcpu-alloc: s212992 r8192 d28672 u262144 alloc=1*2097152
[    0.016927] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 -- -- -- -- 
[    0.016954] Fallback order for Node 0: 0 
[    0.016956] Built 1 zonelists, mobility grouping on.  Total pages: 8207814
[    0.016957] Policy zone: Normal
[    0.016958] Kernel command line: BOOT_IMAGE=/vmlinuz-6.0.0-0.deb11.6-amd64 root=/dev/mapper/vg_arktos_ssd-lv_system ro rootflags=subvol=system quiet ip=192.168.20.10::192.168.20.1:255.255.255.0::enp1s0f0
[    0.017004] Unknown kernel command line parameters "BOOT_IMAGE=/vmlinuz-6.0.0-0.deb11.6-amd64 ip=192.168.20.10::192.168.20.1:255.255.255.0::enp1s0f0", will be passed to user space.
[    0.018194] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.018791] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.018888] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.018895] software IO TLB: area num 16.
[    0.041432] Memory: 2145004K/33353036K available (12294K kernel code, 2266K rwdata, 8576K rodata, 2724K init, 5348K bss, 817416K reserved, 0K cma-reserved)
[    0.042288] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=12, Nodes=1
[    0.042320] Kernel/User page tables isolation: enabled
[    0.042333] ftrace: allocating 39425 entries in 155 pages
[    0.047800] ftrace: allocated 155 pages with 5 groups
[    0.048340] Dynamic Preempt: voluntary
[    0.048411] rcu: Preemptible hierarchical RCU implementation.
[    0.048411] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=12.
[    0.048412] 	Trampoline variant of Tasks RCU enabled.
[    0.048413] 	Rude variant of Tasks RCU enabled.
[    0.048413] 	Tracing variant of Tasks RCU enabled.
[    0.048414] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.048414] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=12
[    0.052357] NR_IRQS: 524544, nr_irqs: 2152, preallocated irqs: 16
[    0.052627] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.052893] random: crng init done
[    0.052912] Console: colour dummy device 80x25
[    0.052927] printk: console [tty0] enabled
[    0.052940] ACPI: Core revision 20220331
[    0.053352] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[    0.053490] APIC: Switch to symmetric I/O mode setup
[    0.053492] DMAR: Host address width 39
[    0.053493] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.053497] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.053499] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.053502] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.053504] DMAR: RMRR base: 0x0000008b6ac000 end: 0x0000008b8f5fff
[    0.053505] DMAR: RMRR base: 0x0000008d000000 end: 0x0000008f7fffff
[    0.053506] DMAR: RMRR base: 0x0000008adc0000 end: 0x0000008ae3ffff
[    0.053508] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.053509] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.053510] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.056633] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.056634] x2apic enabled
[    0.056666] Switched APIC routing to cluster x2apic.
[    0.065941] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.085480] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x3272fd97217, max_idle_ns: 440795241220 ns
[    0.085484] Calibrating delay loop (skipped), value calculated using timer frequency.. 6999.82 BogoMIPS (lpj=13999648)
[    0.085487] pid_max: default: 32768 minimum: 301
[    0.088988] LSM: Security Framework initializing
[    0.088999] landlock: Up and running.
[    0.089000] Yama: disabled by default; enable with sysctl kernel.yama.*
[    0.089020] AppArmor: AppArmor initialized
[    0.089021] TOMOYO Linux initialized
[    0.089026] LSM support for eBPF active
[    0.089076] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.089113] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.089285] x86/cpu: SGX disabled by BIOS.
[    0.089292] CPU0: Thermal monitoring enabled (TM1)
[    0.089331] process: using mwait in idle threads
[    0.089332] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.089333] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.089337] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.089338] Spectre V2 : Mitigation: IBRS
[    0.089339] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.089340] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.089341] RETBleed: Mitigation: IBRS
[    0.089342] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.089343] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
[    0.089349] MDS: Mitigation: Clear CPU buffers
[    0.089349] TAA: Mitigation: TSX disabled
[    0.089350] MMIO Stale Data: Mitigation: Clear CPU buffers
[    0.089354] SRBDS: Mitigation: Microcode
[    0.102658] Freeing SMP alternatives memory: 36K
[    0.103273] smpboot: CPU0: Intel(R) Xeon(R) E-2146G CPU @ 3.50GHz (family: 0x6, model: 0x9e, stepping: 0xa)
[    0.103379] cblist_init_generic: Setting adjustable number of callback queues.
[    0.103381] cblist_init_generic: Setting shift to 4 and lim to 1.
[    0.103393] cblist_init_generic: Setting shift to 4 and lim to 1.
[    0.103405] cblist_init_generic: Setting shift to 4 and lim to 1.
[    0.103418] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.
[    0.103439] ... version:                4
[    0.103440] ... bit width:              48
[    0.103440] ... generic registers:      4
[    0.103441] ... value mask:             0000ffffffffffff
[    0.103442] ... max period:             00007fffffffffff
[    0.103443] ... fixed-purpose events:   3
[    0.103443] ... event mask:             000000070000000f
[    0.103534] Estimated ratio of average max frequency by base frequency (times 1024): 1258
[    0.103551] rcu: Hierarchical SRCU implementation.
[    0.103552] rcu: 	Max phase no-delay instances is 1000.
[    0.104393] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[    0.104484] smp: Bringing up secondary CPUs ...
[    0.104557] x86: Booting SMP configuration:
[    0.104558] .... node  #0, CPUs:        #1  #2  #3  #4  #5  #6
[    0.113567] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[    0.113567] MMIO Stale Data CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/processor_mmio_stale_data.html for more details.
[    0.113581]   #7  #8  #9 #10 #11
[    0.114019] smp: Brought up 1 node, 12 CPUs
[    0.114019] smpboot: Max logical packages: 1
[    0.114019] smpboot: Total of 12 processors activated (83997.88 BogoMIPS)
[    0.148503] node 0 deferred pages initialised in 28ms
[    0.148503] devtmpfs: initialized
[    0.148503] x86/mm: Memory block size: 128MB
[    0.151787] ACPI: PM: Registering ACPI NVS region [mem 0x8afb6000-0x8b396fff] (4067328 bytes)
[    0.151787] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.151787] futex hash table entries: 4096 (order: 6, 262144 bytes, linear)
[    0.151787] pinctrl core: initialized pinctrl subsystem
[    0.151787] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.151787] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations
[    0.151787] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.151787] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.151787] audit: initializing netlink subsys (disabled)
[    0.151787] audit: type=2000 audit(1672809400.088:1): state=initialized audit_enabled=0 res=1
[    0.151787] thermal_sys: Registered thermal governor 'fair_share'
[    0.151787] thermal_sys: Registered thermal governor 'bang_bang'
[    0.151787] thermal_sys: Registered thermal governor 'step_wise'
[    0.151787] thermal_sys: Registered thermal governor 'user_space'
[    0.151787] thermal_sys: Registered thermal governor 'power_allocator'
[    0.151787] cpuidle: using governor ladder
[    0.151787] cpuidle: using governor menu
[    0.151787] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.151787] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.151787] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.151787] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[    0.151787] PCI: Using configuration type 1 for base access
[    0.153764] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.154311] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.154316] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.154316] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.154316] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.154316] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    0.154316] ACPI: Added _OSI(Module Device)
[    0.154316] ACPI: Added _OSI(Processor Device)
[    0.154316] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.154316] ACPI: Added _OSI(Processor Aggregator Device)
[    0.154316] ACPI: Added _OSI(Linux-Dell-Video)
[    0.154316] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.154316] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.220037] ACPI: 11 ACPI AML tables successfully acquired and loaded
[    0.227691] ACPI: Dynamic OEM Table Load:
[    0.227691] ACPI: SSDT 0xFFFF89588C5CFA00 0000F4 (v02 PmRef  Cpu0Psd  00003000 INTL 20160527)
[    0.229511] ACPI: \_SB_.PR00: _OSC native thermal LVT Acked
[    0.231009] ACPI: Dynamic OEM Table Load:
[    0.231014] ACPI: SSDT 0xFFFF89588C5EF400 000400 (v02 PmRef  Cpu0Cst  00003001 INTL 20160527)
[    0.232539] ACPI: Dynamic OEM Table Load:
[    0.232545] ACPI: SSDT 0xFFFF895141087800 0006AA (v02 PmRef  Cpu0Ist  00003000 INTL 20160527)
[    0.234155] ACPI: Dynamic OEM Table Load:
[    0.234160] ACPI: SSDT 0xFFFF895140A3E600 000149 (v02 PmRef  Cpu0Hwp  00003000 INTL 20160527)
[    0.235603] ACPI: Dynamic OEM Table Load:
[    0.235609] ACPI: SSDT 0xFFFF89588C5F6800 000724 (v02 PmRef  HwpLvt   00003000 INTL 20160527)
[    0.237274] ACPI: Dynamic OEM Table Load:
[    0.237280] ACPI: SSDT 0xFFFF89588C5F0000 0005FC (v02 PmRef  ApIst    00003000 INTL 20160527)
[    0.238862] ACPI: Dynamic OEM Table Load:
[    0.238867] ACPI: SSDT 0xFFFF89588C5E9000 000317 (v02 PmRef  ApHwp    00003000 INTL 20160527)
[    0.240448] ACPI: Dynamic OEM Table Load:
[    0.240454] ACPI: SSDT 0xFFFF89514011D000 000AB0 (v02 PmRef  ApPsd    00003000 INTL 20160527)
[    0.242456] ACPI: Dynamic OEM Table Load:
[    0.242462] ACPI: SSDT 0xFFFF89588C5E9400 00030A (v02 PmRef  ApCst    00003000 INTL 20160527)
[    0.248982] ACPI: Interpreter enabled
[    0.249018] ACPI: PM: (supports S0 S3 S4 S5)
[    0.249019] ACPI: Using IOAPIC for interrupt routing
[    0.249050] HEST: Enabling Firmware First mode for corrected errors.
[    0.249097] mce: [Firmware Bug]: Ignoring request to disable invalid MCA bank 12.
[    0.249098] mce: [Firmware Bug]: Ignoring request to disable invalid MCA bank 13.
[    0.249099] mce: [Firmware Bug]: Ignoring request to disable invalid MCA bank 14.
[    0.249139] HEST: Table parsing has been initialized.
[    0.249191] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
[    0.249193] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.249194] PCI: Using E820 reservations for host bridge windows
[    0.250346] ACPI: Enabled 9 GPEs in block 00 to 7F
[    0.264992] ACPI: PM: Power Resource [USBC]
[    0.265145] ACPI: PM: Power Resource [PAUD]
[    0.268196] ACPI: PM: Power Resource [SPR0]
[    0.268352] ACPI: PM: Power Resource [SPR1]
[    0.268642] ACPI: PM: Power Resource [ZPDR]
[    0.268802] ACPI: PM: Power Resource [SPR2]
[    0.268990] ACPI: PM: Power Resource [SPR5]
[    0.269195] ACPI: PM: Power Resource [V0PR]
[    0.269565] ACPI: PM: Power Resource [V1PR]
[    0.269925] ACPI: PM: Power Resource [V2PR]
[    0.272472] ACPI: PM: Power Resource [PXTC]
[    0.275608] ACPI: PM: Power Resource [WRST]
[    0.278544] ACPI: PM: Power Resource [FN00]
[    0.278618] ACPI: PM: Power Resource [FN01]
[    0.278687] ACPI: PM: Power Resource [FN02]
[    0.278758] ACPI: PM: Power Resource [FN03]
[    0.278828] ACPI: PM: Power Resource [FN04]
[    0.279594] ACPI: PM: Power Resource [PIN]
[    0.279762] ACPI: PM: Power Resource [SPR3]
[    0.280112] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe])
[    0.280118] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.281685] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug AER]
[    0.284816] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability LTR]
[    0.284818] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
[    0.286290] PCI host bridge to bus 0000:00
[    0.286292] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.286294] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.286295] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.286296] pci_bus 0000:00: root bus resource [mem 0x8f800000-0xdfffffff window]
[    0.286298] pci_bus 0000:00: root bus resource [mem 0xfc800000-0xfe7fffff window]
[    0.286299] pci_bus 0000:00: root bus resource [bus 00-fe]
[    0.286390] pci 0000:00:00.0: [8086:3ec6] type 00 class 0x060000
[    0.286457] pci 0000:00:01.0: [8086:1901] type 01 class 0x060400
[    0.286495] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[    0.286909] pci 0000:00:02.0: [8086:3e96] type 00 class 0x030000
[    0.286916] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x90ffffff 64bit]
[    0.286922] pci 0000:00:02.0: reg 0x18: [mem 0xb0000000-0xbfffffff 64bit pref]
[    0.286925] pci 0000:00:02.0: reg 0x20: [io  0x4000-0x403f]
[    0.286939] pci 0000:00:02.0: BAR 2: assigned to efifb
[    0.286942] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.287136] pci 0000:00:08.0: [8086:1911] type 00 class 0x088000
[    0.287144] pci 0000:00:08.0: reg 0x10: [mem 0x91441000-0x91441fff 64bit]
[    0.287330] pci 0000:00:12.0: [8086:a379] type 00 class 0x118000
[    0.287373] pci 0000:00:12.0: reg 0x10: [mem 0x91440000-0x91440fff 64bit]
[    0.287677] pci 0000:00:14.0: [8086:a36d] type 00 class 0x0c0330
[    0.287718] pci 0000:00:14.0: reg 0x10: [mem 0x91420000-0x9142ffff 64bit]
[    0.287873] pci 0000:00:14.0: PME# supported from D3hot D3cold
[    0.289243] pci 0000:00:14.2: [8086:a36f] type 00 class 0x050000
[    0.289279] pci 0000:00:14.2: reg 0x10: [mem 0x91436000-0x91437fff 64bit]
[    0.289301] pci 0000:00:14.2: reg 0x18: [mem 0x9143f000-0x9143ffff 64bit]
[    0.289897] pci 0000:00:15.0: [8086:a368] type 00 class 0x0c8000
[    0.290511] pci 0000:00:15.0: reg 0x10: [mem 0x00000000-0x00000fff 64bit]
[    0.293469] pci 0000:00:15.1: [8086:a369] type 00 class 0x0c8000
[    0.294057] pci 0000:00:15.1: reg 0x10: [mem 0x00000000-0x00000fff 64bit]
[    0.296683] pci 0000:00:16.0: [8086:a360] type 00 class 0x078000
[    0.296720] pci 0000:00:16.0: reg 0x10: [mem 0x9143c000-0x9143cfff 64bit]
[    0.296862] pci 0000:00:16.0: PME# supported from D3hot
[    0.297343] pci 0000:00:17.0: [8086:a352] type 00 class 0x010601
[    0.297377] pci 0000:00:17.0: reg 0x10: [mem 0x91434000-0x91435fff]
[    0.297393] pci 0000:00:17.0: reg 0x14: [mem 0x9143b000-0x9143b0ff]
[    0.297409] pci 0000:00:17.0: reg 0x18: [io  0x4090-0x4097]
[    0.297426] pci 0000:00:17.0: reg 0x1c: [io  0x4080-0x4083]
[    0.297443] pci 0000:00:17.0: reg 0x20: [io  0x4060-0x407f]
[    0.297459] pci 0000:00:17.0: reg 0x24: [mem 0x9143a000-0x9143a7ff]
[    0.297563] pci 0000:00:17.0: PME# supported from D3hot
[    0.297939] pci 0000:00:1c.0: [8086:a338] type 01 class 0x060400
[    0.298163] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    0.298208] pci 0000:00:1c.0: PTM enabled (root), 4ns granularity
[    0.298791] pci 0000:00:1c.5: [8086:a33d] type 01 class 0x060400
[    0.299006] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
[    0.299051] pci 0000:00:1c.5: PTM enabled (root), 4ns granularity
[    0.299593] pci 0000:00:1c.7: [8086:a33f] type 01 class 0x060400
[    0.299811] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
[    0.299856] pci 0000:00:1c.7: PTM enabled (root), 4ns granularity
[    0.300387] pci 0000:00:1d.0: [8086:a330] type 01 class 0x060400
[    0.300605] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    0.300652] pci 0000:00:1d.0: PTM enabled (root), 4ns granularity
[    0.301650] pci 0000:00:1e.0: [8086:a328] type 00 class 0x078000
[    0.302264] pci 0000:00:1e.0: reg 0x10: [mem 0x00000000-0x00000fff 64bit]
[    0.304897] pci 0000:00:1f.0: [8086:a309] type 00 class 0x060100
[    0.305451] pci 0000:00:1f.3: [8086:a348] type 00 class 0x040300
[    0.305524] pci 0000:00:1f.3: reg 0x10: [mem 0x91430000-0x91433fff 64bit]
[    0.305626] pci 0000:00:1f.3: reg 0x20: [mem 0x91000000-0x910fffff 64bit]
[    0.305799] pci 0000:00:1f.3: PME# supported from D3hot D3cold
[    0.308420] pci 0000:00:1f.4: [8086:a323] type 00 class 0x0c0500
[    0.308602] pci 0000:00:1f.4: reg 0x10: [mem 0x91438000-0x914380ff 64bit]
[    0.308817] pci 0000:00:1f.4: reg 0x20: [io  0xefa0-0xefbf]
[    0.309253] pci 0000:00:1f.5: [8086:a324] type 00 class 0x0c8000
[    0.309284] pci 0000:00:1f.5: reg 0x10: [mem 0xfe010000-0xfe010fff]
[    0.309497] pci 0000:00:1f.6: [8086:15bb] type 00 class 0x020000
[    0.309550] pci 0000:00:1f.6: reg 0x10: [mem 0x91400000-0x9141ffff]
[    0.309823] pci 0000:00:1f.6: PME# supported from D0 D3hot D3cold
[    0.309967] pci 0000:01:00.0: [8086:1563] type 00 class 0x020000
[    0.309981] pci 0000:01:00.0: reg 0x10: [mem 0xc0400000-0xc07fffff 64bit pref]
[    0.310001] pci 0000:01:00.0: reg 0x20: [mem 0xc0804000-0xc0807fff 64bit pref]
[    0.310007] pci 0000:01:00.0: reg 0x30: [mem 0x91380000-0x913fffff pref]
[    0.310069] pci 0000:01:00.0: PME# supported from D0 D3hot
[    0.310089] pci 0000:01:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit]
[    0.310090] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit] (contains BAR0 for 64 VFs)
[    0.310103] pci 0000:01:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit]
[    0.310104] pci 0000:01:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit] (contains BAR3 for 64 VFs)
[    0.310312] pci 0000:01:00.1: [8086:1563] type 00 class 0x020000
[    0.310327] pci 0000:01:00.1: reg 0x10: [mem 0xc0000000-0xc03fffff 64bit pref]
[    0.310346] pci 0000:01:00.1: reg 0x20: [mem 0xc0800000-0xc0803fff 64bit pref]
[    0.310352] pci 0000:01:00.1: reg 0x30: [mem 0x91300000-0x9137ffff pref]
[    0.310411] pci 0000:01:00.1: PME# supported from D0 D3hot
[    0.310430] pci 0000:01:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit]
[    0.310432] pci 0000:01:00.1: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit] (contains BAR0 for 64 VFs)
[    0.310444] pci 0000:01:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit]
[    0.310445] pci 0000:01:00.1: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit] (contains BAR3 for 64 VFs)
[    0.310632] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.310635] pci 0000:00:01.0:   bridge window [mem 0x91300000-0x913fffff]
[    0.310638] pci 0000:00:01.0:   bridge window [mem 0xc0000000-0xc08fffff 64bit pref]
[    0.310639] pci 0000:00:01.0: bridge has subordinate 01 but max busn 02
[    0.310761] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.310963] pci 0000:03:00.0: working around ROM BAR overlap defect
[    0.310964] pci 0000:03:00.0: [8086:1533] type 00 class 0x020000
[    0.311014] pci 0000:03:00.0: reg 0x10: [mem 0x91200000-0x9127ffff]
[    0.311064] pci 0000:03:00.0: reg 0x18: [io  0x3000-0x301f]
[    0.311088] pci 0000:03:00.0: reg 0x1c: [mem 0x91280000-0x91283fff]
[    0.311362] pci 0000:03:00.0: PME# supported from D0 D3hot D3cold
[    0.311632] pci 0000:00:1c.5: PCI bridge to [bus 03]
[    0.311638] pci 0000:00:1c.5:   bridge window [io  0x3000-0x3fff]
[    0.311644] pci 0000:00:1c.5:   bridge window [mem 0x91200000-0x912fffff]
[    0.311820] pci 0000:04:00.0: [10e3:8113] type 01 class 0x060401
[    0.312086] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.312186] pci 0000:00:1c.7: PCI bridge to [bus 04-05]
[    0.312266] pci_bus 0000:05: extended config space not accessible
[    0.312401] pci 0000:04:00.0: PCI bridge to [bus 05] (subtractive decode)
[    0.312570] pci 0000:06:00.0: [144d:a808] type 00 class 0x010802
[    0.312612] pci 0000:06:00.0: reg 0x10: [mem 0x91110000-0x91113fff 64bit]
[    0.312691] pci 0000:06:00.0: reg 0x30: [mem 0x91100000-0x9110ffff pref]
[    0.313022] pci 0000:00:1d.0: PCI bridge to [bus 06]
[    0.313032] pci 0000:00:1d.0:   bridge window [mem 0x91100000-0x911fffff]
[    0.314764] ACPI: PCI: Interrupt link LNKA configured for IRQ 0
[    0.314835] ACPI: PCI: Interrupt link LNKB configured for IRQ 1
[    0.314905] ACPI: PCI: Interrupt link LNKC configured for IRQ 0
[    0.314975] ACPI: PCI: Interrupt link LNKD configured for IRQ 0
[    0.315045] ACPI: PCI: Interrupt link LNKE configured for IRQ 0
[    0.315113] ACPI: PCI: Interrupt link LNKF configured for IRQ 0
[    0.315182] ACPI: PCI: Interrupt link LNKG configured for IRQ 0
[    0.315251] ACPI: PCI: Interrupt link LNKH configured for IRQ 0
[    0.323860] iommu: Default domain type: Translated 
[    0.323860] iommu: DMA domain TLB invalidation policy: lazy mode 
[    0.323860] EDAC MC: Ver: 3.0.0
[    0.323860] Registered efivars operations
[    0.323860] NetLabel: Initializing
[    0.323860] NetLabel:  domain hash size = 128
[    0.323860] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.323860] NetLabel:  unlabeled traffic allowed by default
[    0.323860] PCI: Using ACPI for IRQ routing
[    0.409524] PCI: pci_cache_line_size set to 64 bytes
[    0.409904] e820: reserve RAM buffer [mem 0x0005f000-0x0005ffff]
[    0.409906] e820: reserve RAM buffer [mem 0x8104c000-0x83ffffff]
[    0.409907] e820: reserve RAM buffer [mem 0x82633000-0x83ffffff]
[    0.409908] e820: reserve RAM buffer [mem 0x8723f000-0x87ffffff]
[    0.409909] e820: reserve RAM buffer [mem 0x8afb6000-0x8bffffff]
[    0.409910] e820: reserve RAM buffer [mem 0x8bc10000-0x8bffffff]
[    0.409911] e820: reserve RAM buffer [mem 0x86c800000-0x86fffffff]
[    0.409923] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[    0.409923] pci 0000:00:02.0: vgaarb: bridge control possible
[    0.409923] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.409923] vgaarb: loaded
[    0.410237] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    0.410244] hpet0: 8 comparators, 64-bit 24.000000 MHz counter
[    0.412349] clocksource: Switched to clocksource tsc-early
[    0.412438] VFS: Disk quotas dquot_6.6.0
[    0.412449] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.412557] AppArmor: AppArmor Filesystem Enabled
[    0.412567] pnp: PnP ACPI init
[    0.412635] system 00:00: [mem 0x40000000-0x403fffff] could not be reserved
[    0.412635] system 00:02: [io  0x0a00-0x0a1f] has been reserved
[    0.412635] system 00:02: [io  0x0a20-0x0a2f] has been reserved
[    0.412635] system 00:02: [io  0x0a30-0x0a3f] has been reserved
[    0.412635] system 00:02: [io  0x0a40-0x0a4f] has been reserved
[    0.412635] pnp 00:03: [dma 0 disabled]
[    0.412635] system 00:04: [io  0x0680-0x069f] has been reserved
[    0.412635] system 00:04: [io  0x164e-0x164f] has been reserved
[    0.412635] system 00:05: [io  0x1854-0x1857] has been reserved
[    0.412635] system 00:06: [mem 0xfed10000-0xfed17fff] has been reserved
[    0.412635] system 00:06: [mem 0xfed18000-0xfed18fff] has been reserved
[    0.412635] system 00:06: [mem 0xfed19000-0xfed19fff] has been reserved
[    0.412635] system 00:06: [mem 0xe0000000-0xefffffff] has been reserved
[    0.412635] system 00:06: [mem 0xfed20000-0xfed3ffff] has been reserved
[    0.412635] system 00:06: [mem 0xfed90000-0xfed93fff] could not be reserved
[    0.412635] system 00:06: [mem 0xfed45000-0xfed8ffff] has been reserved
[    0.412635] system 00:06: [mem 0xfee00000-0xfeefffff] could not be reserved
[    0.412635] system 00:07: [io  0x1800-0x18fe] could not be reserved
[    0.412635] system 00:07: [mem 0xfd000000-0xfd69ffff] has been reserved
[    0.412635] system 00:07: [mem 0xfd6c0000-0xfd6cffff] has been reserved
[    0.412635] system 00:07: [mem 0xfd6f0000-0xfdffffff] has been reserved
[    0.412635] system 00:07: [mem 0xfe000000-0xfe01ffff] could not be reserved
[    0.412635] system 00:07: [mem 0xfe200000-0xfe7fffff] has been reserved
[    0.412635] system 00:07: [mem 0xff000000-0xffffffff] has been reserved
[    0.412635] system 00:08: [io  0x2000-0x20fe] has been reserved
[    0.412814] system 00:09: [mem 0xfd6e0000-0xfd6effff] has been reserved
[    0.412817] system 00:09: [mem 0xfd6d0000-0xfd6dffff] has been reserved
[    0.412818] system 00:09: [mem 0xfd6b0000-0xfd6bffff] has been reserved
[    0.412820] system 00:09: [mem 0xfd6a0000-0xfd6affff] has been reserved
[    0.413831] pnp: PnP ACPI: found 10 devices
[    0.419419] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.419466] NET: Registered PF_INET protocol family
[    0.419582] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.421870] tcp_listen_portaddr_hash hash table entries: 16384 (order: 6, 262144 bytes, linear)
[    0.421899] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.422000] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.422204] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.422263] TCP: Hash tables configured (established 262144 bind 65536)
[    0.422353] MPTCP token hash table entries: 32768 (order: 7, 786432 bytes, linear)
[    0.422433] UDP hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    0.422493] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    0.422557] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.422562] NET: Registered PF_XDP protocol family
[    0.422567] pci_bus 0000:00: max bus depth: 2 pci_try_num: 3
[    0.422578] clipped [mem size 0x00020000 64bit] to [mem size 0xfffffffffffc0000 64bit] for e820 entry [mem 0x000a0000-0x000fffff]
[    0.422581] pci 0000:00:15.0: BAR 0: assigned [mem 0x8f800000-0x8f800fff 64bit]
[    0.422855] clipped [mem size 0x00020000 64bit] to [mem size 0xfffffffffffc0000 64bit] for e820 entry [mem 0x000a0000-0x000fffff]
[    0.422858] pci 0000:00:15.1: BAR 0: assigned [mem 0x8f801000-0x8f801fff 64bit]
[    0.423138] clipped [mem size 0x00020000 64bit] to [mem size 0xfffffffffffc0000 64bit] for e820 entry [mem 0x000a0000-0x000fffff]
[    0.423140] pci 0000:00:1e.0: BAR 0: assigned [mem 0x8f802000-0x8f802fff 64bit]
[    0.423409] pci 0000:01:00.0: BAR 7: no space for [mem size 0x00100000 64bit]
[    0.423410] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit]
[    0.423412] pci 0000:01:00.0: BAR 10: no space for [mem size 0x00100000 64bit]
[    0.423413] pci 0000:01:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit]
[    0.423415] pci 0000:01:00.1: BAR 7: no space for [mem size 0x00100000 64bit]
[    0.423416] pci 0000:01:00.1: BAR 7: failed to assign [mem size 0x00100000 64bit]
[    0.423417] pci 0000:01:00.1: BAR 10: no space for [mem size 0x00100000 64bit]
[    0.423419] pci 0000:01:00.1: BAR 10: failed to assign [mem size 0x00100000 64bit]
[    0.423420] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.423423] pci 0000:00:01.0:   bridge window [mem 0x91300000-0x913fffff]
[    0.423425] pci 0000:00:01.0:   bridge window [mem 0xc0000000-0xc08fffff 64bit pref]
[    0.423429] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.423459] pci 0000:00:1c.5: PCI bridge to [bus 03]
[    0.423463] pci 0000:00:1c.5:   bridge window [io  0x3000-0x3fff]
[    0.423470] pci 0000:00:1c.5:   bridge window [mem 0x91200000-0x912fffff]
[    0.423485] pci 0000:04:00.0: PCI bridge to [bus 05]
[    0.423508] pci 0000:00:1c.7: PCI bridge to [bus 04-05]
[    0.423528] pci 0000:00:1d.0: PCI bridge to [bus 06]
[    0.423535] pci 0000:00:1d.0:   bridge window [mem 0x91100000-0x911fffff]
[    0.423549] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.423550] release child resource [mem 0x91300000-0x9137ffff pref]
[    0.423551] release child resource [mem 0x91380000-0x913fffff pref]
[    0.423552] pci 0000:00:01.0: resource 14 [mem 0x91300000-0x913fffff] released
[    0.423554] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.423558] clipped [mem size 0x00020000] to [mem size 0xfffffffffffc0000] for e820 entry [mem 0x000a0000-0x000fffff]
[    0.423560] pci 0000:00:01.0: BAR 14: assigned [mem 0x8f900000-0x8fdfffff]
[    0.423563] pci 0000:01:00.0: BAR 6: assigned [mem 0x8f900000-0x8f97ffff pref]
[    0.423564] pci 0000:01:00.1: BAR 6: assigned [mem 0x8f980000-0x8f9fffff pref]
[    0.423566] pci 0000:01:00.0: BAR 7: assigned [mem 0x8fa00000-0x8fafffff 64bit]
[    0.423573] pci 0000:01:00.0: BAR 10: assigned [mem 0x8fb00000-0x8fbfffff 64bit]
[    0.423576] pci 0000:01:00.1: BAR 7: assigned [mem 0x8fc00000-0x8fcfffff 64bit]
[    0.423580] pci 0000:01:00.1: BAR 10: assigned [mem 0x8fd00000-0x8fdfffff 64bit]
[    0.423583] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.423585] pci 0000:00:01.0:   bridge window [mem 0x8f900000-0x8fdfffff]
[    0.423586] pci 0000:00:01.0:   bridge window [mem 0xc0000000-0xc08fffff 64bit pref]
[    0.423590] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.423620] pci 0000:00:1c.5: PCI bridge to [bus 03]
[    0.423623] pci 0000:00:1c.5:   bridge window [io  0x3000-0x3fff]
[    0.423631] pci 0000:00:1c.5:   bridge window [mem 0x91200000-0x912fffff]
[    0.423646] pci 0000:04:00.0: PCI bridge to [bus 05]
[    0.423669] pci 0000:00:1c.7: PCI bridge to [bus 04-05]
[    0.423690] pci 0000:00:1d.0: PCI bridge to [bus 06]
[    0.423697] pci 0000:00:1d.0:   bridge window [mem 0x91100000-0x911fffff]
[    0.423712] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.423713] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.423715] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.423716] pci_bus 0000:00: resource 7 [mem 0x8f800000-0xdfffffff window]
[    0.423717] pci_bus 0000:00: resource 8 [mem 0xfc800000-0xfe7fffff window]
[    0.423719] pci_bus 0000:01: resource 1 [mem 0x8f900000-0x8fdfffff]
[    0.423720] pci_bus 0000:01: resource 2 [mem 0xc0000000-0xc08fffff 64bit pref]
[    0.423721] pci_bus 0000:03: resource 0 [io  0x3000-0x3fff]
[    0.423722] pci_bus 0000:03: resource 1 [mem 0x91200000-0x912fffff]
[    0.423724] pci_bus 0000:06: resource 1 [mem 0x91100000-0x911fffff]
[    0.424377] PCI: CLS 64 bytes, default 64
[    0.424389] DMAR: No ATSR found
[    0.424390] DMAR: No SATC found
[    0.424391] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.424392] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.424393] DMAR: IOMMU feature nwfs inconsistent
[    0.424394] DMAR: IOMMU feature pasid inconsistent
[    0.424394] DMAR: IOMMU feature eafs inconsistent
[    0.424395] DMAR: IOMMU feature prs inconsistent
[    0.424395] DMAR: IOMMU feature nest inconsistent
[    0.424396] DMAR: IOMMU feature mts inconsistent
[    0.424396] DMAR: IOMMU feature sc_support inconsistent
[    0.424397] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.424398] DMAR: dmar0: Using Queued invalidation
[    0.424401] DMAR: dmar1: Using Queued invalidation
[    0.424419] Trying to unpack rootfs image as initramfs...
[    0.424657] pci 0000:00:00.0: Adding to iommu group 0
[    0.424670] pci 0000:00:01.0: Adding to iommu group 1
[    0.424679] pci 0000:00:02.0: Adding to iommu group 2
[    0.424687] pci 0000:00:08.0: Adding to iommu group 3
[    0.424699] pci 0000:00:12.0: Adding to iommu group 4
[    0.424713] pci 0000:00:14.0: Adding to iommu group 5
[    0.424721] pci 0000:00:14.2: Adding to iommu group 5
[    0.424735] pci 0000:00:15.0: Adding to iommu group 6
[    0.424743] pci 0000:00:15.1: Adding to iommu group 6
[    0.424754] pci 0000:00:16.0: Adding to iommu group 7
[    0.424762] pci 0000:00:17.0: Adding to iommu group 8
[    0.424783] pci 0000:00:1c.0: Adding to iommu group 9
[    0.424800] pci 0000:00:1c.5: Adding to iommu group 10
[    0.424814] pci 0000:00:1c.7: Adding to iommu group 11
[    0.424831] pci 0000:00:1d.0: Adding to iommu group 12
[    0.424841] pci 0000:00:1e.0: Adding to iommu group 13
[    0.424866] pci 0000:00:1f.0: Adding to iommu group 14
[    0.424874] pci 0000:00:1f.3: Adding to iommu group 14
[    0.424884] pci 0000:00:1f.4: Adding to iommu group 14
[    0.424892] pci 0000:00:1f.5: Adding to iommu group 14
[    0.424901] pci 0000:00:1f.6: Adding to iommu group 14
[    0.424905] pci 0000:01:00.0: Adding to iommu group 1
[    0.424909] pci 0000:01:00.1: Adding to iommu group 1
[    0.424926] pci 0000:03:00.0: Adding to iommu group 15
[    0.424947] pci 0000:04:00.0: Adding to iommu group 16
[    0.424963] pci 0000:06:00.0: Adding to iommu group 17
[    0.426191] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    0.426193] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.426194] software IO TLB: mapped [mem 0x0000000078f50000-0x000000007cf50000] (64MB)
[    0.426257] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.426604] Initialise system trusted keyrings
[    0.426612] Key type blacklist registered
[    0.426652] workingset: timestamp_bits=36 max_order=23 bucket_order=0
[    0.427862] zbud: loaded
[    0.428003] integrity: Platform Keyring initialized
[    0.428004] Key type asymmetric registered
[    0.428005] Asymmetric key parser 'x509' registered
[    1.018028] Freeing initrd memory: 55228K
[    1.021426] alg: self-tests for CTR-KDF (hmac(sha256)) passed
[    1.021444] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    1.021472] io scheduler mq-deadline registered
[    1.022355] pcieport 0000:00:01.0: PME: Signaling with IRQ 122
[    1.022622] pcieport 0000:00:1c.0: PME: Signaling with IRQ 123
[    1.022960] pcieport 0000:00:1c.5: PME: Signaling with IRQ 124
[    1.023279] pcieport 0000:00:1c.7: PME: Signaling with IRQ 125
[    1.023582] pcieport 0000:00:1d.0: PME: Signaling with IRQ 126
[    1.023732] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.026175] thermal LNXTHERM:00: registered as thermal_zone0
[    1.026177] ACPI: thermal: Thermal Zone [TZ00] (28 C)
[    1.026243] ERST: Error Record Serialization Table (ERST) support is initialized.
[    1.026245] pstore: Registered erst as persistent store backend
[    1.026462] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    1.027259] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.029253] Linux agpgart interface v0.103
[    1.029319] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.
[    1.030222] i8042: PNP: No PS/2 controller found.
[    1.030335] mousedev: PS/2 mouse device common for all mice
[    1.030378] rtc_cmos rtc_cmos: RTC can wake from S4
[    1.031931] rtc_cmos rtc_cmos: registered as rtc0
[    1.032247] rtc_cmos rtc_cmos: setting system clock to 2023-01-04T05:16:41 UTC (1672809401)
[    1.032259] rtc_cmos rtc_cmos: alarms up to one month, y3k, 114 bytes nvram
[    1.032670] intel_pstate: Intel P-state driver initializing
[    1.033371] intel_pstate: Disabling energy efficiency optimization
[    1.033371] intel_pstate: HWP enabled
[    1.033430] ledtrig-cpu: registered to indicate activity on CPUs
[    1.033617] efifb: probing for efifb
[    1.033626] efifb: framebuffer at 0xb0000000, using 3072k, total 3072k
[    1.033627] efifb: mode is 1024x768x32, linelength=4096, pages=1
[    1.033628] efifb: scrolling: redraw
[    1.033629] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.033696] Console: switching to colour frame buffer device 128x48
[    1.034079] fb0: EFI VGA frame buffer device
[    1.041938] NET: Registered PF_INET6 protocol family
[    1.045372] Segment Routing with IPv6
[    1.045378] In-situ OAM (IOAM) with IPv6
[    1.045394] mip6: Mobile IPv6
[    1.045395] NET: Registered PF_PACKET protocol family
[    1.045507] mpls_gso: MPLS GSO support
[    1.046300] microcode: sig=0x906ea, pf=0x2, revision=0xf0
[    1.046587] microcode: Microcode Update Driver: v2.2.
[    1.046591] IPI shorthand broadcast: enabled
[    1.046606] sched_clock: Marking stable (1037632417, 8620494)->(1053985162, -7732251)
[    1.046996] registered taskstats version 1
[    1.047000] Loading compiled-in X.509 certificates
[    1.059497] Loaded X.509 cert 'Debian Secure Boot CA: 6ccece7e4c6c0d1f6149f3dd27dfcc5cbb419ea1'
[    1.059507] Loaded X.509 cert 'Debian Secure Boot Signer 2022 - linux: 14011249c2675ea8e5148542202005810584b25f'
[    1.060096] zswap: loaded using pool lzo/zbud
[    1.060528] Key type .fscrypt registered
[    1.060530] Key type fscrypt-provisioning registered
[    1.060798] pstore: Using crash dump compression: deflate
[    1.063751] Key type encrypted registered
[    1.063753] AppArmor: AppArmor sha1 policy hashing enabled
[    1.064201] integrity: Loading X.509 certificate: UEFI:db
[    1.064218] integrity: Loaded X.509 cert 'Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4'
[    1.064219] integrity: Loading X.509 certificate: UEFI:db
[    1.064232] integrity: Loaded X.509 cert 'Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53'
[    1.064643] ima: No TPM chip found, activating TPM-bypass!
[    1.064646] ima: Allocated hash algorithm: sha256
[    1.064652] ima: No architecture policies found
[    1.064664] evm: Initialising EVM extended attributes:
[    1.064665] evm: security.selinux
[    1.064665] evm: security.SMACK64 (disabled)
[    1.064666] evm: security.SMACK64EXEC (disabled)
[    1.064667] evm: security.SMACK64TRANSMUTE (disabled)
[    1.064667] evm: security.SMACK64MMAP (disabled)
[    1.064668] evm: security.apparmor
[    1.064668] evm: security.ima
[    1.064669] evm: security.capability
[    1.064669] evm: HMAC attrs: 0x1
[    1.128246] Freeing unused decrypted memory: 2036K
[    1.128634] Freeing unused kernel image (initmem) memory: 2724K
[    1.145796] Write protecting the kernel read-only data: 24576k
[    1.146499] Freeing unused kernel image (text/rodata gap) memory: 2040K
[    1.146857] Freeing unused kernel image (rodata/data gap) memory: 1664K
[    1.180411] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.180412] x86/mm: Checking user space page tables
[    1.212490] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.212493] Run /init as init process
[    1.212494]   with arguments:
[    1.212495]     /init
[    1.212496]   with environment:
[    1.212496]     HOME=/
[    1.212497]     TERM=linux
[    1.212497]     BOOT_IMAGE=/vmlinuz-6.0.0-0.deb11.6-amd64
[    1.212498]     ip=192.168.20.10::192.168.20.1:255.255.255.0::enp1s0f0
[    1.329852] acpi PNP0C14:01: duplicate WMI GUID 05901221-D566-11D1-B2F0-00A0C9062910 (first instance was on PNP0C14:00)
[    1.330749] pps_core: LinuxPPS API ver. 1 registered
[    1.330751] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.332444] PTP clock support registered
[    1.332846] intel-lpss 0000:00:15.0: enabling device (0000 -> 0002)
[    1.333109] idma64 idma64.0: Found Intel integrated DMA 64-bit
[    1.333367] dca service started, version 1.12.1
[    1.334313] i801_smbus 0000:00:1f.4: SPD Write Disable is set
[    1.334387] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt
[    1.338340] i2c i2c-0: 2/4 memory slots populated (from DMI)
[    1.338669] cryptd: max_cpu_qlen set to 1000
[    1.339326] i2c i2c-0: Successfully instantiated SPD at 0x51
[    1.340030] i2c i2c-0: Successfully instantiated SPD at 0x53
[    1.343822] intel-lpss 0000:00:15.1: enabling device (0000 -> 0002)
[    1.344097] idma64 idma64.1: Found Intel integrated DMA 64-bit
[    1.346029] ACPI: bus type USB registered
[    1.346050] usbcore: registered new interface driver usbfs
[    1.346058] usbcore: registered new interface driver hub
[    1.346070] usbcore: registered new device driver usb
[    1.347486] igb: Intel(R) Gigabit Ethernet Network Driver
[    1.347487] igb: Copyright (c) 2007-2014 Intel Corporation.
[    1.349007] e1000e: Intel(R) PRO/1000 Network Driver
[    1.349009] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.349692] SCSI subsystem initialized
[    1.350118] AVX2 version of gcm_enc/dec engaged.
[    1.350150] AES CTR mode by8 optimization enabled
[    1.350479] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    1.355442] intel-lpss 0000:00:1e.0: enabling device (0000 -> 0002)
[    1.355800] idma64 idma64.2: Found Intel integrated DMA 64-bit
[    1.356029] dw-apb-uart.2: ttyS1 at MMIO 0x8f802000 (irq = 20, base_baud = 7500000) is a 16550A
[    1.359637] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.359643] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[    1.360892] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x110 quirks 0x0000000000009810
[    1.361174] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.361177] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[    1.361179] xhci_hcd 0000:00:14.0: Host supports USB 3.1 Enhanced SuperSpeed
[    1.361225] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.00
[    1.361228] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.361229] usb usb1: Product: xHCI Host Controller
[    1.361230] usb usb1: Manufacturer: Linux 6.0.0-0.deb11.6-amd64 xhci-hcd
[    1.361232] usb usb1: SerialNumber: 0000:00:14.0
[    1.361374] hub 1-0:1.0: USB hub found
[    1.361398] hub 1-0:1.0: 16 ports detected
[    1.363257] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.00
[    1.363259] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.363260] usb usb2: Product: xHCI Host Controller
[    1.363261] usb usb2: Manufacturer: Linux 6.0.0-0.deb11.6-amd64 xhci-hcd
[    1.363262] usb usb2: SerialNumber: 0000:00:14.0
[    1.363758] hub 2-0:1.0: USB hub found
[    1.363780] hub 2-0:1.0: 10 ports detected
[    1.364796] usb: port power management may be unreliable
[    1.367620] libata version 3.00 loaded.
[    1.371092] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    1.371094] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    1.373492] ahci 0000:00:17.0: version 3.0
[    1.373597] nvme 0000:06:00.0: platform quirk: setting simple suspend
[    1.373694] nvme nvme0: pci function 0000:06:00.0
[    1.375028] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 8 ports 6 Gbps 0xff impl SATA mode
[    1.375031] ahci 0000:00:17.0: flags: 64bit ncq sntf pm clo only pio slum part ems deso sadm sds apst 
[    1.381411] pps pps0: new PPS source ptp0
[    1.381441] igb 0000:03:00.0: added PHC on eth0
[    1.381442] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
[    1.381443] igb 0000:03:00.0: eth0: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:ad:b9:1d
[    1.381658] igb 0000:03:00.0: eth0: PBA No: 010B00-000
[    1.381660] igb 0000:03:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    1.381796] nvme nvme0: Shutdown timeout set to 8 seconds
[    1.393496] nvme nvme0: 12/0/0 default/read/poll queues
[    1.398507]  nvme0n1: p1 p2 p3
[    1.404627] igb 0000:03:00.0 eno2: renamed from eth0
[    1.431151] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    1.433631] tsc: Refined TSC clocksource calibration: 3504.053 MHz
[    1.433639] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x3282455eb3b, max_idle_ns: 440795213309 ns
[    1.433698] clocksource: Switched to clocksource tsc
[    1.475751] scsi host0: ahci
[    1.476391] scsi host1: ahci
[    1.476959] scsi host2: ahci
[    1.477430] scsi host3: ahci
[    1.477881] scsi host4: ahci
[    1.478277] scsi host5: ahci
[    1.478682] scsi host6: ahci
[    1.479088] scsi host7: ahci
[    1.479240] ata1: SATA max UDMA/133 abar m2048@0x9143a000 port 0x9143a100 irq 135
[    1.479249] ata2: SATA max UDMA/133 abar m2048@0x9143a000 port 0x9143a180 irq 135
[    1.479257] ata3: SATA max UDMA/133 abar m2048@0x9143a000 port 0x9143a200 irq 135
[    1.479262] ata4: SATA max UDMA/133 abar m2048@0x9143a000 port 0x9143a280 irq 135
[    1.479267] ata5: SATA max UDMA/133 abar m2048@0x9143a000 port 0x9143a300 irq 135
[    1.479271] ata6: SATA max UDMA/133 abar m2048@0x9143a000 port 0x9143a380 irq 135
[    1.479275] ata7: SATA max UDMA/133 abar m2048@0x9143a000 port 0x9143a400 irq 135
[    1.479279] ata8: SATA max UDMA/133 abar m2048@0x9143a000 port 0x9143a480 irq 135
[    1.501839] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) ac:1f:6b:ad:b9:1c
[    1.501849] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    1.502024] e1000e 0000:00:1f.6 eth0: MAC: 13, PHY: 12, PBA No: 010BFF-0FF
[    1.793025] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.793105] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.793183] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.793257] ata8: SATA link down (SStatus 0 SControl 300)
[    1.793328] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.793406] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.793997] ata6.00: ATA-11: ST18000NM003D-3DL103, SN03, max UDMA/133
[    1.794057] ata5.00: ATA-11: ST18000NM003D-3DL103, SN03, max UDMA/133
[    1.794071] ata3.00: ATA-8: WDC WD4001FFSX-68JNUN0, 81.00A81, max UDMA/133
[    1.794496] ata3.00: 7814037168 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.795515] ata1.00: ATA-10: ST12000NM0007-2A1101, SN02, max UDMA/133
[    1.795555] ata2.00: ATA-10: ST12000NM0007-2A1101, SN02, max UDMA/133
[    1.796241] ata3.00: configured for UDMA/133
[    1.796257] ata1.00: 23437770752 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.796268] ata1.00: Features: NCQ-sndrcv
[    1.796288] ata2.00: 23437770752 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.796299] ata2.00: Features: NCQ-sndrcv
[    1.797532] ata7: SATA link down (SStatus 0 SControl 300)
[    1.797586] ata4: SATA link down (SStatus 0 SControl 300)
[    1.799745] ata1.00: configured for UDMA/133
[    1.799776] ata2.00: configured for UDMA/133
[    1.800122] scsi 0:0:0:0: Direct-Access     ATA      ST12000NM0007-2A SN02 PQ: 0 ANSI: 5
[    1.801778] scsi 1:0:0:0: Direct-Access     ATA      ST12000NM0007-2A SN02 PQ: 0 ANSI: 5
[    1.802610] scsi 2:0:0:0: Direct-Access     ATA      WDC WD4001FFSX-6 0A81 PQ: 0 ANSI: 5
[    1.845966] ata6.00: 4394582016 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.846577] ata5.00: 4394582016 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.846630] ata6.00: Features: NCQ-sndrcv
[    1.847169] ata5.00: Features: NCQ-sndrcv
[    1.880049] ata6.00: configured for UDMA/133
[    1.880544] ata5.00: configured for UDMA/133
[    1.880951] scsi 4:0:0:0: Direct-Access     ATA      ST18000NM003D-3D SN03 PQ: 0 ANSI: 5
[    1.882611] scsi 5:0:0:0: Direct-Access     ATA      ST18000NM003D-3D SN03 PQ: 0 ANSI: 5
[    1.892002] e1000e 0000:00:1f.6 eno1: renamed from eth0
[    1.898354] sd 0:0:0:0: [sda] 23437770752 512-byte logical blocks: (12.0 TB/10.9 TiB)
[    1.898356] sd 0:0:0:0: [sda] 4096-byte physical blocks
[    1.898376] sd 0:0:0:0: [sda] Write Protect is off
[    1.898378] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    1.898397] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.898418] sd 0:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
[    1.898439] sd 2:0:0:0: [sdc] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[    1.898442] sd 2:0:0:0: [sdc] 4096-byte physical blocks
[    1.898445] sd 5:0:0:0: [sdd] 4394582016 4096-byte logical blocks: (18.0 TB/16.4 TiB)
[    1.898445] sd 4:0:0:0: [sde] 4394582016 4096-byte logical blocks: (18.0 TB/16.4 TiB)
[    1.898453] sd 2:0:0:0: [sdc] Write Protect is off
[    1.898453] sd 1:0:0:0: [sdb] 23437770752 512-byte logical blocks: (12.0 TB/10.9 TiB)
[    1.898454] sd 1:0:0:0: [sdb] 4096-byte physical blocks
[    1.898454] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    1.898459] sd 4:0:0:0: [sde] Write Protect is off
[    1.898462] sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00
[    1.898463] sd 5:0:0:0: [sdd] Write Protect is off
[    1.898464] sd 1:0:0:0: [sdb] Write Protect is off
[    1.898464] sd 5:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[    1.898466] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    1.898469] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.898480] sd 5:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.898482] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.898483] sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.898496] sd 2:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
[    1.898503] sd 5:0:0:0: [sdd] Preferred minimum I/O size 4096 bytes
[    1.898504] sd 1:0:0:0: [sdb] Preferred minimum I/O size 4096 bytes
[    1.898506] sd 4:0:0:0: [sde] Preferred minimum I/O size 4096 bytes
[    1.990929]  sda: sda1 sda2 sda3 sda4
[    1.991716] sd 0:0:0:0: [sda] Attached SCSI disk
[    1.994792]  sdc: sdc1
[    1.995034] sd 2:0:0:0: [sdc] Attached SCSI disk
[    1.997257]  sde: sde1
[    1.997594] sd 4:0:0:0: [sde] Attached SCSI disk
[    2.002725]  sdd: sdd1
[    2.003010] sd 5:0:0:0: [sdd] Attached SCSI disk
[    2.006221]  sdb: sdb1 sdb2 sdb3 sdb4
[    2.007117] sd 1:0:0:0: [sdb] Attached SCSI disk
[    2.070117] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[    2.167698] ixgbe 0000:01:00.0: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
[    2.279657] ixgbe 0000:01:00.0: MAC: 4, PHY: 0, PBA No: H86377-006
[    2.279668] ixgbe 0000:01:00.0: b4:96:91:68:1a:04
[    2.443768] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[    3.142723] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[    3.242189] ixgbe 0000:01:00.1: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
[    3.354024] ixgbe 0000:01:00.1: MAC: 4, PHY: 0, PBA No: H86377-006
[    3.354036] ixgbe 0000:01:00.1: b4:96:91:68:1a:05
[    3.518339] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[    3.521567] ixgbe 0000:01:00.0 enp1s0f0: renamed from eth0
[    3.561930] ixgbe 0000:01:00.1 enp1s0f1: renamed from eth1
[    3.753030] pps pps1: new PPS source ptp2
[    3.753246] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0
[    3.863722] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
[    3.863742] device-mapper: uevent: version 1.0.3
[    3.863850] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: dm-devel@redhat.com
[    9.067623] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: None
[    9.067841] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0f0: link becomes ready
[   16.501484] raid6: avx2x4   gen() 46191 MB/s
[   16.569483] raid6: avx2x2   gen() 48553 MB/s
[   16.637484] raid6: avx2x1   gen() 41811 MB/s
[   16.637484] raid6: using algorithm avx2x2 gen() 48553 MB/s
[   16.705483] raid6: .... xor() 26228 MB/s, rmw enabled
[   16.705484] raid6: using avx2x2 recovery algorithm
[   16.706322] xor: automatically using best checksumming function   avx       
[   16.781734] Btrfs loaded, crc32c=crc32c-intel, zoned=yes, fsverity=yes
[   16.829413] BTRFS: device fsid 02659301-429c-4bfc-8e5f-a953704522f5 devid 1 transid 5778238 /dev/mapper/vg_arktos_ssd-lv_vm scanned by btrfs (971)
[   16.829537] BTRFS: device fsid adb099cc-f2f6-46e4-897c-189bfb00871f devid 1 transid 5805 /dev/mapper/vg_arktos_ssd-lv_database scanned by btrfs (971)
[   16.829632] BTRFS: device fsid ac4ee7e7-9d32-4b96-9a30-af184c88c449 devid 1 transid 9446 /dev/mapper/vg_arktos_ssd-lv_home scanned by btrfs (971)
[   16.829778] BTRFS: device fsid bb058a62-c1e2-469d-bfb3-711cf50bcf30 devid 1 transid 796432 /dev/mapper/vg_arktos_ssd-lv_system scanned by btrfs (971)
[   16.838458] PM: Image not found (code -22)
[   16.846621] BTRFS info (device dm-2): using crc32c (crc32c-intel) checksum algorithm
[   16.846631] BTRFS info (device dm-2): disk space caching is enabled
[   16.856488] BTRFS info (device dm-2): enabling ssd optimizations
[   16.884074] ixgbe 0000:01:00.0: removed PHC on enp1s0f0
[   17.283789] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[   17.345826] systemd[1]: Inserted module 'autofs4'
[   17.387810] systemd[1]: systemd 247.3-7+deb11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
[   17.387916] systemd[1]: Detected architecture x86-64.
[   17.388269] systemd[1]: Set hostname to <arktos>.
[   17.581132] systemd[1]: Queued start job for default target Graphical Interface.
[   17.598129] systemd[1]: Created slice Virtual Machine and Container Slice.
[   17.598498] systemd[1]: Created slice system-getty.slice.
[   17.598673] systemd[1]: Created slice system-modprobe.slice.
[   17.598835] systemd[1]: Created slice system-postfix.slice.
[   17.599015] systemd[1]: Created slice system-postgresql.slice.
[   17.599173] systemd[1]: Created slice Cryptsetup Units Slice.
[   17.599331] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[   17.599489] systemd[1]: Created slice User and Session Slice.
[   17.599546] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[   17.599589] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[   17.599712] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[   17.599824] systemd[1]: Reached target Paths.
[   17.599840] systemd[1]: Reached target Remote File Systems.
[   17.599848] systemd[1]: Reached target Slices.
[   17.599869] systemd[1]: Reached target Libvirt guests shutdown.
[   17.599923] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[   17.600013] systemd[1]: Listening on LVM2 poll daemon socket.
[   17.600089] systemd[1]: Listening on Syslog Socket.
[   17.600160] systemd[1]: Listening on fsck to fsckd communication Socket.
[   17.600199] systemd[1]: Listening on initctl Compatibility Named Pipe.
[   17.600322] systemd[1]: Listening on Journal Audit Socket.
[   17.600397] systemd[1]: Listening on Journal Socket (/dev/log).
[   17.600484] systemd[1]: Listening on Journal Socket.
[   17.600807] systemd[1]: Listening on udev Control Socket.
[   17.600885] systemd[1]: Listening on udev Kernel Socket.
[   17.601434] systemd[1]: Mounting Huge Pages File System...
[   17.601992] systemd[1]: Mounting POSIX Message Queue File System...
[   17.602573] systemd[1]: Mounting Kernel Debug File System...
[   17.603217] systemd[1]: Mounting Kernel Trace File System...
[   17.603346] systemd[1]: Finished Availability of block devices.
[   17.604065] systemd[1]: Starting Set the console keyboard layout...
[   17.604629] systemd[1]: Starting Create list of static device nodes for the current kernel...
[   17.605188] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[   17.605807] systemd[1]: Starting Load Kernel Module configfs...
[   17.606445] systemd[1]: Starting Load Kernel Module drm...
[   17.607106] systemd[1]: Starting Load Kernel Module fuse...
[   17.607422] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[   17.608625] systemd[1]: Starting Journal Service...
[   17.609447] systemd[1]: Starting Load Kernel Modules...
[   17.610117] systemd[1]: Starting Remount Root and Kernel File Systems...
[   17.611033] systemd[1]: Starting Coldplug All udev Devices...
[   17.612785] systemd[1]: Mounted Huge Pages File System.
[   17.612885] systemd[1]: Mounted POSIX Message Queue File System.
[   17.612971] systemd[1]: Mounted Kernel Debug File System.
[   17.613059] systemd[1]: Mounted Kernel Trace File System.
[   17.613381] systemd[1]: Finished Set the console keyboard layout.
[   17.613754] systemd[1]: Finished Create list of static device nodes for the current kernel.
[   17.613999] systemd[1]: modprobe@configfs.service: Succeeded.
[   17.614254] systemd[1]: Finished Load Kernel Module configfs.
[   17.615155] systemd[1]: Mounting Kernel Configuration File System...
[   17.616135] fuse: init (API version 7.36)
[   17.616813] systemd[1]: modprobe@fuse.service: Succeeded.
[   17.617073] systemd[1]: Finished Load Kernel Module fuse.
[   17.617704] systemd[1]: Finished Remount Root and Kernel File Systems.
[   17.617874] systemd[1]: Mounted Kernel Configuration File System.
[   17.618456] systemd[1]: Mounting FUSE Control File System...
[   17.618900] systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
[   17.618951] systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
[   17.619458] systemd[1]: Starting Load/Save Random Seed...
[   17.620098] systemd[1]: Starting Create System Users...
[   17.620872] systemd[1]: Mounted FUSE Control File System.
[   17.626005] systemd[1]: Finished Load/Save Random Seed.
[   17.626122] systemd[1]: Condition check resulted in First Boot Complete being skipped.
[   17.628647] systemd[1]: Finished Create System Users.
[   17.629249] systemd[1]: Starting Create Static Device Nodes in /dev...
[   17.636060] ACPI: bus type drm_connector registered
[   17.637601] systemd[1]: modprobe@drm.service: Succeeded.
[   17.637806] systemd[1]: Finished Load Kernel Module drm.
[   17.638100] systemd[1]: Finished Create Static Device Nodes in /dev.
[   17.639146] systemd[1]: Starting Rule-based Manager for Device Events and Files...
[   17.641067] nct6775: Found NCT6796D or compatible chip at 0x2e:0xa30
[   17.646316] systemd[1]: Finished Load Kernel Modules.
[   17.646970] systemd[1]: Starting Apply Kernel Variables...
[   17.653831] systemd[1]: Finished Apply Kernel Variables.
[   17.662618] systemd[1]: Started Rule-based Manager for Device Events and Files.
[   17.681861] systemd[1]: Started Journal Service.
[   17.686226] systemd-journald[1086]: Received client request to flush runtime journal.
[   17.699757] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0
[   17.699798] ACPI: button: Sleep Button [SLPB]
[   17.699848] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[   17.699908] ACPI: button: Power Button [PWRB]
[   17.701410] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[   17.701834] intel_pmc_core INT33A1:00:  initialized
[   17.720203] ACPI: button: Power Button [PWRF]
[   17.728916] EDAC MC0: Giving out device to module ie31200_edac controller IE31200: DEV 0000:00:00.0 (POLLED)
[   17.729156] intel_pch_thermal 0000:00:12.0: enabling device (0000 -> 0002)
[   17.734557] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   17.734601] sd 1:0:0:0: Attached scsi generic sg1 type 0
[   17.734647] sd 2:0:0:0: Attached scsi generic sg2 type 0
[   17.734682] sd 4:0:0:0: Attached scsi generic sg3 type 0
[   17.734711] sd 5:0:0:0: Attached scsi generic sg4 type 0
[   17.741093] mei_me 0000:00:16.0: enabling device (0000 -> 0002)
[   17.748543] ee1004 0-0051: 512 byte EE1004-compliant SPD EEPROM, read-only
[   17.748566] ee1004 0-0053: 512 byte EE1004-compliant SPD EEPROM, read-only
[   17.778021] input: PC Speaker as /devices/platform/pcspkr/input/input3
[   17.778088] pstore: ignoring unexpected backend 'efi'
[   17.780868] iTCO_vendor_support: vendor-support=0
[   17.785957] iTCO_wdt iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
[   17.789848] RAPL PMU: API unit is 2^-32 Joules, 4 fixed counters, 655360 ms ovfl timer
[   17.789850] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[   17.789851] RAPL PMU: hw unit of domain package 2^-14 Joules
[   17.789852] RAPL PMU: hw unit of domain dram 2^-14 Joules
[   17.789852] RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
[   17.828341] BTRFS info: devid 1 device path /dev/mapper/vg_arktos_ssd-lv_system changed to /dev/dm-2 scanned by systemd-udevd (1137)
[   17.829144] BTRFS info: devid 1 device path /dev/dm-2 changed to /dev/mapper/vg_arktos_ssd-lv_system scanned by systemd-udevd (1137)
[   17.884536] BTRFS info (device dm-3): using crc32c (crc32c-intel) checksum algorithm
[   17.884542] BTRFS info (device dm-3): disk space caching is enabled
[   17.885836] Adding 16777212k swap on /dev/mapper/vg_arktos_ssd-lv_swap.  Priority:-2 extents:1 across:16777212k SSFS
[   17.885964] BTRFS info (device dm-4): using crc32c (crc32c-intel) checksum algorithm
[   17.885969] BTRFS info (device dm-4): setting nodatacow, compression disabled
[   17.885972] BTRFS info (device dm-4): disk space caching is enabled
[   17.886951] BTRFS info (device dm-5): using crc32c (crc32c-intel) checksum algorithm
[   17.886955] BTRFS info (device dm-5): setting nodatacow, compression disabled
[   17.886957] BTRFS info (device dm-5): disk space caching is enabled
[   17.888080] BTRFS info (device dm-4): enabling ssd optimizations
[   17.891620] BTRFS info (device dm-3): enabling ssd optimizations
[   17.914097] BTRFS info (device dm-5): enabling ssd optimizations
[   18.119359] i915 0000:00:02.0: [drm] VT-d active for gfx access
[   18.119439] Console: switching to colour dummy device 80x25
[   18.119481] i915 0000:00:02.0: vgaarb: deactivate vga console
[   18.119530] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[   18.120964] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[   18.121805] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[   18.135272] i915 0000:00:02.0: firmware: direct-loading firmware i915/kbl_dmc_ver1_04.bin
[   18.135531] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/kbl_dmc_ver1_04.bin (v1.4)
[   18.295273] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Quota mode: none.
[   19.135219] intel_rapl_common: Found RAPL domain package
[   19.135223] intel_rapl_common: Found RAPL domain core
[   19.135224] intel_rapl_common: Found RAPL domain uncore
[   19.135225] intel_rapl_common: Found RAPL domain dram
[   19.236582] i915 0000:00:02.0: [drm] failed to retrieve link info, disabling eDP
[   19.246235] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[   19.248530] ACPI: video: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
[   19.248857] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
[   19.249034] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[   19.276863] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   19.306166] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   19.333633] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   19.407304] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC888-VD: line_outs=3 (0x14/0x17/0x16/0x0/0x0) type:line
[   19.407309] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[   19.407310] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[   19.407311] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[   19.407312] snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x1e/0x0
[   19.407313] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[   19.407314] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[   19.407315] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[   19.407316] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[   19.500677] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input5
[   19.500715] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input6
[   19.500752] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1f.3/sound/card0/input7
[   19.500782] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1f.3/sound/card0/input8
[   19.500813] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[   19.500845] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[   19.500875] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[   19.500905] input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12
[   19.500951] input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13
[   19.501612] input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input14
[   19.502735] input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input15
[   19.507101] input: HDA Intel PCH HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input16
[   24.063223] BTRFS: device fsid d246fd05-53d2-4554-b0e7-a6ba1d05d1c3 devid 1 transid 323 /dev/dm-10 scanned by systemd-udevd (1122)
[   24.127870] BTRFS: device fsid 17d3ee39-a3cc-4dc6-a104-1c2ea1cfc0f5 devid 1 transid 626 /dev/dm-12 scanned by systemd-udevd (1136)
[   24.147158] BTRFS: device fsid 1e1b12e9-add1-49b7-a9b5-453af72bfb36 devid 1 transid 538 /dev/dm-11 scanned by systemd-udevd (1116)
[   24.185259] SGI XFS with ACLs, security attributes, realtime, quota, no debug enabled
[   24.186979] XFS (dm-9): Mounting V5 Filesystem
[   24.810578] XFS (dm-9): Ending clean mount
[   24.822559] xfs filesystem being mounted at /mnt/public supports timestamps until 2038 (0x7fffffff)
[   25.778301] BTRFS: device fsid 84bba855-578d-48db-80eb-49f1029c7543 devid 2 transid 3135 /dev/dm-13 scanned by systemd-udevd (1136)
[   27.615087] BTRFS: device fsid 84bba855-578d-48db-80eb-49f1029c7543 devid 1 transid 3135 /dev/dm-14 scanned by systemd-udevd (1136)
[   27.686987] BTRFS info (device dm-14): using sha256 (sha256-avx2) checksum algorithm
[   27.687015] BTRFS info (device dm-14): force zstd compression, level 3
[   27.687016] BTRFS info (device dm-14): using free space tree
[   31.646609] BTRFS: device fsid a6617e50-06aa-41de-b74a-4e5156ad750d devid 1 transid 1184 /dev/dm-20 scanned by systemd-udevd (1136)
[   31.665332] BTRFS: device fsid 133fc4c7-b17d-44ab-aa46-1543f2652c47 devid 1 transid 87 /dev/dm-19 scanned by systemd-udevd (1122)
[   31.686539] BTRFS: device fsid cecb56e6-e9e4-41e8-9cc2-5c6f9f65e5d8 devid 1 transid 718 /dev/dm-17 scanned by systemd-udevd (1140)
[   31.717261] BTRFS info (device dm-20): using sha256 (sha256-avx2) checksum algorithm
[   31.717284] BTRFS info (device dm-20): force zstd compression, level 3
[   31.717287] BTRFS info (device dm-20): using free space tree
[   31.719932] BTRFS info (device dm-19): using sha256 (sha256-avx2) checksum algorithm
[   31.719953] BTRFS info (device dm-19): force zstd compression, level 3
[   31.719955] BTRFS info (device dm-19): using free space tree
[   31.839258] BTRFS: device fsid ccc55254-81f9-4b90-bf32-20efc2a56975 devid 1 transid 416 /dev/dm-21 scanned by systemd-udevd (1142)
[   31.844937] BTRFS: device fsid cecb56e6-e9e4-41e8-9cc2-5c6f9f65e5d8 devid 2 transid 718 /dev/dm-18 scanned by systemd-udevd (1116)
[   38.935959] BTRFS info (device dm-10): using crc32c (crc32c-intel) checksum algorithm
[   38.935966] BTRFS info (device dm-10): force zstd compression, level 3
[   38.935968] BTRFS info (device dm-10): disk space caching is enabled
[   38.955717] BTRFS info (device dm-12): using crc32c (crc32c-intel) checksum algorithm
[   38.955726] BTRFS info (device dm-12): force zstd compression, level 3
[   38.955729] BTRFS info (device dm-12): disk space caching is enabled
[   38.956260] BTRFS info (device dm-11): using crc32c (crc32c-intel) checksum algorithm
[   38.956269] BTRFS info (device dm-11): force zstd compression, level 3
[   38.956273] BTRFS info (device dm-11): disk space caching is enabled
[   46.579269] audit: type=1400 audit(1672809447.041:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=2539 comm="apparmor_parser"
[   46.579670] audit: type=1400 audit(1672809447.041:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=2540 comm="apparmor_parser"
[   46.579674] audit: type=1400 audit(1672809447.041:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=2540 comm="apparmor_parser"
[   46.579919] audit: type=1400 audit(1672809447.041:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=2541 comm="apparmor_parser"
[   46.579923] audit: type=1400 audit(1672809447.041:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=2541 comm="apparmor_parser"
[   46.579925] audit: type=1400 audit(1672809447.041:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=2541 comm="apparmor_parser"
[   46.580049] audit: type=1400 audit(1672809447.041:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="virt-aa-helper" pid=2542 comm="apparmor_parser"
[   46.581061] audit: type=1400 audit(1672809447.041:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd" pid=2543 comm="apparmor_parser"
[   46.581065] audit: type=1400 audit(1672809447.041:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd//qemu_bridge_helper" pid=2543 comm="apparmor_parser"
[   46.618144] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   47.622990] br0: port 1(enp1s0f0) entered blocking state
[   47.622993] br0: port 1(enp1s0f0) entered disabled state
[   47.623031] device enp1s0f0 entered promiscuous mode
[   47.689031] pps pps1: new PPS source ptp2
[   47.689146] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0
[   52.315968] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: None
[   52.316199] br0: port 1(enp1s0f0) entered blocking state
[   52.316208] br0: port 1(enp1s0f0) entered forwarding state
[   52.316290] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
[   52.724012] audit: type=1400 audit(1672809453.185:11): apparmor="DENIED" operation="capable" profile="libvirtd" pid=2960 comm="daemon-init" capability=17  capname="sys_rawio"
[   52.725434] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
[   52.725967] audit: type=1400 audit(1672809453.189:12): apparmor="DENIED" operation="capable" profile="libvirtd" pid=2960 comm="daemon-init" capability=17  capname="sys_rawio"
[   53.406787] audit: type=1400 audit(1672809453.869:13): apparmor="DENIED" operation="capable" profile="libvirtd" pid=2960 comm="daemon-init" capability=17  capname="sys_rawio"
[   53.588711] audit: type=1400 audit(1672809454.049:14): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-ff9afa92-5401-4390-829b-66731dac4598" pid=3320 comm="apparmor_parser"
[   53.674260] audit: type=1400 audit(1672809454.137:15): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-ff9afa92-5401-4390-829b-66731dac4598" pid=3323 comm="apparmor_parser"
[   53.762945] audit: type=1400 audit(1672809454.225:16): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-ff9afa92-5401-4390-829b-66731dac4598" pid=3328 comm="apparmor_parser"
[   53.851366] audit: type=1400 audit(1672809454.313:17): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-ff9afa92-5401-4390-829b-66731dac4598" pid=3339 comm="apparmor_parser"
[   53.901666] tun: Universal TUN/TAP device driver, 1.6
[   53.902300] br0: port 2(vnet0) entered blocking state
[   53.902303] br0: port 2(vnet0) entered disabled state
[   53.902337] device vnet0 entered promiscuous mode
[   53.902415] br0: port 2(vnet0) entered blocking state
[   53.902416] br0: port 2(vnet0) entered forwarding state
[   54.008177] audit: type=1400 audit(1672809454.469:18): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-ff9afa92-5401-4390-829b-66731dac4598" pid=3356 comm="apparmor_parser"
[   54.131317] audit: type=1400 audit(1672809454.593:19): apparmor="DENIED" operation="capable" profile="libvirtd" pid=2960 comm="rpc-worker" capability=39  capname="bpf"
[   54.133581] audit: type=1400 audit(1672809454.597:20): apparmor="DENIED" operation="capable" profile="libvirtd" pid=2960 comm="rpc-worker" capability=38  capname="perfmon"
[   54.629885] br0: port 2(vnet0) entered disabled state
[   54.630257] device vnet0 left promiscuous mode
[   54.630266] br0: port 2(vnet0) entered disabled state
[   55.405187] br0: port 2(vnet1) entered blocking state
[   55.405190] br0: port 2(vnet1) entered disabled state
[   55.405221] device vnet1 entered promiscuous mode
[   55.405288] br0: port 2(vnet1) entered blocking state
[   55.405290] br0: port 2(vnet1) entered forwarding state
[   56.169778] br0: port 2(vnet1) entered disabled state
[   56.169833] device vnet1 left promiscuous mode
[   56.169835] br0: port 2(vnet1) entered disabled state
[  263.298649] BTRFS info (device dm-17): using sha256 (sha256-avx2) checksum algorithm
[  263.298706] BTRFS info (device dm-17): force zstd compression, level 3
[  263.298713] BTRFS info (device dm-17): using free space tree
[  328.564776] BTRFS: device fsid ebb94498-644c-42cd-892f-37886173523c devid 1 transid 6 /dev/dm-17 scanned by systemd-udevd (9222)
[  328.566690] BTRFS: device fsid ebb94498-644c-42cd-892f-37886173523c devid 2 transid 6 /dev/vg_arktos_hdd_b/lv_randy_2 scanned by mkfs.btrfs (9177)
[  398.059965] BTRFS info (device dm-17): using sha256 (sha256-avx2) checksum algorithm
[  398.060021] BTRFS info (device dm-17): setting incompat feature flag for COMPRESS_ZSTD (0x10)
[  398.060028] BTRFS info (device dm-17): force zstd compression, level 3
[  398.060034] BTRFS info (device dm-17): using free space tree
[  398.075924] BTRFS info (device dm-17): cleaning free space cache v1
[  398.147260] BTRFS info (device dm-17): checking UUID tree
[ 9288.176184] perf: interrupt took too long (2506 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[16490.372302] Attempt to set a LOCK_MAND lock via flock(2). This support has been removed and the request ignored.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-04 12:54 btrfs send and receive showing errors Randy Nürnberger
@ 2023-01-04 13:41 ` Filipe Manana
  2023-01-05 10:10   ` Randy Nürnberger
  0 siblings, 1 reply; 13+ messages in thread
From: Filipe Manana @ 2023-01-04 13:41 UTC (permalink / raw)
  To: Randy Nürnberger; +Cc: linux-btrfs

On Wed, Jan 4, 2023 at 1:05 PM Randy Nürnberger <ranuberger@posteo.de> wrote:
>
> Hello,
>
> I’m in the process of copying a btrfs filesystem (a couple years old)
> from one disk to another by using btrfs send and receive on all
> snapshots. The snapshots were created by a tool every hour on the hour
> as one backup measure and automatically removed as they became older.
>
> I got errors like the following and when I compare the copied snapshots
> with the originals, some files are missing:
>
> ERROR: unlink █████ failed: No such file or directory
> ERROR: link █████ -> █████ failed: No such file or directory
> ERROR: utimes █████ failed: No such file or directory
> ERROR: rmdir █████ failed: No such file or directory
>
> Is this a known bug and how can I help diagnosing and fixing this?

So this is a problem caused by the sender issuing commands with outdated paths.

First, try doing the send operation again with a 6.1 kernel - there
was at least one fix here that may be relevant.

To actual debug things, in case it's not working with 6.1:

1) Show how you invoked send and receive. I.e. the full command lines
for 'btrfs send ...' and 'btrfs receive ...'

2) Provide the whole output of 'btrfs receive' with the -vvv command
line option.
    This will reveal all path names, but it's necessary to debug things.
    You've hidden path names above, so I suppose that's not acceptable for you.

Thanks.

>
>
> # uname -a  # this is the kernel on which this originally happend
> Linux arktos 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
> x86_64 GNU/Linux
>
>
> # uname -a  # I already retried everything on the latest Debian
> backports kernel with the same results
> Linux arktos 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian
> 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v5.10.1
>
> # btrfs fi sh /mnt/randy  # this is the source
> Label: none  uuid: 84bba855-578d-48db-80eb-49f1029c7543
>          Total devices 2 FS bytes used 2.04TiB
>          devid    1 size 4.00TiB used 2.05TiB path /dev/mapper/randy_1_crypt
>          devid    2 size 4.00TiB used 2.05TiB path /dev/mapper/randy_2_crypt
>
> # btrfs fi df /mnt/randy
> Data, RAID1: total=2.02TiB, used=2.02TiB
> System, RAID1: total=8.00MiB, used=320.00KiB
> Metadata, RAID1: total=25.00GiB, used=22.99GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
>
> # btrfs fi sh /mnt/intern  # this is the target
> Label: none  uuid: ebb94498-644c-42cd-892f-37886173523c
>          Total devices 2 FS bytes used 1.91TiB
>          devid    1 size 8.00TiB used 1.91TiB path
> /dev/mapper/vg_arktos_hdd_b-lv_randy_1
>          devid    2 size 8.00TiB used 1.91TiB path
> /dev/mapper/vg_arktos_hdd_b-lv_randy_2
>
> # btrfs fi df /mnt/intern
> Data, RAID1: total=1.89TiB, used=1.89TiB
> System, RAID1: total=8.00MiB, used=288.00KiB
> Metadata, RAID1: total=21.00GiB, used=20.76GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-04 13:41 ` Filipe Manana
@ 2023-01-05 10:10   ` Randy Nürnberger
  2023-01-05 10:42     ` Filipe Manana
  0 siblings, 1 reply; 13+ messages in thread
From: Randy Nürnberger @ 2023-01-05 10:10 UTC (permalink / raw)
  To: Filipe Manana; +Cc: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 6377 bytes --]

On Wed, Jan 4, 2023 at 14:41, Filipe Manana wrote:
> On Wed, Jan 4, 2023 at 1:05 PM Randy Nürnberger <ranuberger@posteo.de> wrote:
>> Hello,
>>
>> I’m in the process of copying a btrfs filesystem (a couple years old)
>> from one disk to another by using btrfs send and receive on all
>> snapshots. The snapshots were created by a tool every hour on the hour
>> as one backup measure and automatically removed as they became older.
>>
>> I got errors like the following and when I compare the copied snapshots
>> with the originals, some files are missing:
>>
>> ERROR: unlink █████ failed: No such file or directory
>> ERROR: link █████ -> █████ failed: No such file or directory
>> ERROR: utimes █████ failed: No such file or directory
>> ERROR: rmdir █████ failed: No such file or directory
>>
>> Is this a known bug and how can I help diagnosing and fixing this?
> So this is a problem caused by the sender issuing commands with outdated paths.
>
> First, try doing the send operation again with a 6.1 kernel - there
> was at least one fix here that may be relevant.

I tried again with the following kernel version and still got the same 
errors:
Linux arktos 6.1.0-0-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.2-1~exp1 
(2023-01-01) x86_64 GNU/Linux

>
> To actual debug things, in case it's not working with 6.1:
>
> 1) Show how you invoked send and receive. I.e. the full command lines
> for 'btrfs send ...' and 'btrfs receive ...'

Those are my full command lines:

btrfs send -p /mnt/randy/randy-snapshots/2022-01-29--07-00 
/mnt/randy/randy-snapshots/2022-02-27--10-00 | btrfs receive -vvv 
/mnt/intern/randy-snapshots/ 2>receive-1.txt

btrfs send -p /mnt/randy/randy-snapshots/2022-02-27--10-00 
/mnt/randy/randy-snapshots/2022-03-26--07-00 | btrfs receive -vvv 
/mnt/intern/randy-snapshots/ 2>receive-2.txt

>
> 2) Provide the whole output of 'btrfs receive' with the -vvv command
> line option.
>      This will reveal all path names, but it's necessary to debug things.
>      You've hidden path names above, so I suppose that's not acceptable for you.

At least I’m not comfortable sharing the file names on this public 
mailing list. I carefully tried to extract and redact what may be the 
relevant parts.

The second command line above is the first one that fails with the 
following error: “ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No 
such file or directory”.

This is the directory listing for the snapshot before said file was 
created (this snapshot can be copied without errors):

root@arktos /m/r/randy-snapshots# ls -alh 2022-01-29--07-00/Hase/Fuchs/
insgesamt 2,6M
drwxrws--- 1 randy randy  136 19. Dez 2021   ./
drwxrws--- 1 randy randy  134 24. Nov 2021   ../
-rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
-rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
-rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'

This is the directory listing for the snapshot after the file has been 
created (this snapshot can be copied without errors):

root@arktos /m/r/randy-snapshots# ls -alh 2022-02-27--10-00/Hase/Fuchs/
insgesamt 2,7M
drwxrws---  1 randy randy  178 27. Feb 2022   ./
drwxrws---  1 randy randy  134 24. Nov 2021   ../
-rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
-rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
-rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
-rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*

This is the directory listing for the snapshot after the file has been 
edited in LibreOffice and *renamed* (trying to copy this one fails):

root@arktos /m/r/randy-snapshots# ls -alh 2022-03-26--07-00/Hase/Fuchs/
insgesamt 2,6M
drwxrws--- 1 randy randy  178  5. Mär 2022   ./
drwxrws--- 1 randy randy  134 24. Nov 2021   ../
-rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
-rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
-rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
-rw-rw---- 1 randy randy  18K  4. Mär 2022  '2022-03-03 Reh.odt'

I’ve attached the outputs of the commands above. Apparently ‘btrfs send’ 
instructs ‘btrfs receive’ to unlink the file ‘Hase/Fuchs/2022-02-23 
Reh.odt’ which *does* exist in the copied snapshot ‘2022-02-27--10-00’ 
and this fails for whatever reason.

# sha256sum 
/mnt/randy/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\ Reh.odt
09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226

# sha256sum 
/mnt/intern/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\ 
Reh.odt
09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226

>
> Thanks.
>
>>
>> # uname -a  # this is the kernel on which this originally happend
>> Linux arktos 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
>> x86_64 GNU/Linux
>>
>>
>> # uname -a  # I already retried everything on the latest Debian
>> backports kernel with the same results
>> Linux arktos 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian
>> 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux
>>
>> # btrfs --version
>> btrfs-progs v5.10.1
>>
>> # btrfs fi sh /mnt/randy  # this is the source
>> Label: none  uuid: 84bba855-578d-48db-80eb-49f1029c7543
>>           Total devices 2 FS bytes used 2.04TiB
>>           devid    1 size 4.00TiB used 2.05TiB path /dev/mapper/randy_1_crypt
>>           devid    2 size 4.00TiB used 2.05TiB path /dev/mapper/randy_2_crypt
>>
>> # btrfs fi df /mnt/randy
>> Data, RAID1: total=2.02TiB, used=2.02TiB
>> System, RAID1: total=8.00MiB, used=320.00KiB
>> Metadata, RAID1: total=25.00GiB, used=22.99GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>>
>> # btrfs fi sh /mnt/intern  # this is the target
>> Label: none  uuid: ebb94498-644c-42cd-892f-37886173523c
>>           Total devices 2 FS bytes used 1.91TiB
>>           devid    1 size 8.00TiB used 1.91TiB path
>> /dev/mapper/vg_arktos_hdd_b-lv_randy_1
>>           devid    2 size 8.00TiB used 1.91TiB path
>> /dev/mapper/vg_arktos_hdd_b-lv_randy_2
>>
>> # btrfs fi df /mnt/intern
>> Data, RAID1: total=1.89TiB, used=1.89TiB
>> System, RAID1: total=8.00MiB, used=288.00KiB
>> Metadata, RAID1: total=21.00GiB, used=20.76GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B

[-- Attachment #2: receive-1-redacted.txt --]
[-- Type: text/plain, Size: 750 bytes --]

receiving snapshot 2022-02-27--10-00 uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, ctransid=1060 parent_uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, parent_ctransid=810
[…]
utimes Hase/Fuchs
[…]
mkfile o1525604-1014-0
rename o1525604-1014-0 -> Hase/Fuchs/2022-02-23 Reh.odt
utimes Hase/Fuchs
set_xattr Hase/Fuchs/2022-02-23 Reh.odt - name=system.posix_acl_access data_len=52 data=\x02
set_xattr Hase/Fuchs/2022-02-23 Reh.odt - name=user.DOSATTRIB data_len=32 data=
write Hase/Fuchs/2022-02-23 Reh.odt - offset=0 length=42184
chown Hase/Fuchs/2022-02-23 Reh.odt - uid=3000, gid=3000
chmod Hase/Fuchs/2022-02-23 Reh.odt - mode=0670
utimes Hase/Fuchs/2022-02-23 Reh.odt
BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, stransid=1060

[-- Attachment #3: receive-2-redacted.txt --]
[-- Type: text/plain, Size: 310 bytes --]

receiving snapshot 2022-03-26--07-00 uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, ctransid=1411 parent_uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, parent_ctransid=1060
[…]
utimes Hase/Fuchs
[…]
unlink Hase/Fuchs/2022-02-23 Reh.odt
ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No such file or directory

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 10:10   ` Randy Nürnberger
@ 2023-01-05 10:42     ` Filipe Manana
  2023-01-05 11:33       ` Randy Nürnberger
  0 siblings, 1 reply; 13+ messages in thread
From: Filipe Manana @ 2023-01-05 10:42 UTC (permalink / raw)
  To: Randy Nürnberger; +Cc: linux-btrfs

On Thu, Jan 5, 2023 at 10:10 AM Randy Nürnberger <ranuberger@posteo.de> wrote:
>
> On Wed, Jan 4, 2023 at 14:41, Filipe Manana wrote:
> > On Wed, Jan 4, 2023 at 1:05 PM Randy Nürnberger <ranuberger@posteo.de> wrote:
> >> Hello,
> >>
> >> I’m in the process of copying a btrfs filesystem (a couple years old)
> >> from one disk to another by using btrfs send and receive on all
> >> snapshots. The snapshots were created by a tool every hour on the hour
> >> as one backup measure and automatically removed as they became older.
> >>
> >> I got errors like the following and when I compare the copied snapshots
> >> with the originals, some files are missing:
> >>
> >> ERROR: unlink █████ failed: No such file or directory
> >> ERROR: link █████ -> █████ failed: No such file or directory
> >> ERROR: utimes █████ failed: No such file or directory
> >> ERROR: rmdir █████ failed: No such file or directory
> >>
> >> Is this a known bug and how can I help diagnosing and fixing this?
> > So this is a problem caused by the sender issuing commands with outdated paths.
> >
> > First, try doing the send operation again with a 6.1 kernel - there
> > was at least one fix here that may be relevant.
>
> I tried again with the following kernel version and still got the same
> errors:
> Linux arktos 6.1.0-0-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.2-1~exp1
> (2023-01-01) x86_64 GNU/Linux
>
> >
> > To actual debug things, in case it's not working with 6.1:
> >
> > 1) Show how you invoked send and receive. I.e. the full command lines
> > for 'btrfs send ...' and 'btrfs receive ...'
>
> Those are my full command lines:
>
> btrfs send -p /mnt/randy/randy-snapshots/2022-01-29--07-00
> /mnt/randy/randy-snapshots/2022-02-27--10-00 | btrfs receive -vvv
> /mnt/intern/randy-snapshots/ 2>receive-1.txt
>
> btrfs send -p /mnt/randy/randy-snapshots/2022-02-27--10-00
> /mnt/randy/randy-snapshots/2022-03-26--07-00 | btrfs receive -vvv
> /mnt/intern/randy-snapshots/ 2>receive-2.txt
>
> >
> > 2) Provide the whole output of 'btrfs receive' with the -vvv command
> > line option.
> >      This will reveal all path names, but it's necessary to debug things.
> >      You've hidden path names above, so I suppose that's not acceptable for you.
>
> At least I’m not comfortable sharing the file names on this public
> mailing list. I carefully tried to extract and redact what may be the
> relevant parts.
>
> The second command line above is the first one that fails with the
> following error: “ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No
> such file or directory”.
>
> This is the directory listing for the snapshot before said file was
> created (this snapshot can be copied without errors):
>
> root@arktos /m/r/randy-snapshots# ls -alh 2022-01-29--07-00/Hase/Fuchs/
> insgesamt 2,6M
> drwxrws--- 1 randy randy  136 19. Dez 2021   ./
> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
>
> This is the directory listing for the snapshot after the file has been
> created (this snapshot can be copied without errors):
>
> root@arktos /m/r/randy-snapshots# ls -alh 2022-02-27--10-00/Hase/Fuchs/
> insgesamt 2,7M
> drwxrws---  1 randy randy  178 27. Feb 2022   ./
> drwxrws---  1 randy randy  134 24. Nov 2021   ../
> -rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
> -rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
> -rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
> -rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*
>
> This is the directory listing for the snapshot after the file has been
> edited in LibreOffice and *renamed* (trying to copy this one fails):
>
> root@arktos /m/r/randy-snapshots# ls -alh 2022-03-26--07-00/Hase/Fuchs/
> insgesamt 2,6M
> drwxrws--- 1 randy randy  178  5. Mär 2022   ./
> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
> -rw-rw---- 1 randy randy  18K  4. Mär 2022  '2022-03-03 Reh.odt'
>
> I’ve attached the outputs of the commands above. Apparently ‘btrfs send’
> instructs ‘btrfs receive’ to unlink the file ‘Hase/Fuchs/2022-02-23
> Reh.odt’ which *does* exist in the copied snapshot ‘2022-02-27--10-00’
> and this fails for whatever reason.

The reason is very likely because the file was renamed, but the unlink
operation is using the old name (pre-rename name), instead of the new
name.

For the second receive, there should be other operations affecting the
file path Hase/Fuchs/2022-02-23 Reh.odt:

utimes Hase/Fuchs
[…]
unlink Hase/Fuchs/2022-02-23 Reh.odt
ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No such file or directory

Somewhere in the [...] there must be at least one rename of
Hase/Fuchs/2022-02-23 Reh.odt into something else.
It would be interesting to see more of the receive log to determine if
and why that rename happened.

If this is blocking you right now, you can do a full send of the
snapshot at /mnt/randy/randy-snapshots/2022-03-26--07-00.
That will, almost certainly, succeed. Though it will be slower.

Thanks.


>
> # sha256sum
> /mnt/randy/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\ Reh.odt
> 09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226
>
> # sha256sum
> /mnt/intern/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\
> Reh.odt
> 09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226
>
> >
> > Thanks.
> >
> >>
> >> # uname -a  # this is the kernel on which this originally happend
> >> Linux arktos 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
> >> x86_64 GNU/Linux
> >>
> >>
> >> # uname -a  # I already retried everything on the latest Debian
> >> backports kernel with the same results
> >> Linux arktos 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian
> >> 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux
> >>
> >> # btrfs --version
> >> btrfs-progs v5.10.1
> >>
> >> # btrfs fi sh /mnt/randy  # this is the source
> >> Label: none  uuid: 84bba855-578d-48db-80eb-49f1029c7543
> >>           Total devices 2 FS bytes used 2.04TiB
> >>           devid    1 size 4.00TiB used 2.05TiB path /dev/mapper/randy_1_crypt
> >>           devid    2 size 4.00TiB used 2.05TiB path /dev/mapper/randy_2_crypt
> >>
> >> # btrfs fi df /mnt/randy
> >> Data, RAID1: total=2.02TiB, used=2.02TiB
> >> System, RAID1: total=8.00MiB, used=320.00KiB
> >> Metadata, RAID1: total=25.00GiB, used=22.99GiB
> >> GlobalReserve, single: total=512.00MiB, used=0.00B
> >>
> >>
> >> # btrfs fi sh /mnt/intern  # this is the target
> >> Label: none  uuid: ebb94498-644c-42cd-892f-37886173523c
> >>           Total devices 2 FS bytes used 1.91TiB
> >>           devid    1 size 8.00TiB used 1.91TiB path
> >> /dev/mapper/vg_arktos_hdd_b-lv_randy_1
> >>           devid    2 size 8.00TiB used 1.91TiB path
> >> /dev/mapper/vg_arktos_hdd_b-lv_randy_2
> >>
> >> # btrfs fi df /mnt/intern
> >> Data, RAID1: total=1.89TiB, used=1.89TiB
> >> System, RAID1: total=8.00MiB, used=288.00KiB
> >> Metadata, RAID1: total=21.00GiB, used=20.76GiB
> >> GlobalReserve, single: total=512.00MiB, used=0.00B

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 10:42     ` Filipe Manana
@ 2023-01-05 11:33       ` Randy Nürnberger
  2023-01-05 11:49         ` Andrei Borzenkov
  0 siblings, 1 reply; 13+ messages in thread
From: Randy Nürnberger @ 2023-01-05 11:33 UTC (permalink / raw)
  To: Filipe Manana; +Cc: linux-btrfs

On Thu, Jan 5, 2023 at 11:42, Filipe Manana wrote:
> On Thu, Jan 5, 2023 at 10:10 AM Randy Nürnberger <ranuberger@posteo.de> wrote:
>> On Wed, Jan 4, 2023 at 14:41, Filipe Manana wrote:
>>> On Wed, Jan 4, 2023 at 1:05 PM Randy Nürnberger <ranuberger@posteo.de> wrote:
>>>> Hello,
>>>>
>>>> I’m in the process of copying a btrfs filesystem (a couple years old)
>>>> from one disk to another by using btrfs send and receive on all
>>>> snapshots. The snapshots were created by a tool every hour on the hour
>>>> as one backup measure and automatically removed as they became older.
>>>>
>>>> I got errors like the following and when I compare the copied snapshots
>>>> with the originals, some files are missing:
>>>>
>>>> ERROR: unlink █████ failed: No such file or directory
>>>> ERROR: link █████ -> █████ failed: No such file or directory
>>>> ERROR: utimes █████ failed: No such file or directory
>>>> ERROR: rmdir █████ failed: No such file or directory
>>>>
>>>> Is this a known bug and how can I help diagnosing and fixing this?
>>> So this is a problem caused by the sender issuing commands with outdated paths.
>>>
>>> First, try doing the send operation again with a 6.1 kernel - there
>>> was at least one fix here that may be relevant.
>> I tried again with the following kernel version and still got the same
>> errors:
>> Linux arktos 6.1.0-0-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.2-1~exp1
>> (2023-01-01) x86_64 GNU/Linux
>>
>>> To actual debug things, in case it's not working with 6.1:
>>>
>>> 1) Show how you invoked send and receive. I.e. the full command lines
>>> for 'btrfs send ...' and 'btrfs receive ...'
>> Those are my full command lines:
>>
>> btrfs send -p /mnt/randy/randy-snapshots/2022-01-29--07-00
>> /mnt/randy/randy-snapshots/2022-02-27--10-00 | btrfs receive -vvv
>> /mnt/intern/randy-snapshots/ 2>receive-1.txt
>>
>> btrfs send -p /mnt/randy/randy-snapshots/2022-02-27--10-00
>> /mnt/randy/randy-snapshots/2022-03-26--07-00 | btrfs receive -vvv
>> /mnt/intern/randy-snapshots/ 2>receive-2.txt
>>
>>> 2) Provide the whole output of 'btrfs receive' with the -vvv command
>>> line option.
>>>       This will reveal all path names, but it's necessary to debug things.
>>>       You've hidden path names above, so I suppose that's not acceptable for you.
>> At least I’m not comfortable sharing the file names on this public
>> mailing list. I carefully tried to extract and redact what may be the
>> relevant parts.
>>
>> The second command line above is the first one that fails with the
>> following error: “ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No
>> such file or directory”.
>>
>> This is the directory listing for the snapshot before said file was
>> created (this snapshot can be copied without errors):
>>
>> root@arktos /m/r/randy-snapshots# ls -alh 2022-01-29--07-00/Hase/Fuchs/
>> insgesamt 2,6M
>> drwxrws--- 1 randy randy  136 19. Dez 2021   ./
>> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
>> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
>>
>> This is the directory listing for the snapshot after the file has been
>> created (this snapshot can be copied without errors):
>>
>> root@arktos /m/r/randy-snapshots# ls -alh 2022-02-27--10-00/Hase/Fuchs/
>> insgesamt 2,7M
>> drwxrws---  1 randy randy  178 27. Feb 2022   ./
>> drwxrws---  1 randy randy  134 24. Nov 2021   ../
>> -rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>> -rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>> -rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
>> -rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*
>>
>> This is the directory listing for the snapshot after the file has been
>> edited in LibreOffice and *renamed* (trying to copy this one fails):
>>
>> root@arktos /m/r/randy-snapshots# ls -alh 2022-03-26--07-00/Hase/Fuchs/
>> insgesamt 2,6M
>> drwxrws--- 1 randy randy  178  5. Mär 2022   ./
>> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
>> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
>> -rw-rw---- 1 randy randy  18K  4. Mär 2022  '2022-03-03 Reh.odt'
>>
>> I’ve attached the outputs of the commands above. Apparently ‘btrfs send’
>> instructs ‘btrfs receive’ to unlink the file ‘Hase/Fuchs/2022-02-23
>> Reh.odt’ which *does* exist in the copied snapshot ‘2022-02-27--10-00’
>> and this fails for whatever reason.
> The reason is very likely because the file was renamed, but the unlink
> operation is using the old name (pre-rename name), instead of the new
> name.
>
> For the second receive, there should be other operations affecting the
> file path Hase/Fuchs/2022-02-23 Reh.odt:

Unfortunately, there are not.

I’m not sure how LibreOffice saves files, but it may create a new file 
and then replace the old one, so the unlink of the old file name can 
make sense, I think.

When I compare the two copied snapshots, the file is indeed gone, so the 
unlink operation actually seems to have worked?

root@arktos /m/i/randy-snapshots# ls -alh 2022-02-27--10-00/Hase/Fuchs/
insgesamt 2,7M
drwxrws---  1 randy randy  178 27. Feb 2022   ./
drwxrws---  1 randy randy  134 24. Nov 2021   ../
-rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
-rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
-rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
-rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*

root@arktos /m/i/randy-snapshots# ls -alh 2022-03-26--07-00/Hase/Fuchs/
insgesamt 2,6M
drwxrws--- 1 randy randy  136  5. Mär 2022   ./
drwxrws--- 1 randy randy  134 24. Nov 2021   ../
-rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
-rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
-rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'

I also just compiled btrfs-progs v6.1.1 and tried those and still got 
the same error.

>
> utimes Hase/Fuchs
> […]
> unlink Hase/Fuchs/2022-02-23 Reh.odt
> ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No such file or directory
>
> Somewhere in the [...] there must be at least one rename of
> Hase/Fuchs/2022-02-23 Reh.odt into something else.
> It would be interesting to see more of the receive log to determine if
> and why that rename happened.
>
> If this is blocking you right now, you can do a full send of the
> snapshot at /mnt/randy/randy-snapshots/2022-03-26--07-00.
> That will, almost certainly, succeed. Though it will be slower.

Thanks, I’m just copying my filesystem onto bigger disks and that can 
wait some time.

>
> Thanks.
>
>
>> # sha256sum
>> /mnt/randy/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\ Reh.odt
>> 09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226
>>
>> # sha256sum
>> /mnt/intern/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\
>> Reh.odt
>> 09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226
>>
>>> Thanks.
>>>
>>>> # uname -a  # this is the kernel on which this originally happend
>>>> Linux arktos 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
>>>> x86_64 GNU/Linux
>>>>
>>>>
>>>> # uname -a  # I already retried everything on the latest Debian
>>>> backports kernel with the same results
>>>> Linux arktos 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian
>>>> 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux
>>>>
>>>> # btrfs --version
>>>> btrfs-progs v5.10.1
>>>>
>>>> # btrfs fi sh /mnt/randy  # this is the source
>>>> Label: none  uuid: 84bba855-578d-48db-80eb-49f1029c7543
>>>>            Total devices 2 FS bytes used 2.04TiB
>>>>            devid    1 size 4.00TiB used 2.05TiB path /dev/mapper/randy_1_crypt
>>>>            devid    2 size 4.00TiB used 2.05TiB path /dev/mapper/randy_2_crypt
>>>>
>>>> # btrfs fi df /mnt/randy
>>>> Data, RAID1: total=2.02TiB, used=2.02TiB
>>>> System, RAID1: total=8.00MiB, used=320.00KiB
>>>> Metadata, RAID1: total=25.00GiB, used=22.99GiB
>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>
>>>>
>>>> # btrfs fi sh /mnt/intern  # this is the target
>>>> Label: none  uuid: ebb94498-644c-42cd-892f-37886173523c
>>>>            Total devices 2 FS bytes used 1.91TiB
>>>>            devid    1 size 8.00TiB used 1.91TiB path
>>>> /dev/mapper/vg_arktos_hdd_b-lv_randy_1
>>>>            devid    2 size 8.00TiB used 1.91TiB path
>>>> /dev/mapper/vg_arktos_hdd_b-lv_randy_2
>>>>
>>>> # btrfs fi df /mnt/intern
>>>> Data, RAID1: total=1.89TiB, used=1.89TiB
>>>> System, RAID1: total=8.00MiB, used=288.00KiB
>>>> Metadata, RAID1: total=21.00GiB, used=20.76GiB
>>>> GlobalReserve, single: total=512.00MiB, used=0.00B


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 11:33       ` Randy Nürnberger
@ 2023-01-05 11:49         ` Andrei Borzenkov
  2023-01-05 15:55           ` Randy Nürnberger
  0 siblings, 1 reply; 13+ messages in thread
From: Andrei Borzenkov @ 2023-01-05 11:49 UTC (permalink / raw)
  To: Randy Nürnberger, Filipe Manana; +Cc: linux-btrfs

On 05.01.2023 14:33, Randy Nürnberger wrote:
> On Thu, Jan 5, 2023 at 11:42, Filipe Manana wrote:
>> On Thu, Jan 5, 2023 at 10:10 AM Randy Nürnberger <ranuberger@posteo.de> wrote:
>>> On Wed, Jan 4, 2023 at 14:41, Filipe Manana wrote:
>>>> On Wed, Jan 4, 2023 at 1:05 PM Randy Nürnberger <ranuberger@posteo.de> wrote:
>>>>> Hello,
>>>>>
>>>>> I’m in the process of copying a btrfs filesystem (a couple years old)
>>>>> from one disk to another by using btrfs send and receive on all
>>>>> snapshots. The snapshots were created by a tool every hour on the hour
>>>>> as one backup measure and automatically removed as they became older.
>>>>>
>>>>> I got errors like the following and when I compare the copied snapshots
>>>>> with the originals, some files are missing:
>>>>>
>>>>> ERROR: unlink █████ failed: No such file or directory
>>>>> ERROR: link █████ -> █████ failed: No such file or directory
>>>>> ERROR: utimes █████ failed: No such file or directory
>>>>> ERROR: rmdir █████ failed: No such file or directory
>>>>>
>>>>> Is this a known bug and how can I help diagnosing and fixing this?
>>>> So this is a problem caused by the sender issuing commands with outdated paths.
>>>>
>>>> First, try doing the send operation again with a 6.1 kernel - there
>>>> was at least one fix here that may be relevant.
>>> I tried again with the following kernel version and still got the same
>>> errors:
>>> Linux arktos 6.1.0-0-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.2-1~exp1
>>> (2023-01-01) x86_64 GNU/Linux
>>>
>>>> To actual debug things, in case it's not working with 6.1:
>>>>
>>>> 1) Show how you invoked send and receive. I.e. the full command lines
>>>> for 'btrfs send ...' and 'btrfs receive ...'
>>> Those are my full command lines:
>>>
>>> btrfs send -p /mnt/randy/randy-snapshots/2022-01-29--07-00
>>> /mnt/randy/randy-snapshots/2022-02-27--10-00 | btrfs receive -vvv
>>> /mnt/intern/randy-snapshots/ 2>receive-1.txt
>>>
>>> btrfs send -p /mnt/randy/randy-snapshots/2022-02-27--10-00
>>> /mnt/randy/randy-snapshots/2022-03-26--07-00 | btrfs receive -vvv
>>> /mnt/intern/randy-snapshots/ 2>receive-2.txt
>>>
>>>> 2) Provide the whole output of 'btrfs receive' with the -vvv command
>>>> line option.
>>>>        This will reveal all path names, but it's necessary to debug things.
>>>>        You've hidden path names above, so I suppose that's not acceptable for you.
>>> At least I’m not comfortable sharing the file names on this public
>>> mailing list. I carefully tried to extract and redact what may be the
>>> relevant parts.
>>>
>>> The second command line above is the first one that fails with the
>>> following error: “ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No
>>> such file or directory”.
>>>
>>> This is the directory listing for the snapshot before said file was
>>> created (this snapshot can be copied without errors):
>>>
>>> root@arktos /m/r/randy-snapshots# ls -alh 2022-01-29--07-00/Hase/Fuchs/
>>> insgesamt 2,6M
>>> drwxrws--- 1 randy randy  136 19. Dez 2021   ./
>>> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
>>> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>>> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>>> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
>>>
>>> This is the directory listing for the snapshot after the file has been
>>> created (this snapshot can be copied without errors):
>>>
>>> root@arktos /m/r/randy-snapshots# ls -alh 2022-02-27--10-00/Hase/Fuchs/
>>> insgesamt 2,7M
>>> drwxrws---  1 randy randy  178 27. Feb 2022   ./
>>> drwxrws---  1 randy randy  134 24. Nov 2021   ../
>>> -rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>>> -rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>>> -rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
>>> -rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*
>>>
>>> This is the directory listing for the snapshot after the file has been
>>> edited in LibreOffice and *renamed* (trying to copy this one fails):
>>>
>>> root@arktos /m/r/randy-snapshots# ls -alh 2022-03-26--07-00/Hase/Fuchs/
>>> insgesamt 2,6M
>>> drwxrws--- 1 randy randy  178  5. Mär 2022   ./
>>> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
>>> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>>> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>>> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
>>> -rw-rw---- 1 randy randy  18K  4. Mär 2022  '2022-03-03 Reh.odt'
>>>
>>> I’ve attached the outputs of the commands above. Apparently ‘btrfs send’
>>> instructs ‘btrfs receive’ to unlink the file ‘Hase/Fuchs/2022-02-23
>>> Reh.odt’ which *does* exist in the copied snapshot ‘2022-02-27--10-00’
>>> and this fails for whatever reason.

Are you sure "btrfs receive" finds the correct parent? Have you ever 
flipped read-only property on any snapshot involved in btrfs send/receive?

Check "btrfs subvolume -uqR list" output for duplicated received UUID.

>> The reason is very likely because the file was renamed, but the unlink
>> operation is using the old name (pre-rename name), instead of the new
>> name.
>>
>> For the second receive, there should be other operations affecting the
>> file path Hase/Fuchs/2022-02-23 Reh.odt:
> 
> Unfortunately, there are not.
> 
> I’m not sure how LibreOffice saves files, but it may create a new file
> and then replace the old one, so the unlink of the old file name can
> make sense, I think.
> 
> When I compare the two copied snapshots, the file is indeed gone, so the
> unlink operation actually seems to have worked?
> 
> root@arktos /m/i/randy-snapshots# ls -alh 2022-02-27--10-00/Hase/Fuchs/
> insgesamt 2,7M
> drwxrws---  1 randy randy  178 27. Feb 2022   ./
> drwxrws---  1 randy randy  134 24. Nov 2021   ../
> -rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
> -rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
> -rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
> -rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*
> 
> root@arktos /m/i/randy-snapshots# ls -alh 2022-03-26--07-00/Hase/Fuchs/
> insgesamt 2,6M
> drwxrws--- 1 randy randy  136  5. Mär 2022   ./
> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
> 
> I also just compiled btrfs-progs v6.1.1 and tried those and still got
> the same error.
> 
>>
>> utimes Hase/Fuchs
>> […]
>> unlink Hase/Fuchs/2022-02-23 Reh.odt
>> ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No such file or directory
>>
>> Somewhere in the [...] there must be at least one rename of
>> Hase/Fuchs/2022-02-23 Reh.odt into something else.
>> It would be interesting to see more of the receive log to determine if
>> and why that rename happened.
>>
>> If this is blocking you right now, you can do a full send of the
>> snapshot at /mnt/randy/randy-snapshots/2022-03-26--07-00.
>> That will, almost certainly, succeed. Though it will be slower.
> 
> Thanks, I’m just copying my filesystem onto bigger disks and that can
> wait some time.
> 
>>
>> Thanks.
>>
>>
>>> # sha256sum
>>> /mnt/randy/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\ Reh.odt
>>> 09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226
>>>
>>> # sha256sum
>>> /mnt/intern/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\
>>> Reh.odt
>>> 09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226
>>>
>>>> Thanks.
>>>>
>>>>> # uname -a  # this is the kernel on which this originally happend
>>>>> Linux arktos 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
>>>>> x86_64 GNU/Linux
>>>>>
>>>>>
>>>>> # uname -a  # I already retried everything on the latest Debian
>>>>> backports kernel with the same results
>>>>> Linux arktos 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian
>>>>> 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux
>>>>>
>>>>> # btrfs --version
>>>>> btrfs-progs v5.10.1
>>>>>
>>>>> # btrfs fi sh /mnt/randy  # this is the source
>>>>> Label: none  uuid: 84bba855-578d-48db-80eb-49f1029c7543
>>>>>             Total devices 2 FS bytes used 2.04TiB
>>>>>             devid    1 size 4.00TiB used 2.05TiB path /dev/mapper/randy_1_crypt
>>>>>             devid    2 size 4.00TiB used 2.05TiB path /dev/mapper/randy_2_crypt
>>>>>
>>>>> # btrfs fi df /mnt/randy
>>>>> Data, RAID1: total=2.02TiB, used=2.02TiB
>>>>> System, RAID1: total=8.00MiB, used=320.00KiB
>>>>> Metadata, RAID1: total=25.00GiB, used=22.99GiB
>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>>
>>>>>
>>>>> # btrfs fi sh /mnt/intern  # this is the target
>>>>> Label: none  uuid: ebb94498-644c-42cd-892f-37886173523c
>>>>>             Total devices 2 FS bytes used 1.91TiB
>>>>>             devid    1 size 8.00TiB used 1.91TiB path
>>>>> /dev/mapper/vg_arktos_hdd_b-lv_randy_1
>>>>>             devid    2 size 8.00TiB used 1.91TiB path
>>>>> /dev/mapper/vg_arktos_hdd_b-lv_randy_2
>>>>>
>>>>> # btrfs fi df /mnt/intern
>>>>> Data, RAID1: total=1.89TiB, used=1.89TiB
>>>>> System, RAID1: total=8.00MiB, used=288.00KiB
>>>>> Metadata, RAID1: total=21.00GiB, used=20.76GiB
>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
> 


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 11:49         ` Andrei Borzenkov
@ 2023-01-05 15:55           ` Randy Nürnberger
  2023-01-05 16:01             ` Andrei Borzenkov
  0 siblings, 1 reply; 13+ messages in thread
From: Randy Nürnberger @ 2023-01-05 15:55 UTC (permalink / raw)
  To: Andrei Borzenkov, Filipe Manana; +Cc: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 11634 bytes --]

On Thu, Jan 5, 2023 at 12:49, Andrei Borzenkov wrote:
> On 05.01.2023 14:33, Randy Nürnberger wrote:
>> On Thu, Jan 5, 2023 at 11:42, Filipe Manana wrote:
>>> On Thu, Jan 5, 2023 at 10:10 AM Randy Nürnberger 
>>> <ranuberger@posteo.de> wrote:
>>>> On Wed, Jan 4, 2023 at 14:41, Filipe Manana wrote:
>>>>> On Wed, Jan 4, 2023 at 1:05 PM Randy Nürnberger 
>>>>> <ranuberger@posteo.de> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I’m in the process of copying a btrfs filesystem (a couple years 
>>>>>> old)
>>>>>> from one disk to another by using btrfs send and receive on all
>>>>>> snapshots. The snapshots were created by a tool every hour on the 
>>>>>> hour
>>>>>> as one backup measure and automatically removed as they became 
>>>>>> older.
>>>>>>
>>>>>> I got errors like the following and when I compare the copied 
>>>>>> snapshots
>>>>>> with the originals, some files are missing:
>>>>>>
>>>>>> ERROR: unlink █████ failed: No such file or directory
>>>>>> ERROR: link █████ -> █████ failed: No such file or directory
>>>>>> ERROR: utimes █████ failed: No such file or directory
>>>>>> ERROR: rmdir █████ failed: No such file or directory
>>>>>>
>>>>>> Is this a known bug and how can I help diagnosing and fixing this?
>>>>> So this is a problem caused by the sender issuing commands with 
>>>>> outdated paths.
>>>>>
>>>>> First, try doing the send operation again with a 6.1 kernel - there
>>>>> was at least one fix here that may be relevant.
>>>> I tried again with the following kernel version and still got the same
>>>> errors:
>>>> Linux arktos 6.1.0-0-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.2-1~exp1
>>>> (2023-01-01) x86_64 GNU/Linux
>>>>
>>>>> To actual debug things, in case it's not working with 6.1:
>>>>>
>>>>> 1) Show how you invoked send and receive. I.e. the full command lines
>>>>> for 'btrfs send ...' and 'btrfs receive ...'
>>>> Those are my full command lines:
>>>>
>>>> btrfs send -p /mnt/randy/randy-snapshots/2022-01-29--07-00
>>>> /mnt/randy/randy-snapshots/2022-02-27--10-00 | btrfs receive -vvv
>>>> /mnt/intern/randy-snapshots/ 2>receive-1.txt
>>>>
>>>> btrfs send -p /mnt/randy/randy-snapshots/2022-02-27--10-00
>>>> /mnt/randy/randy-snapshots/2022-03-26--07-00 | btrfs receive -vvv
>>>> /mnt/intern/randy-snapshots/ 2>receive-2.txt
>>>>
>>>>> 2) Provide the whole output of 'btrfs receive' with the -vvv command
>>>>> line option.
>>>>>        This will reveal all path names, but it's necessary to 
>>>>> debug things.
>>>>>        You've hidden path names above, so I suppose that's not 
>>>>> acceptable for you.
>>>> At least I’m not comfortable sharing the file names on this public
>>>> mailing list. I carefully tried to extract and redact what may be the
>>>> relevant parts.
>>>>
>>>> The second command line above is the first one that fails with the
>>>> following error: “ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt 
>>>> failed: No
>>>> such file or directory”.
>>>>
>>>> This is the directory listing for the snapshot before said file was
>>>> created (this snapshot can be copied without errors):
>>>>
>>>> root@arktos /m/r/randy-snapshots# ls -alh 
>>>> 2022-01-29--07-00/Hase/Fuchs/
>>>> insgesamt 2,6M
>>>> drwxrws--- 1 randy randy  136 19. Dez 2021   ./
>>>> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
>>>> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>>>> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>>>> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 
>>>> Wildschwein.pdf'
>>>>
>>>> This is the directory listing for the snapshot after the file has been
>>>> created (this snapshot can be copied without errors):
>>>>
>>>> root@arktos /m/r/randy-snapshots# ls -alh 
>>>> 2022-02-27--10-00/Hase/Fuchs/
>>>> insgesamt 2,7M
>>>> drwxrws---  1 randy randy  178 27. Feb 2022   ./
>>>> drwxrws---  1 randy randy  134 24. Nov 2021   ../
>>>> -rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>>>> -rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>>>> -rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 
>>>> Wildschwein.pdf'
>>>> -rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*
>>>>
>>>> This is the directory listing for the snapshot after the file has been
>>>> edited in LibreOffice and *renamed* (trying to copy this one fails):
>>>>
>>>> root@arktos /m/r/randy-snapshots# ls -alh 
>>>> 2022-03-26--07-00/Hase/Fuchs/
>>>> insgesamt 2,6M
>>>> drwxrws--- 1 randy randy  178  5. Mär 2022   ./
>>>> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
>>>> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>>>> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>>>> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 
>>>> Wildschwein.pdf'
>>>> -rw-rw---- 1 randy randy  18K  4. Mär 2022  '2022-03-03 Reh.odt'
>>>>
>>>> I’ve attached the outputs of the commands above. Apparently ‘btrfs 
>>>> send’
>>>> instructs ‘btrfs receive’ to unlink the file ‘Hase/Fuchs/2022-02-23
>>>> Reh.odt’ which *does* exist in the copied snapshot ‘2022-02-27--10-00’
>>>> and this fails for whatever reason.
>
> Are you sure "btrfs receive" finds the correct parent? Have you ever 
> flipped read-only property on any snapshot involved in btrfs 
> send/receive?
>
> Check "btrfs subvolume -uqR list" output for duplicated received UUID.

You are right, the parent UUID of the ‘2022-03-26--07-00’ snapshot is 
the UUID of ‘2022-01-29--07-00’ and not ‘2022-02-27--10-00’ as it should 
be. The question is why.

One year ago I once already recreated the *source* filesystem. I did 
this on a 5.10 kernel, by running the following fish-shell script:

mkfs.btrfs --metadata raid1 --data raid1 --checksum sha256 --features 
no-holes --runtime-features free-space-tree /dev/mapper/randy_1_crypt 
/dev/mapper/randy_2_crypt
mount -o noatime,compress-force=zstd:3 /dev/mapper/randy_1_crypt /mnt/randy
mkdir /mnt/randy/randy-snapshots

set snapshots /mnt/old-source/randy-snapshots/*
set destination /mnt/randy/randy-snapshots/
set subvol $snapshots[1]
set i_max (count $snapshots)
echo -e "\033[1m$subvol\033[0m (1 von $i_max)"
btrfs send -q $subvol | pv | btrfs receive $destination
for i in (seq 2 $i_max)
     set parent $snapshots[(math $i-1)]
     set subvol $snapshots[$i]
     echo -e "\033[1m$parent → $subvol\033[0m ($i von $i_max)"
     btrfs send -q -p $parent $subvol | pv | btrfs receive $destination
end

Then I moved the last snapshot and flipped its read-only property to get 
my new main working directory:

mv /mnt/randy/randy-snapshots/2022-01-24--11-53 /mnt/randy/randy
btrfs property set -ts /mnt/randy/randy ro false

I can’t remember getting any errors and all the file hashes underneath 
the final /mnt/randy/randy did equal those of the old source.

I’ve attached the output of ‘btrfs subvolume list’ for the source and 
the target filesystem.

>
>>> The reason is very likely because the file was renamed, but the unlink
>>> operation is using the old name (pre-rename name), instead of the new
>>> name.
>>>
>>> For the second receive, there should be other operations affecting the
>>> file path Hase/Fuchs/2022-02-23 Reh.odt:
>>
>> Unfortunately, there are not.
>>
>> I’m not sure how LibreOffice saves files, but it may create a new file
>> and then replace the old one, so the unlink of the old file name can
>> make sense, I think.
>>
>> When I compare the two copied snapshots, the file is indeed gone, so the
>> unlink operation actually seems to have worked?
>>
>> root@arktos /m/i/randy-snapshots# ls -alh 2022-02-27--10-00/Hase/Fuchs/
>> insgesamt 2,7M
>> drwxrws---  1 randy randy  178 27. Feb 2022   ./
>> drwxrws---  1 randy randy  134 24. Nov 2021   ../
>> -rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>> -rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>> -rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 
>> Wildschwein.pdf'
>> -rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*
>>
>> root@arktos /m/i/randy-snapshots# ls -alh 2022-03-26--07-00/Hase/Fuchs/
>> insgesamt 2,6M
>> drwxrws--- 1 randy randy  136  5. Mär 2022   ./
>> drwxrws--- 1 randy randy  134 24. Nov 2021   ../
>> -rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
>> -rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
>> -rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
>>
>> I also just compiled btrfs-progs v6.1.1 and tried those and still got
>> the same error.
>>
>>>
>>> utimes Hase/Fuchs
>>> […]
>>> unlink Hase/Fuchs/2022-02-23 Reh.odt
>>> ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No such file or 
>>> directory
>>>
>>> Somewhere in the [...] there must be at least one rename of
>>> Hase/Fuchs/2022-02-23 Reh.odt into something else.
>>> It would be interesting to see more of the receive log to determine if
>>> and why that rename happened.
>>>
>>> If this is blocking you right now, you can do a full send of the
>>> snapshot at /mnt/randy/randy-snapshots/2022-03-26--07-00.
>>> That will, almost certainly, succeed. Though it will be slower.
>>
>> Thanks, I’m just copying my filesystem onto bigger disks and that can
>> wait some time.
>>
>>>
>>> Thanks.
>>>
>>>
>>>> # sha256sum
>>>> /mnt/randy/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\ 
>>>> Reh.odt
>>>> 09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226
>>>>
>>>> # sha256sum
>>>> /mnt/intern/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\
>>>> Reh.odt
>>>> 09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226
>>>>
>>>>> Thanks.
>>>>>
>>>>>> # uname -a  # this is the kernel on which this originally happend
>>>>>> Linux arktos 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
>>>>>> x86_64 GNU/Linux
>>>>>>
>>>>>>
>>>>>> # uname -a  # I already retried everything on the latest Debian
>>>>>> backports kernel with the same results
>>>>>> Linux arktos 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian
>>>>>> 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux
>>>>>>
>>>>>> # btrfs --version
>>>>>> btrfs-progs v5.10.1
>>>>>>
>>>>>> # btrfs fi sh /mnt/randy  # this is the source
>>>>>> Label: none  uuid: 84bba855-578d-48db-80eb-49f1029c7543
>>>>>>             Total devices 2 FS bytes used 2.04TiB
>>>>>>             devid    1 size 4.00TiB used 2.05TiB path 
>>>>>> /dev/mapper/randy_1_crypt
>>>>>>             devid    2 size 4.00TiB used 2.05TiB path 
>>>>>> /dev/mapper/randy_2_crypt
>>>>>>
>>>>>> # btrfs fi df /mnt/randy
>>>>>> Data, RAID1: total=2.02TiB, used=2.02TiB
>>>>>> System, RAID1: total=8.00MiB, used=320.00KiB
>>>>>> Metadata, RAID1: total=25.00GiB, used=22.99GiB
>>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>>>
>>>>>>
>>>>>> # btrfs fi sh /mnt/intern  # this is the target
>>>>>> Label: none  uuid: ebb94498-644c-42cd-892f-37886173523c
>>>>>>             Total devices 2 FS bytes used 1.91TiB
>>>>>>             devid    1 size 8.00TiB used 1.91TiB path
>>>>>> /dev/mapper/vg_arktos_hdd_b-lv_randy_1
>>>>>>             devid    2 size 8.00TiB used 1.91TiB path
>>>>>> /dev/mapper/vg_arktos_hdd_b-lv_randy_2
>>>>>>
>>>>>> # btrfs fi df /mnt/intern
>>>>>> Data, RAID1: total=1.89TiB, used=1.89TiB
>>>>>> System, RAID1: total=8.00MiB, used=288.00KiB
>>>>>> Metadata, RAID1: total=21.00GiB, used=20.76GiB
>>>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>

[-- Attachment #2: subvolume-list-source.txt --]
[-- Type: text/plain, Size: 12354 bytes --]

ID 2020 gen 596 top level 5 parent_uuid 9b66b1e4-fa6c-2f4a-8379-231cbaa7769d received_uuid a98dd202-b528-264d-bd82-c9e07562e810 uuid e6f1bd57-5235-8c4d-8082-deaef3351daa path randy-snapshots/2021-06-27--13-00
ID 2022 gen 599 top level 5 parent_uuid e6f1bd57-5235-8c4d-8082-deaef3351daa received_uuid 5bc04547-28b8-ad4c-8fc0-fdc336882dd5 uuid 3c9c72c8-a190-7d43-92d5-0098e38e3fd6 path randy-snapshots/2021-07-30--22-00
ID 2024 gen 602 top level 5 parent_uuid 3c9c72c8-a190-7d43-92d5-0098e38e3fd6 received_uuid 1056fbaf-6e92-ba4a-bcef-cae50db71112 uuid ddde7784-bc30-5249-b1c2-3c9edc86d0f1 path randy-snapshots/2021-08-28--18-00
ID 2026 gen 606 top level 5 parent_uuid ddde7784-bc30-5249-b1c2-3c9edc86d0f1 received_uuid 215c8c10-996a-7446-a6ae-76e99433640e uuid 66e89882-2065-d248-ba88-b3eb49d29cee path randy-snapshots/2021-09-25--20-00
ID 2046 gen 628 top level 5 parent_uuid c5c55660-1290-064c-8452-2ae42ddd5a90 received_uuid 1543a180-3064-4447-94ea-850d4ae754dd uuid e5eb4cc2-e4d7-3f4d-9505-6ecfb405474d path randy-snapshots/2021-10-31--13-00
ID 2073 gen 644 top level 5 parent_uuid f689a4f9-fb83-2e4b-aa13-38e097be81b5 received_uuid 2f1746d8-a25c-204c-b74f-0605d445be8f uuid 217c661b-4a47-ef41-8511-65043ee3b645 path randy-snapshots/2021-11-27--19-00
ID 2270 gen 717 top level 5 parent_uuid 9a520141-fbf4-fd4b-853f-690e3981ae87 received_uuid 7840fe0c-8dda-d94f-a07d-c4bd9e3b429b uuid cacb7fa0-37ac-9c44-abd9-d3aeda499ea3 path randy-snapshots/2021-12-31--22-00
ID 2326 gen 3141 top level 5 parent_uuid f0d7eee1-ae15-4d41-b3a6-60bb71aaf030 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 161d4216-88ad-494e-8f10-a4b798c43632 path randy
ID 2331 gen 811 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 5f32e737-d24b-9847-8829-ab53c9854bff path randy-snapshots/2022-01-29--07-00
ID 2393 gen 1061 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 24c1e164-af08-a342-9e55-96d779f68107 path randy-snapshots/2022-02-27--10-00
ID 2468 gen 1412 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid ebc8010a-4353-9b42-8fc6-eeaeb4ed7160 path randy-snapshots/2022-03-26--07-00
ID 2520 gen 1591 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 685c3cc2-bc7d-184b-8b98-7079414337c2 path randy-snapshots/2022-04-30--07-00
ID 2643 gen 1853 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 0b3ca4e6-12a5-c644-9c9c-d56c8b085951 path randy-snapshots/2022-05-28--13-00
ID 2678 gen 1989 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 71d48faf-5f1a-c444-bb7d-d383b1760a02 path randy-snapshots/2022-06-25--07-00
ID 2728 gen 2167 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid c409a4b1-9c9e-c247-b5bf-c325dcf37442 path randy-snapshots/2022-07-30--07-00
ID 2774 gen 2338 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 9a228d8d-6354-734b-9629-da36522210c9 path randy-snapshots/2022-08-27--07-00
ID 2798 gen 2471 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 9daeab29-291a-e749-90ef-94015b714962 path randy-snapshots/2022-09-04--22-00
ID 2805 gen 2490 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 0da2d153-632f-6243-b129-056e81cbab8b path randy-snapshots/2022-09-10--07-00
ID 2814 gen 2518 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 16841f00-1630-8a40-ac9c-7d7f4f6f35e3 path randy-snapshots/2022-09-17--07-00
ID 2828 gen 2555 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 715b2403-e22c-2043-963c-4837f995fbc3 path randy-snapshots/2022-09-25--21-00
ID 2838 gen 2584 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 474c1eb9-fa5d-8c49-aed0-a07b92d24c35 path randy-snapshots/2022-10-01--07-00
ID 2846 gen 2611 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 8e09dac0-8df9-0142-b2c8-bbe64f6f6c35 path randy-snapshots/2022-10-08--07-00
ID 2854 gen 2634 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 1de30ecc-f31b-7d4e-90e0-b90af5cabd3e path randy-snapshots/2022-10-15--07-00
ID 2860 gen 2651 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 4083508a-b024-d449-b612-602a4f36df5b path randy-snapshots/2022-10-22--07-00
ID 2872 gen 2697 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 8909aaf6-b818-1642-b3b2-253cf45b6999 path randy-snapshots/2022-10-30--20-00
ID 2880 gen 2721 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 43bfbf87-2a7f-5e41-8471-6e5228b40490 path randy-snapshots/2022-11-06--02-00
ID 2891 gen 2749 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid b48ad9fa-b444-8e4f-b8a3-c4f9e487b57c path randy-snapshots/2022-11-12--07-00
ID 2899 gen 2780 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid b8c1c9a8-86e0-5a4b-9d8a-99db726c604a path randy-snapshots/2022-11-19--07-00
ID 2912 gen 2826 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid fa0d25dc-af89-2644-968b-c82773af7201 path randy-snapshots/2022-11-27--22-00
ID 2939 gen 2891 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid effc10b4-14a7-6843-a10b-d635fb072c17 path randy-snapshots/2022-12-04--09-00
ID 2940 gen 2893 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 74530e4a-505b-4b44-be0e-e8bebfb72f49 path randy-snapshots/2022-12-05--07-00
ID 2942 gen 2901 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid ae1771e8-9ab9-f64b-a51c-335416802736 path randy-snapshots/2022-12-07--14-00
ID 2943 gen 2903 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 643cf260-7104-1242-9271-f8dbe1412383 path randy-snapshots/2022-12-08--07-00
ID 2944 gen 2905 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 8eb7369f-3261-2742-bc39-9e3c5d0ac011 path randy-snapshots/2022-12-09--07-00
ID 2945 gen 2909 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 1363d673-f805-1946-abe7-bbeab15fc3ef path randy-snapshots/2022-12-10--07-00
ID 2947 gen 2916 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid ca320f95-39ae-d845-acb3-7fd34e91ad50 path randy-snapshots/2022-12-12--07-00
ID 2948 gen 2918 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid ce769c6d-bb4f-9941-9f33-5efe9affa4f4 path randy-snapshots/2022-12-13--07-00
ID 2951 gen 2926 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 3d40a32d-203d-de40-93a7-db79d40d1b5e path randy-snapshots/2022-12-14--15-00
ID 2954 gen 2935 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 0a66d431-2fa2-9542-8689-0eba3eea6017 path randy-snapshots/2022-12-15--22-00
ID 2957 gen 2950 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 47807843-f86e-fb49-98e8-51d539f8ebcc path randy-snapshots/2022-12-16--20-00
ID 2959 gen 2960 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid bbe8b671-8c68-aa49-bb92-e2fb32573305 path randy-snapshots/2022-12-17--23-00
ID 2961 gen 2971 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 59a6958b-b2da-494f-af7c-3dfa4af33fc0 path randy-snapshots/2022-12-18--16-00
ID 2963 gen 2977 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 9c120102-4d82-3049-a4cf-19320df05d42 path randy-snapshots/2022-12-19--21-00
ID 2965 gen 2983 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 9bb99dee-0c4c-fa41-ac79-a93a3c51caf2 path randy-snapshots/2022-12-20--21-00
ID 2970 gen 3005 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 0a792d68-6374-744e-a3df-ece3de5765b9 path randy-snapshots/2022-12-21--21-00
ID 2976 gen 3036 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid e013f9db-91d8-6f44-8996-8b6ff0cad61f path randy-snapshots/2022-12-22--23-00
ID 2979 gen 3050 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid d42085f5-df4c-c34e-8689-740d3c830acc path randy-snapshots/2022-12-23--07-00
ID 2980 gen 3052 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid e6e6591e-cf89-5e4e-97e8-fad0c56c1cb8 path randy-snapshots/2022-12-24--07-00
ID 2981 gen 3059 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid e55ebaaf-903b-4b4d-9d8e-3a61bcff690b path randy-snapshots/2022-12-26--02-00
ID 2983 gen 3070 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid aca28ea9-56d8-0847-96db-bbbbcd40c763 path randy-snapshots/2022-12-27--15-00
ID 2984 gen 3072 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 015a4b18-7e61-2445-a7c7-efb572185a5a path randy-snapshots/2022-12-28--07-00
ID 2985 gen 3080 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 4b4902d4-eef9-fe40-9c0c-05919cdd8701 path randy-snapshots/2022-12-29--01-00
ID 2986 gen 3082 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid c5d26887-2344-e043-a795-25a6ef7cf3f4 path randy-snapshots/2022-12-29--07-00
ID 2987 gen 3084 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 8b75274e-3321-0849-8456-6a76c6636a5e path randy-snapshots/2022-12-29--11-00
ID 2988 gen 3089 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 68853890-6443-904f-81a2-6800ce784a7c path randy-snapshots/2022-12-29--12-00
ID 2989 gen 3091 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid bade1883-bfbf-504a-87a9-7bdc39949cfd path randy-snapshots/2022-12-29--15-00
ID 2990 gen 3093 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid c6963d0f-5ab0-6e4c-a120-e56a7aaa717e path randy-snapshots/2022-12-30--07-00
ID 2991 gen 3096 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid baba0331-e9ea-e449-b673-36ce7d339f14 path randy-snapshots/2022-12-31--01-00
ID 2992 gen 3100 top level 5 parent_uuid 161d4216-88ad-494e-8f10-a4b798c43632 received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 4a043acc-bbc1-5d48-a864-9d5c21cca0d1 path randy-snapshots/2022-12-31--07-00

[-- Attachment #3: subvolume-list-target.txt --]
[-- Type: text/plain, Size: 2080 bytes --]

ID 256 gen 467 top level 5 parent_uuid -                                    received_uuid a98dd202-b528-264d-bd82-c9e07562e810 uuid ad177e5b-9fc5-4f43-b603-1121d5fc965d path randy-snapshots/2021-06-27--13-00
ID 257 gen 470 top level 5 parent_uuid ad177e5b-9fc5-4f43-b603-1121d5fc965d received_uuid 5bc04547-28b8-ad4c-8fc0-fdc336882dd5 uuid 3a0570f8-13a0-5b46-bc90-8418f5284f48 path randy-snapshots/2021-07-30--22-00
ID 258 gen 473 top level 5 parent_uuid 3a0570f8-13a0-5b46-bc90-8418f5284f48 received_uuid 1056fbaf-6e92-ba4a-bcef-cae50db71112 uuid 7d7423c2-dfbf-3545-8c43-8beac745a97c path randy-snapshots/2021-08-28--18-00
ID 259 gen 476 top level 5 parent_uuid 7d7423c2-dfbf-3545-8c43-8beac745a97c received_uuid 215c8c10-996a-7446-a6ae-76e99433640e uuid 29acc7a2-e82a-1e49-97a2-8c30e326baa6 path randy-snapshots/2021-09-25--20-00
ID 260 gen 483 top level 5 parent_uuid 29acc7a2-e82a-1e49-97a2-8c30e326baa6 received_uuid 1543a180-3064-4447-94ea-850d4ae754dd uuid 12d05de7-d93c-0341-8f7e-f06e07bcbaf3 path randy-snapshots/2021-10-31--13-00
ID 261 gen 489 top level 5 parent_uuid 12d05de7-d93c-0341-8f7e-f06e07bcbaf3 received_uuid 2f1746d8-a25c-204c-b74f-0605d445be8f uuid 0a1be929-08e6-a042-8eaa-9a915ed7e68a path randy-snapshots/2021-11-27--19-00
ID 262 gen 559 top level 5 parent_uuid 0a1be929-08e6-a042-8eaa-9a915ed7e68a received_uuid 7840fe0c-8dda-d94f-a07d-c4bd9e3b429b uuid f0198883-2fa7-7840-87c2-08c04fb6c13e path randy-snapshots/2021-12-31--22-00
ID 263 gen 692 top level 5 parent_uuid f0198883-2fa7-7840-87c2-08c04fb6c13e received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid d17fdecb-240a-8744-857b-c1352f3ab0ea path randy-snapshots/2022-01-29--07-00
ID 317 gen 692 top level 5 parent_uuid d17fdecb-240a-8744-857b-c1352f3ab0ea received_uuid 5b98e81e-0e9b-814a-b191-19fe36517d1c uuid 81a5ef26-3a42-9049-9f40-dc78aeef4198 path randy-snapshots/2022-02-27--10-00
ID 318 gen 693 top level 5 parent_uuid d17fdecb-240a-8744-857b-c1352f3ab0ea received_uuid -                                    uuid 2d2caa0a-3707-6843-b60d-79dd63df3cec path randy-snapshots/2022-03-26--07-00

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 15:55           ` Randy Nürnberger
@ 2023-01-05 16:01             ` Andrei Borzenkov
  2023-01-05 16:35               ` Randy Nürnberger
  0 siblings, 1 reply; 13+ messages in thread
From: Andrei Borzenkov @ 2023-01-05 16:01 UTC (permalink / raw)
  To: Randy Nürnberger, Filipe Manana; +Cc: linux-btrfs

On 05.01.2023 18:55, Randy Nürnberger wrote:
> 
> I’ve attached the output of ‘btrfs subvolume list’ for the source and
> the target filesystem.
> 

You have a lot of source subvolumes with the same received_uuid and you 
have at least two destination subvolumes with the same received_uuid. 
This is wrong, received_uuid is supposed to be unique identification 
which explains your errors (incorrect subvolume is selected).

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 16:01             ` Andrei Borzenkov
@ 2023-01-05 16:35               ` Randy Nürnberger
  2023-01-05 16:47                 ` Randy Nürnberger
  0 siblings, 1 reply; 13+ messages in thread
From: Randy Nürnberger @ 2023-01-05 16:35 UTC (permalink / raw)
  To: Andrei Borzenkov, Filipe Manana; +Cc: linux-btrfs

On Thu, Jan 5, 2023 at 17:01, Andrei Borzenkov wrote:
> On 05.01.2023 18:55, Randy Nürnberger wrote:
>>
>> I’ve attached the output of ‘btrfs subvolume list’ for the source and
>> the target filesystem.
>>
>
> You have a lot of source subvolumes with the same received_uuid and 
> you have at least two destination subvolumes with the same 
> received_uuid. This is wrong, received_uuid is supposed to be unique 
> identification which explains your errors (incorrect subvolume is 
> selected).

I guess many of my source subvolumes have the same received_uuid, 
because I created new snapshots from a snapshot that was previously 
received and the received_uuid just did get copied.

I just did a small experiment on another system, created a snapshot from 
a snapshot that previously was received and could confirm that the 
received_uuid does indeed get copied. Is this a problem?

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 16:35               ` Randy Nürnberger
@ 2023-01-05 16:47                 ` Randy Nürnberger
  2023-01-05 17:48                   ` Andrei Borzenkov
  0 siblings, 1 reply; 13+ messages in thread
From: Randy Nürnberger @ 2023-01-05 16:47 UTC (permalink / raw)
  To: Andrei Borzenkov, Filipe Manana; +Cc: linux-btrfs

On Thu, Jan 5, 2023 at 17:35, Randy Nürnberger wrote:
> On Thu, Jan 5, 2023 at 17:01, Andrei Borzenkov wrote:
>> On 05.01.2023 18:55, Randy Nürnberger wrote:
>>>
>>> I’ve attached the output of ‘btrfs subvolume list’ for the source and
>>> the target filesystem.
>>>
>>
>> You have a lot of source subvolumes with the same received_uuid and 
>> you have at least two destination subvolumes with the same 
>> received_uuid. This is wrong, received_uuid is supposed to be unique 
>> identification which explains your errors (incorrect subvolume is 
>> selected).
>
> I guess many of my source subvolumes have the same received_uuid, 
> because I created new snapshots from a snapshot that was previously 
> received and the received_uuid just did get copied.
>
> I just did a small experiment on another system, created a snapshot 
> from a snapshot that previously was received and could confirm that 
> the received_uuid does indeed get copied. Is this a problem?

Ha! This seems to be the problem! I’m able to write a small script that 
reproduces the bug! Give me a couple minutes.

If this is a bug in the kernel, may I write the patch? :D Would be my 
first one :)

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 16:47                 ` Randy Nürnberger
@ 2023-01-05 17:48                   ` Andrei Borzenkov
  2023-01-05 18:04                     ` Andrei Borzenkov
  0 siblings, 1 reply; 13+ messages in thread
From: Andrei Borzenkov @ 2023-01-05 17:48 UTC (permalink / raw)
  To: Randy Nürnberger, Filipe Manana; +Cc: linux-btrfs

On 05.01.2023 19:47, Randy Nürnberger wrote:
> On Thu, Jan 5, 2023 at 17:35, Randy Nürnberger wrote:
>> On Thu, Jan 5, 2023 at 17:01, Andrei Borzenkov wrote:
>>> On 05.01.2023 18:55, Randy Nürnberger wrote:
>>>>
>>>> I’ve attached the output of ‘btrfs subvolume list’ for the source and
>>>> the target filesystem.
>>>>
>>>
>>> You have a lot of source subvolumes with the same received_uuid and
>>> you have at least two destination subvolumes with the same
>>> received_uuid. This is wrong, received_uuid is supposed to be unique
>>> identification which explains your errors (incorrect subvolume is
>>> selected).
>>
>> I guess many of my source subvolumes have the same received_uuid,
>> because I created new snapshots from a snapshot that was previously
>> received and the received_uuid just did get copied.
>>
>> I just did a small experiment on another system, created a snapshot
>> from a snapshot that previously was received and could confirm that
>> the received_uuid does indeed get copied. Is this a problem?
> 

No, it is not a problem by itself. Subvolumes used for send/receive are 
supposed to be a remain read-only, so any clone of any subvolume should 
have identical content and you can use any of the clones with the same 
received_uuid. But as soon as you made subvolume read-write, you got 
multiple subvolumes with different content and the same receive_uuid. 
"Doctor, it hurts when I stab myself in the eye".

Never, NEVER, ever, change  subvolume involved in send/receive to 
read-write to use it; clone it (create writable snapshot) and use clone.

> Ha! This seems to be the problem! I’m able to write a small script that
> reproduces the bug! Give me a couple minutes.
> 
> If this is a bug in the kernel,

It is not so much a bug as the consequence of unfortunate design.

> may I write the patch? :D Would be my
> first one :)

The problem is known for years (just search this list) and there are 
patches to clear received_uuid on ro -> rw transition but they are still 
not applied. Personally I would just prohibit such transition completely.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 17:48                   ` Andrei Borzenkov
@ 2023-01-05 18:04                     ` Andrei Borzenkov
  2023-01-06 11:13                       ` Randy Nürnberger
  0 siblings, 1 reply; 13+ messages in thread
From: Andrei Borzenkov @ 2023-01-05 18:04 UTC (permalink / raw)
  To: Randy Nürnberger, Filipe Manana; +Cc: linux-btrfs

On 05.01.2023 20:48, Andrei Borzenkov wrote:
> On 05.01.2023 19:47, Randy Nürnberger wrote:
>> On Thu, Jan 5, 2023 at 17:35, Randy Nürnberger wrote:
>>> On Thu, Jan 5, 2023 at 17:01, Andrei Borzenkov wrote:
>>>> On 05.01.2023 18:55, Randy Nürnberger wrote:
>>>>>
>>>>> I’ve attached the output of ‘btrfs subvolume list’ for the source and
>>>>> the target filesystem.
>>>>>
>>>>
>>>> You have a lot of source subvolumes with the same received_uuid and
>>>> you have at least two destination subvolumes with the same
>>>> received_uuid. This is wrong, received_uuid is supposed to be unique
>>>> identification which explains your errors (incorrect subvolume is
>>>> selected).
>>>
>>> I guess many of my source subvolumes have the same received_uuid,
>>> because I created new snapshots from a snapshot that was previously
>>> received and the received_uuid just did get copied.
>>>
>>> I just did a small experiment on another system, created a snapshot
>>> from a snapshot that previously was received and could confirm that
>>> the received_uuid does indeed get copied. Is this a problem?
>>
> 
> No, it is not a problem by itself. Subvolumes used for send/receive are
> supposed to be a remain read-only, so any clone of any subvolume should
> have identical content and you can use any of the clones with the same
> received_uuid. But as soon as you made subvolume read-write, you got
> multiple subvolumes with different content and the same receive_uuid.
> "Doctor, it hurts when I stab myself in the eye".
> 
> Never, NEVER, ever, change  subvolume involved in send/receive to
> read-write to use it; clone it (create writable snapshot) and use clone.
> 
>> Ha! This seems to be the problem! I’m able to write a small script that
>> reproduces the bug! Give me a couple minutes.
>>
>> If this is a bug in the kernel,
> 
> It is not so much a bug as the consequence of unfortunate design.
> 
>> may I write the patch? :D Would be my
>> first one :)
> 
> The problem is known for years (just search this list) and there are
> patches to clear received_uuid on ro -> rw transition but they are still
> not applied. Personally I would just prohibit such transition completely.

Correction - it was decided to implement in btrfs utils, not in kernel, 
so "btrfs property" will by default refuse to make read-write subvolume 
with received_uuid. This is implemented in 5.14.2.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: btrfs send and receive showing errors
  2023-01-05 18:04                     ` Andrei Borzenkov
@ 2023-01-06 11:13                       ` Randy Nürnberger
  0 siblings, 0 replies; 13+ messages in thread
From: Randy Nürnberger @ 2023-01-06 11:13 UTC (permalink / raw)
  To: Andrei Borzenkov, Filipe Manana; +Cc: linux-btrfs

On Thu, Jan 5, 2023 at 19:04, Andrei Borzenkov wrote:
> On 05.01.2023 20:48, Andrei Borzenkov wrote:
>> On 05.01.2023 19:47, Randy Nürnberger wrote:
>>> On Thu, Jan 5, 2023 at 17:35, Randy Nürnberger wrote:
>>>> On Thu, Jan 5, 2023 at 17:01, Andrei Borzenkov wrote:
>>>>> On 05.01.2023 18:55, Randy Nürnberger wrote:
>>>>>>
>>>>>> I’ve attached the output of ‘btrfs subvolume list’ for the source 
>>>>>> and
>>>>>> the target filesystem.
>>>>>>
>>>>>
>>>>> You have a lot of source subvolumes with the same received_uuid and
>>>>> you have at least two destination subvolumes with the same
>>>>> received_uuid. This is wrong, received_uuid is supposed to be unique
>>>>> identification which explains your errors (incorrect subvolume is
>>>>> selected).
>>>>
>>>> I guess many of my source subvolumes have the same received_uuid,
>>>> because I created new snapshots from a snapshot that was previously
>>>> received and the received_uuid just did get copied.
>>>>
>>>> I just did a small experiment on another system, created a snapshot
>>>> from a snapshot that previously was received and could confirm that
>>>> the received_uuid does indeed get copied. Is this a problem?
>>>
>>
>> No, it is not a problem by itself. Subvolumes used for send/receive are
>> supposed to be a remain read-only, so any clone of any subvolume should
>> have identical content and you can use any of the clones with the same
>> received_uuid. But as soon as you made subvolume read-write, you got
>> multiple subvolumes with different content and the same receive_uuid.
>> "Doctor, it hurts when I stab myself in the eye".
>>
>> Never, NEVER, ever, change  subvolume involved in send/receive to
>> read-write to use it; clone it (create writable snapshot) and use clone.

I mistakenly thought that creating a new writable snapshot and making a 
read-only snapshot writable were the same operation.

Thinking about it, I understand the difference and why it is there.

>>
>>> Ha! This seems to be the problem! I’m able to write a small script that
>>> reproduces the bug! Give me a couple minutes.
>>>
>>> If this is a bug in the kernel,
>>
>> It is not so much a bug as the consequence of unfortunate design.
>>
>>> may I write the patch? :D Would be my
>>> first one :)
>>
>> The problem is known for years (just search this list) and there are
>> patches to clear received_uuid on ro -> rw transition but they are still
>> not applied. Personally I would just prohibit such transition 
>> completely.
>
> Correction - it was decided to implement in btrfs utils, not in 
> kernel, so "btrfs property" will by default refuse to make read-write 
> subvolume with received_uuid. This is implemented in 5.14.2.

With this knowledge I managed to repare and successfully copy my filesystem.

I have learned a lot. Thank you both for helping me and for your time.

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2023-01-06 11:13 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-04 12:54 btrfs send and receive showing errors Randy Nürnberger
2023-01-04 13:41 ` Filipe Manana
2023-01-05 10:10   ` Randy Nürnberger
2023-01-05 10:42     ` Filipe Manana
2023-01-05 11:33       ` Randy Nürnberger
2023-01-05 11:49         ` Andrei Borzenkov
2023-01-05 15:55           ` Randy Nürnberger
2023-01-05 16:01             ` Andrei Borzenkov
2023-01-05 16:35               ` Randy Nürnberger
2023-01-05 16:47                 ` Randy Nürnberger
2023-01-05 17:48                   ` Andrei Borzenkov
2023-01-05 18:04                     ` Andrei Borzenkov
2023-01-06 11:13                       ` Randy Nürnberger

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.