Hello, I've been experiencing crashes using zswap with the z3fold allocator. The crashes happen once I'm using more memory than fits in the zpool. I can reliably reproduce the crash on v5.3-rc7 with these steps: 1. Enable swap. My swap is located on a small 16 GB SSD that I dedicate to this purpose, encrypted with dm-crypt. I didn't test other backing devices. 2. echo z3fold > /sys/module/zswap/parameters/zpool echo lz4 > /sys/module/zswap/parameters/compressor echo 1 > /sys/module/zswap/parameters/enabled 3. `watch -d grep -r . /sys/kernel/debug/zswap/` to monitor the statistics. The most important parameter is pool_limit_hit. 4. Use a lot of memory. I spawn several `stress -m 1 --vm-hang 0&`. 5. At some point, pool_limit_hit starts increasing. Soon after TRACE 1 (at the bottom of the email) appears in dmesg. 6. Now at least 1 process is in an unkillable state and I must reset my CPU. Additional notes: - I can't reproduce the issue with zbud or zsmalloc. For now I'm using zsmalloc as a workaround. - Sometimes TRACE 2 and/or TRACE 3 appear in dmesg after or before the protection fault. - If I run swapoff after the fault it seems to only move a handful of pages per second so I haven't left it running long enough to see if it ever finishes. I first noticed this bug on 5.2.11 but I noticed there had been several commits to z3fold in 5.3 so I compiled rc7 using the Arch Linux config file. I think this isn't a regression: A few months ago I tried zswap and ended up disabling it because it crashed my machine, but I didn't have time to figure out why back then. But that might have been a different bug. I'm willing to test older versions, of course. I'm sorry if this has already been reported, I've seen a few similar reports but not one exactly like this. Kind regards, Agustín. Additional information: /proc/version Linux version 5.3.0-rc7-1-ARCH (agustin@hostname) (gcc version 9.1.0 (GCC)) #1 SMP PREEMPT Wed Sep 4 17:23:45 -03 2019 /proc/iomem 00000000-00000fff : Reserved 00001000-00057fff : System RAM 00058000-00058fff : Reserved 00059000-0008bfff : System RAM 0008c000-0009ffff : Reserved 000a0000-000bffff : PCI Bus 0000:00 000c0000-000cfdff : Video ROM 000d0000-000d3fff : pnp 00:00 000d4000-000d7fff : pnp 00:00 000d8000-000dbfff : pnp 00:00 000dc000-000dffff : pnp 00:00 000e0000-000fffff : Reserved 000f0000-000fffff : System ROM 00100000-c006d017 : System RAM 66600000-67200e20 : Kernel code 67200e21-6794a3ff : Kernel data 67e87000-681fffff : Kernel bss c006d018-c007d657 : System RAM c007d658-c007e017 : System RAM c007e018-c008e057 : System RAM c008e058-cbd01fff : System RAM cbd02000-ccbfdfff : Reserved ccbfe000-ccd7dfff : ACPI Non-volatile Storage ccd7e000-ccdfdfff : ACPI Tables ccdfe000-ccdfefff : System RAM ccdff000-ccdfffff : MSFT0101:00 ccdff000-ccdff02f : MSFT0101:00 ccdff080-ccdfffff : MSFT0101:00 cce00000-cdffffff : RAM buffer ce000000-cfffffff : Reserved ce000000-cfffffff : Graphics Stolen Memory d0000000-febfffff : PCI Bus 0000:00 d0000000-d000ffff : pnp 00:05 d0010000-d001ffff : pnp 00:05 e0000000-efffffff : 0000:00:02.0 f0000000-f0ffffff : 0000:00:02.0 f1000000-f10fffff : PCI Bus 0000:03 f1000000-f1001fff : 0000:03:00.0 f1000000-f1001fff : iwlwifi f1100000-f11fffff : PCI Bus 0000:02 f1100000-f1100fff : 0000:02:00.0 f1100000-f1100fff : rtsx_pci f1200000-f121ffff : 0000:00:19.0 f1200000-f121ffff : e1000e f1220000-f122ffff : 0000:00:14.0 f1220000-f122ffff : xhci-hcd f1230000-f1233fff : 0000:00:03.0 f1230000-f1233fff : ICH HD audio f1234000-f1237fff : 0000:00:1b.0 f1234000-f1237fff : ICH HD audio f1238000-f12380ff : 0000:00:1f.3 f1239000-f123901f : 0000:00:16.0 f1239000-f123901f : mei_me f123b000-f123bfff : 0000:00:1f.6 f123b000-f123bfff : Intel PCH thermal driver f123c000-f123c7ff : 0000:00:1f.2 f123c000-f123c7ff : ahci f123d000-f123d3ff : 0000:00:1d.0 f123d000-f123d3ff : ehci_hcd f123e000-f123efff : 0000:00:19.0 f123e000-f123efff : e1000e f8000000-fbffffff : PCI MMCONFIG 0000 [bus 00-3f] f80f8000-f80f8fff : Reserved fec00000-fec003ff : IOAPIC 0 fed00000-fed003ff : HPET 0 fed00000-fed003ff : PNP0103:00 fed10000-fed17fff : pnp 00:01 fed18000-fed18fff : pnp 00:01 fed19000-fed19fff : pnp 00:01 fed1c000-fed1ffff : Reserved fed1c000-fed1ffff : pnp 00:01 fed1f410-fed1f414 : iTCO_wdt.0.auto fed1f410-fed1f414 : iTCO_wdt.0.auto fed1f800-fed1f9ff : intel-spi fed1f800-fed1f9ff : intel-spi fed40000-fed4bfff : PCI Bus 0000:00 fed45000-fed4bfff : pnp 00:01 fed70000-fed70fff : MSFT0101:00 fed70000-fed70fff : MSFT0101:00 fed90000-fed90fff : dmar0 fed91000-fed91fff : dmar1 fee00000-fee00fff : Local APIC 100000000-12dffffff : System RAM 12e000000-12fffffff : RAM buffer /proc/config.gz (attached) ======================================================== TRACE 1: WARNING z3fold.c:428 + general protection fault ======================================================== ------------[ cut here ]------------ WARNING: CPU: 2 PID: 144 at mm/z3fold.c:428 handle_to_buddy.cold+0xc/0x13 Modules linked in: ccm bnep fuse snd_hda_codec_hdmi uvcvideo joydev mousedev rmi_smbus rmi_core videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common btusb videodev btrtl btbcm btintel bluetooth mc msr intel_rapl_msr intel_rapl_common ecdh_generic ecc x86_pkg_temp_thermal crc16 i915 intel_powerclamp coretemp kvm_intel lz4 lz4_compress iwlmvm i2c_algo_bit drm_kms_helper mac80211 kvm drm ofpart snd_hda_codec_realtek cmdlinepart libarc4 nls_iso8859_1 snd_hda_codec_generic intel_spi_platform nls_cp437 intel_spi vfat spi_nor snd_hda_intel wmi_bmof mei_wdt iwlwifi mei_hdcp irqbypass intel_gtt fat iTCO_wdt intel_cstate snd_hda_codec agpgart mtd syscopyarea tpm_crb intel_uncore iTCO_vendor_support snd_hda_core tpm_tis rtsx_pci_ms psmouse snd_hwdep tpm_tis_core thinkpad_acpi input_leds cfg80211 sysfillrect pcspkr intel_rapl_perf nvram sysimgblt memstick ledtrig_audio tpm snd_pcm wmi battery rfkill ac fb_sys_fops mei_me rng_core snd_timer evdev e1000e intel_pch_thermal mac_hid mei snd lpc_ich i2c_i801 soundcore crypto_user ip_tables x_tables btrfs libcrc32c crc32c_generic xor raid6_pq dm_crypt dm_mod sd_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel rtsx_pci_sdmmc serio_raw mmc_core atkbd libps2 aesni_intel ahci aes_x86_64 crypto_simd libahci xhci_pci cryptd glue_helper xhci_hcd libata scsi_mod ehci_pci ehci_hcd rtsx_pci i8042 serio CPU: 2 PID: 144 Comm: kswapd0 Not tainted 5.3.0-rc7-1-ARCH #1 Hardware name: LENOVO 20BXCTO1WW/20BXCTO1WW, BIOS JBET63WW (1.27 ) 11/10/2016 RIP: 0010:handle_to_buddy.cold+0xc/0x13 Code: 44 24 10 85 c0 0f 85 61 ff ff ff 4c 8b 44 24 18 4d 85 c0 0f 85 f8 fe ff ff e9 f7 fd ff ff 48 c7 c7 80 c7 ea 94 e8 42 c2 e5 ff <0f> 0b e9 6f de ff ff 48 c7 c7 80 c7 ea 94 e8 2f c2 e5 ff 0f 0b 48 RSP: 0018:ffffa7ab0023f830 EFLAGS: 00010246 RAX: 0000000000000024 RBX: ffff93a5fe157001 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff93a669a97708 RDI: 00000000ffffffff RBP: ffff93a5fe157000 R08: 0000000000000532 R09: 0000000000000004 R10: 0000000000000000 R11: 0000000000000001 R12: ffff93a5fe157001 R13: ffff93a5fe157010 R14: ffff93a64a505848 R15: ffff93a64a505840 FS: 0000000000000000(0000) GS:ffff93a669a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fb7ee08a928 CR3: 000000005b20a001 CR4: 00000000003606e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: z3fold_zpool_map+0x76/0x100 zswap_writeback_entry+0x5b/0x3b0 z3fold_zpool_shrink+0x28b/0x4a0 zswap_frontswap_store+0x12a/0x6a0 __frontswap_store+0xa6/0xf0 swap_writepage+0x39/0x70 pageout.isra.0+0x12b/0x360 shrink_page_list+0x74b/0xb80 shrink_inactive_list+0x24f/0x410 shrink_node_memcg+0x258/0x7b0 shrink_node+0xe8/0x4f0 balance_pgdat+0x2e3/0x530 kswapd+0x200/0x3f0 ? wait_woken+0x70/0x70 kthread+0xfb/0x130 ? balance_pgdat+0x530/0x530 ? kthread_park+0x80/0x80 ret_from_fork+0x35/0x40 ---[ end trace c799dc3361263fe2 ]--- general protection fault: 0000 [#1] PREEMPT SMP PTI CPU: 2 PID: 144 Comm: kswapd0 Tainted: G W 5.3.0-rc7-1-ARCH #1 Hardware name: LENOVO 20BXCTO1WW/20BXCTO1WW, BIOS JBET63WW (1.27 ) 11/10/2016 RIP: 0010:handle_to_buddy+0x20/0x30 Code: 8b 47 38 48 c1 e0 0c c3 66 90 0f 1f 44 00 00 53 48 89 fb 83 e7 01 0f 85 7e 21 00 00 48 8b 03 5b 48 89 c2 48 81 e2 00 f0 ff ff <0f> b6 52 52 29 d0 83 e0 03 c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 RSP: 0018:ffffa7ab0023f838 EFLAGS: 00010206 RAX: 00ffff93a5fe1570 RBX: ffffd09502f855c0 RCX: 0000000000000000 RDX: 00ffff93a5fe1000 RSI: ffff93a669a97708 RDI: 00000000ffffffff RBP: ffff93a5fe157000 R08: 0000000000000532 R09: 0000000000000004 R10: 0000000000000000 R11: 0000000000000001 R12: ffff93a5fe157001 R13: ffff93a5fe157010 R14: ffff93a64a505848 R15: ffff93a64a505840 FS: 0000000000000000(0000) GS:ffff93a669a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fb7ee08a928 CR3: 000000005b20a001 CR4: 00000000003606e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: z3fold_zpool_map+0x76/0x100 zswap_writeback_entry+0x5b/0x3b0 z3fold_zpool_shrink+0x28b/0x4a0 zswap_frontswap_store+0x12a/0x6a0 __frontswap_store+0xa6/0xf0 swap_writepage+0x39/0x70 pageout.isra.0+0x12b/0x360 shrink_page_list+0x74b/0xb80 shrink_inactive_list+0x24f/0x410 shrink_node_memcg+0x258/0x7b0 shrink_node+0xe8/0x4f0 balance_pgdat+0x2e3/0x530 kswapd+0x200/0x3f0 ? wait_woken+0x70/0x70 kthread+0xfb/0x130 ? balance_pgdat+0x530/0x530 ? kthread_park+0x80/0x80 ret_from_fork+0x35/0x40 Modules linked in: ccm bnep fuse snd_hda_codec_hdmi uvcvideo joydev mousedev rmi_smbus rmi_core videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common btusb videodev btrtl btbcm btintel bluetooth mc msr intel_rapl_msr intel_rapl_common ecdh_generic ecc x86_pkg_temp_thermal crc16 i915 intel_powerclamp coretemp kvm_intel lz4 lz4_compress iwlmvm i2c_algo_bit drm_kms_helper mac80211 kvm drm ofpart snd_hda_codec_realtek cmdlinepart libarc4 nls_iso8859_1 snd_hda_codec_generic intel_spi_platform nls_cp437 intel_spi vfat spi_nor snd_hda_intel wmi_bmof mei_wdt iwlwifi mei_hdcp irqbypass intel_gtt fat iTCO_wdt intel_cstate snd_hda_codec agpgart mtd syscopyarea tpm_crb intel_uncore iTCO_vendor_support snd_hda_core tpm_tis rtsx_pci_ms psmouse snd_hwdep tpm_tis_core thinkpad_acpi input_leds cfg80211 sysfillrect pcspkr intel_rapl_perf nvram sysimgblt memstick ledtrig_audio tpm snd_pcm wmi battery rfkill ac fb_sys_fops mei_me rng_core snd_timer evdev e1000e intel_pch_thermal mac_hid mei snd lpc_ich i2c_i801 soundcore crypto_user ip_tables x_tables btrfs libcrc32c crc32c_generic xor raid6_pq dm_crypt dm_mod sd_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel rtsx_pci_sdmmc serio_raw mmc_core atkbd libps2 aesni_intel ahci aes_x86_64 crypto_simd libahci xhci_pci cryptd glue_helper xhci_hcd libata scsi_mod ehci_pci ehci_hcd rtsx_pci i8042 serio ---[ end trace c799dc3361263fe3 ]--- RIP: 0010:handle_to_buddy+0x20/0x30 Code: 8b 47 38 48 c1 e0 0c c3 66 90 0f 1f 44 00 00 53 48 89 fb 83 e7 01 0f 85 7e 21 00 00 48 8b 03 5b 48 89 c2 48 81 e2 00 f0 ff ff <0f> b6 52 52 29 d0 83 e0 03 c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 RSP: 0018:ffffa7ab0023f838 EFLAGS: 00010206 RAX: 00ffff93a5fe1570 RBX: ffffd09502f855c0 RCX: 0000000000000000 RDX: 00ffff93a5fe1000 RSI: ffff93a669a97708 RDI: 00000000ffffffff RBP: ffff93a5fe157000 R08: 0000000000000532 R09: 0000000000000004 R10: 0000000000000000 R11: 0000000000000001 R12: ffff93a5fe157001 R13: ffff93a5fe157010 R14: ffff93a64a505848 R15: ffff93a64a505840 FS: 0000000000000000(0000) GS:ffff93a669a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fb7ee08a928 CR3: 000000005b20a001 CR4: 00000000003606e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 note: kswapd0[144] exited with preempt_count 1 ===================================== TRACE 2: z3fold_zpool_destroy blocked ===================================== INFO: task kworker/2:3:335 blocked for more than 122 seconds. Not tainted 5.3.0-rc7-1-ARCH #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/2:3 D 0 335 2 0x80004080 Workqueue: events __zswap_pool_release Call Trace: ? __schedule+0x27f/0x6d0 schedule+0x43/0xd0 z3fold_zpool_destroy+0xe9/0x130 ? wait_woken+0x70/0x70 zpool_destroy_pool+0x5c/0x90 __zswap_pool_release+0x6a/0xb0 process_one_work+0x1d1/0x3a0 worker_thread+0x4a/0x3d0 kthread+0xfb/0x130 ? process_one_work+0x3a0/0x3a0 ? kthread_park+0x80/0x80 ret_from_fork+0x35/0x40 ================================ TRACE 3: page allocation failure (probably unrelated) ================================ gnome-shell: page allocation failure: order:0, mode:0x400d0(__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_RECLAIMABLE), nodemask=(null),cpuset=/,mems_allowed=0 CPU: 2 PID: 1022 Comm: gnome-shell Not tainted 5.3.0-rc7-1-ARCH #1 Hardware name: LENOVO 20BXCTO1WW/20BXCTO1WW, BIOS JBET63WW (1.27 ) 11/10/2016 Call Trace: dump_stack+0x5c/0x80 warn_alloc.cold+0x78/0xf8 __alloc_pages_nodemask+0x107d/0x10b0 new_slab+0x29a/0xbe0 ___slab_alloc+0x44c/0x5d0 ? xas_nomem+0x49/0x70 ? xas_alloc+0x9b/0xc0 ? kmem_cache_alloc+0x16f/0x210 ? xas_nomem+0x49/0x70 __slab_alloc.isra.0+0x52/0x70 ? xas_nomem+0x49/0x70 kmem_cache_alloc+0x1e3/0x210 xas_nomem+0x49/0x70 add_to_swap_cache+0x264/0x320 __read_swap_cache_async+0x112/0x220 swap_cluster_readahead+0x1e2/0x320 shmem_swapin+0x74/0xc0 shmem_swapin_page+0x51c/0x770 ? find_get_entry+0x101/0x160 shmem_getpage_gfp.isra.0+0x3dd/0x8c0 shmem_read_mapping_page_gfp+0x48/0x80 shmem_get_pages+0x21d/0x5b0 [i915] __i915_gem_object_get_pages+0x54/0x60 [i915] __i915_vma_do_pin+0x294/0x450 [i915] eb_lookup_vmas+0x7ce/0xb10 [i915] i915_gem_do_execbuffer+0x60f/0x12f0 [i915] ? __alloc_pages_nodemask+0x1c4/0x10b0 i915_gem_execbuffer2_ioctl+0x1d3/0x3c0 [i915] ? put_swap_page+0x102/0x2e0 ? i915_gem_execbuffer_ioctl+0x2d0/0x2d0 [i915] drm_ioctl_kernel+0xb8/0x100 [drm] drm_ioctl+0x23d/0x3d0 [drm] ? i915_gem_execbuffer_ioctl+0x2d0/0x2d0 [i915] do_vfs_ioctl+0x43d/0x6c0 ? syscall_trace_enter+0x1f2/0x2e0 ksys_ioctl+0x5e/0x90 __x64_sys_ioctl+0x16/0x20 do_syscall_64+0x5f/0x1c0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f500ca5421b Code: 0f 1e fa 48 8b 05 75 8c 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 45 8c 0c 00 f7 d8 64 89 01 48 RSP: 002b:00007ffdeab46518 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00007ffdeab46560 RCX: 00007f500ca5421b RDX: 00007ffdeab46560 RSI: 0000000040406469 RDI: 000000000000000b RBP: 0000000040406469 R08: 000055ea80aa8cd0 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000055ea80aab240 R13: 000000000000000b R14: ffffffffffffffff R15: 00007f4ffe539730 Mem-Info: active_anon:505791 inactive_anon:137312 isolated_anon:53 active_file:7074 inactive_file:8072 isolated_file:32 unevictable:2125 dirty:56 writeback:0 unstable:0 slab_reclaimable:14348 slab_unreclaimable:31867 mapped:5631 shmem:10740 pagetables:5907 bounce:0 free:45505 free_pcp:908 free_cma:0 Node 0 active_anon:2023164kB inactive_anon:549248kB active_file:28296kB inactive_file:32288kB unevictable:8500kB isolated(anon):212kB isolated(file):128kB mapped:22524kB dirty:224kB writeback:0kB shmem:42960kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Node 0 DMA free:15456kB min:272kB low:340kB high:408kB active_anon:160kB inactive_anon:68kB active_file:0kB inactive_file:16kB unevictable:0kB writepending:0kB present:15912kB managed:15872kB mlocked:0kB kernel_stack:0kB pagetables:20kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 3137 3801 3801 3801 Node 0 DMA32 free:160740kB min:157952kB low:171840kB high:185728kB active_anon:1861456kB inactive_anon:523796kB active_file:27964kB inactive_file:32576kB unevictable:8240kB writepending:224kB present:3322892kB managed:3234336kB mlocked:48kB kernel_stack:7092kB pagetables:19796kB bounce:0kB free_pcp:3176kB local_pcp:120kB free_cma:0kB lowmem_reserve[]: 0 0 663 663 663 Node 0 Normal free:5824kB min:38188kB low:41124kB high:44060kB active_anon:162152kB inactive_anon:25216kB active_file:68kB inactive_file:276kB unevictable:260kB writepending:0kB present:753664kB managed:679748kB mlocked:16kB kernel_stack:3692kB pagetables:3812kB bounce:0kB free_pcp:296kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 17*4kB (UM) 7*8kB (UM) 1*16kB (U) 2*32kB (UM) 3*64kB (UM) 2*128kB (UM) 0*256kB 1*512kB (M) 2*1024kB (UM) 0*2048kB 3*4096kB (ME) = 15500kB Node 0 DMA32: 1155*4kB (UM) 8387*8kB (UM) 2986*16kB (UME) 1229*32kB (MEH) 0*64kB 0*128kB 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 160612kB Node 0 Normal: 1439*4kB (UMH) 20*8kB (UMH) 2*16kB (UH) 3*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6044kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 122450 total pagecache pages 96549 pages in swap cache Swap cache stats: add 14474738, delete 14376799, find 7531326/8981020 Free swap = 13216084kB Total swap = 15638612kB 1023117 pages RAM 0 pages HighMem/MovableOnly 40628 pages reserved 0 pages hwpoisoned SLUB: Unable to allocate memory on node -1, gfp=0xc0(__GFP_IO|__GFP_FS) cache: radix_tree_node, object size: 576, buffer size: 584, default order: 2, min order: 0 node 0: slabs: 410, objs: 11144, free: 63