linux-xfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Help deciding about backported patch (kernel bug 214767, 19f4e7cc8197 xfs: Fix CIL throttle hang when CIL space used going backwards)
@ 2022-02-17  9:22 Christian Theune
  2022-02-19 21:14 ` Dave Chinner
  0 siblings, 1 reply; 3+ messages in thread
From: Christian Theune @ 2022-02-17  9:22 UTC (permalink / raw)
  To: linux-xfs


[-- Attachment #1.1: Type: text/plain, Size: 1802 bytes --]

Hi,

I’ve been debugging an elusive XFS issue that I could not track down to any other parameters than it being an xfs internal bug. I’ve recorded what I’ve seen so far in https://bugzilla.kernel.org/show_bug.cgi?id=214767 and Dave recommended that "19f4e7cc8197 xfs: Fix CIL throttle hang when CIL space used going backwards” is likely the issue. AFAICT this was not backported to the 5.10 branch and we’ve been updating to vanilla kernels diligently and still keep seeing this issue. Unfortunately within a fleet of around 1k VMs it strikes about once every week or so and there’s no way to predict when and where.

So, I took Dave’s pointer and applied the patch to our 5.10 series (basd on 5.10.76 at that point) and it applied cleanly. The machine boots fine and I ran the XFS test suite. However, I haven’t done any tests using the test suite before and I’m getting a number of errors where I don’t know how to interpret the results. Some of those seem to be due to not having the DEBUG flag set in the kernel, others … I’m not sure.

So, before rolling out this change into our fleet I’d like to double check the results and also whether you think applying that patch to the 5.10 series sounds reasonable. I’m also wondering whether there was a specific reason this wasn’t backported in the first place and whether others think it should be. Other input about the applicability of this patch to the issue I’m seeing are appreciated as well, of course.

I’m attaching the test runner output, unfortunately I lost the actual outputs as the test ran quite long and the outputs where cleaned up by the tempfile watcher faster than I could retrieve them. I can run them again, my estimation currently is it takes around 3-4 days to complete them, though.


[-- Attachment #1.2: xfstests.out --]
[-- Type: application/octet-stream, Size: 105104 bytes --]

FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64 test38 5.10.76 #1-NixOS SMP Wed Oct 27 07:56:57 UTC 2021
MKFS_OPTIONS  -- -f /dev/vda4
MOUNT_OPTIONS -- /dev/vda4 /mnt/scratch

generic/001	 12s
generic/002	 4s
generic/003	 12s
generic/004	 3s
generic/005	 5s
generic/006	 3s
generic/007	 3s
generic/008	 3s
generic/009	 3s
generic/010	[not run] /tmp/xfstests.wVf4eh/src/dbtest not built
generic/011	 5s
generic/012	 4s
generic/013	 20s
generic/014	 7s
generic/015	 3s
generic/016	 4s
generic/017	 75s
generic/018	 4s
generic/019	[not run] /sys/kernel/debug/fail_make_request  not found. Seems that CONFIG_FAULT_INJECTION_DEBUG_FS kernel config option not enabled
generic/020	 7s
generic/021	 4s
generic/022	 3s
generic/023	 3s
generic/024	 3s
generic/025	 4s
generic/026	 8s
generic/027	 50s
generic/028	 8s
generic/029	 2s
generic/030	 3s
generic/031	 3s
generic/032	 14s
generic/033	 2s
generic/034	 3s
generic/035	no qualified output
generic/036	 12s
generic/037	 9s
generic/038	 181s
generic/039	 2s
generic/040	 20s
generic/041	 25s
generic/042	 4s
generic/043	 17s
generic/044	 25s
generic/045	 25s
generic/046	 24s
generic/047	 50s
generic/048	 33s
generic/049	 26s
generic/050	no qualified output
generic/051	 83s
generic/052	 3s
generic/053	 3s
generic/054	 56s
generic/055	 35s
generic/056	 3s
generic/057	 3s
generic/058	 2s
generic/059	 4s
generic/060	 3s
generic/061	 2s
generic/062	 4s
generic/063	 3s
generic/064	 4s
generic/065	 3s
generic/066	 3s
generic/067	 3s
generic/068	 47s
generic/069	 11s
generic/070	 6s
generic/071	 3s
generic/072	 21s
generic/073	 3s
generic/074	 44s
generic/075	 39s
generic/076	 6s
generic/077	[not run] fsgqa user not defined.
generic/078	 3s
generic/079	[not run] daemon user not defined.
generic/080	 4s
generic/081	 5s
generic/082	 3s
generic/083	 13s
generic/084	 8s
generic/085	 13s
generic/086	 3s
generic/087	 3s
generic/088	 2s
generic/089	 12s
generic/090	 3s
generic/091	 61s
generic/092	 3s
generic/093	[not run] fsgqa user not defined.
generic/094	 94s
generic/095	 43s
generic/096	 2s
generic/097	 2s
generic/098	 3s
generic/099	 3s
generic/100	 44s
generic/101	 3s
generic/102	 10s
generic/103	 5s
generic/104	 5s
generic/105	 4s
generic/106	 5s
generic/107	 4s
generic/108	 7s
generic/109	 6s
generic/110	 8s
generic/111	 3s
generic/112	 36s
generic/113	 35s
generic/114	 45s
generic/115	 2s
generic/116	 3s
generic/117	 4s
generic/118	 3s
generic/119	 3s
generic/120	 18s
generic/121	 3s
generic/122	 3s
generic/123	[not run] fsgqa user not defined.
generic/124	 6s
generic/125	[not run] fsgqa user not defined.
generic/126	 2s
generic/127	 656s
generic/128	[not run] fsgqa user not defined.
generic/129	 5s
generic/130	 13s
generic/131	 3s
generic/132	 18s
generic/133	 204s
generic/134	 2s
generic/135	 3s
generic/136	 3s
generic/137	 29s
generic/138	 3s
generic/139	 3s
generic/140	 3s
generic/141	 3s
generic/142	 3s
generic/143	 89s
generic/144	 12s
generic/145	 7s
generic/146	 11s
generic/147	 11s
generic/148	 15s
generic/149	 15s
generic/150	 5s
generic/151	 12s
generic/152	 18s
generic/153	 16s
generic/154	 22s
generic/155	 25s
generic/156	 25s
generic/157	 6s
generic/158	 7s
generic/159	 7s
generic/160	 7s
generic/161	 4s
generic/162	 24s
generic/163	 30s
generic/164	 15s
generic/165	 22s
generic/166	 64s
generic/167	 17s
generic/168	 79s
generic/169	 7s
generic/170	 97s
generic/171	 6s
generic/172	 10s
generic/173	 7s
generic/174	 12s
generic/175	 41s
generic/176	 77s
generic/177	 5s
generic/178	 7s
generic/179	 6s
generic/180	 7s
generic/181	 7s
generic/182	 9s
generic/183	 12s
generic/184	 6s
generic/185	 5s
generic/186	 38s
generic/187	 38s
generic/188	 4s
generic/189	 5s
generic/190	 6s
generic/191	 5s
generic/192	 11s
generic/193	[not run] fsgqa user not defined.
generic/194	 4s
generic/195	 5s
generic/196	 5s
generic/197	 5s
generic/198	 8s
generic/199	 5s
generic/200	 7s
generic/201	 4s
generic/202	 4s
generic/203	 5s
generic/204	 46s
generic/205	 5s
generic/206	 6s
generic/207	 15s
generic/208	 205s
generic/209	 39s
generic/210	 5s
generic/211	 3s
generic/212	 5s
generic/213	 3s
generic/214	 4s
generic/215	 5s
generic/216	 3s
generic/217	 4s
generic/218	 3s
generic/219	[not run] fsgqa user not defined.
generic/220	 3s
generic/221	 5s
generic/222	 8s
generic/223	 12s
generic/224	 15s
generic/225	 137s
generic/226	 17s
generic/227	 4s
generic/228	 5s
generic/229	 9s
generic/230	[not run] fsgqa user not defined.
generic/231	[not run] fsgqa user not defined.
generic/232	 16s
generic/233	[not run] fsgqa user not defined.
generic/234	 12s
generic/235	[not run] fsgqa user not defined.
generic/236	 4s
generic/237	 5s
generic/238	 9s
generic/239	 19s
generic/240	 3s
generic/241	[not run] dbench not found
generic/242	 18s
generic/243	 19s
generic/244	 12s
generic/245	 7s
generic/246	 3s
generic/247	 15s
generic/248	 7s
generic/249	 5s
generic/250	 6s
generic/251	 106s
generic/252	 6s
generic/253	 4s
generic/254	 4s
generic/255	 4s
generic/256	[not run] fsgqa user not defined.
generic/257	 6s
generic/258	 6s
generic/259	 4s
generic/260	 12s
generic/261	 6s
generic/262	 6s
generic/263	 31s
generic/264	 3s
generic/265	 5s
generic/266	 5s
generic/267	 4s
generic/268	 5s
generic/269	 37s
generic/270	[not run] fsgqa user not defined.
generic/271	 4s
generic/272	 5s
generic/273	 14s
generic/274	 15s
generic/275	 5s
generic/276	 5s
generic/277	 4s
generic/278	 5s
generic/279	 5s
generic/280	 4s
generic/281	 5s
generic/282	 5s
generic/283	 15s
generic/284	 4s
generic/285	 2s
generic/286	 5s
generic/287	 4s
generic/288	 3s
generic/289	 5s
generic/290	 4s
generic/291	 4s
generic/292	 5s
generic/293	 5s
generic/294	 3s
generic/295	 5s
generic/296	 4s
generic/297	 41s
generic/298	 38s
generic/299	 113s
generic/300	 26s
generic/301	 7s
generic/302	 49s
generic/303	 4s
generic/304	 2s
generic/305	[not run] fsgqa user not defined.
generic/306	 4s
generic/307	 5s
generic/308	 3s
generic/309	 3s
generic/310	 83s
generic/311	 189s
generic/312	 13s
generic/313	 12s
generic/314	[not run] fsgqa user not defined.
generic/315	 9s
generic/316	 4s
generic/317	[not run] fsgqa user not defined.
generic/318	 3s
generic/319	 6s
generic/320	 20s
generic/321	 5s
generic/322	 5s
generic/323	 124s
generic/324	 23s
generic/325	 8s
generic/326	[not run] fsgqa user not defined.
generic/327	[not run] fsgqa user not defined.
generic/328	[not run] fsgqa user not defined.
generic/329	 6s
generic/330	 9s
generic/331	 10s
generic/332	 7s
generic/333	 270s
generic/334	 104s
generic/335	 5s
generic/336	 5s
generic/337	 3s
generic/338	 8s
generic/339	 15s
generic/340	 9s
generic/341	 8s
generic/342	 8s
generic/343	 10s
generic/344	 10s
generic/345	 11s
generic/346	 8s
generic/347	 85s
generic/348	 7s
generic/349	 7s
generic/350	 4s
generic/351	 7s
generic/352	 39s
generic/353	 4s
generic/354	 7s
generic/355	[not run] fsgqa user not defined.
generic/356	 4s
generic/357	 5s
generic/358	 16s
generic/359	 5s
generic/360	 7s
generic/361	 5s
generic/362	[not run] mkfs.xfs doesn't have richacl feature
generic/363	[not run] mkfs.xfs doesn't have richacl feature
generic/364	[not run] mkfs.xfs doesn't have richacl feature
generic/365	[not run] mkfs.xfs doesn't have richacl feature
generic/366	[not run] mkfs.xfs doesn't have richacl feature
generic/367	[not run] mkfs.xfs doesn't have richacl feature
generic/368	[not run] mkfs.xfs doesn't have richacl feature
generic/369	[not run] mkfs.xfs doesn't have richacl feature
generic/370	[not run] mkfs.xfs doesn't have richacl feature
generic/371	 30s
generic/372	 7s
generic/373	 6s
generic/374	 6s
generic/375	 10s
generic/376	 3s
generic/377	 8s
generic/378	[not run] fsgqa user not defined.
generic/379	 5s
generic/380	 17s
generic/381	[not run] 123456-fsgqa user not defined.
generic/382	[not run] fsgqa user not defined.
generic/383	 3s
generic/384	 13s
generic/385	 8s
generic/386	 7s
generic/387	 78s
generic/388	 211s
generic/389	 12s
generic/390	 11s
generic/391	 21s
generic/392	 8s
generic/393	 5s
generic/394	 10s
generic/395	[not run] No encryption support for xfs
generic/396	[not run] No encryption support for xfs
generic/397	[not run] No encryption support for xfs
generic/398	[not run] No encryption support for xfs
generic/399	[not run] No encryption support for xfs
generic/400	 3s
generic/401	 6s
generic/402	 8s
generic/403	 8s
generic/404	 34s
generic/405	 63s
generic/406	 8s
generic/407	 8s
generic/408	 8s
generic/409	 17s
generic/410	 28s
generic/411	 8s
generic/412	 4s
generic/413	[not run] mount /dev/vda4 with dax failed
generic/414	 5s
generic/415	 88s
generic/416	 276s
generic/417	 23s
generic/418	 68s
generic/419	[not run] No encryption support for xfs
generic/420	 3s
generic/421	[not run] No encryption support for xfs
generic/422	 3s
generic/423	 4s
generic/424	 2s
generic/425	 5s
generic/426	 10s
generic/427	 12s
generic/428	 12s
generic/429	[not run] No encryption support for xfs
generic/430	 4s
generic/431	 8s
generic/432	 9s
generic/433	 7s
generic/434	 7s
generic/435	[not run] No encryption support for xfs
generic/436	 7s
generic/437	 7s
generic/438	 638s
generic/439	 3s
generic/440	[not run] No encryption support for xfs
generic/441	 4s
generic/442	 4s
generic/443	 3s
generic/444	 3s
generic/445	 2s
generic/446	 19s
generic/447	 56s
generic/448	 3s
generic/449	 63s
generic/450	 3s
generic/451	 33s
generic/452	- output mismatch (see /home/ctheune/results/generic/452.out.bad)
    --- tests/generic/452.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/generic/452.out.bad	2022-02-01 08:23:43.356951460 +0100
    @@ -1,3 +1,5 @@
     QA output created by 452
    -SCRATCH_MNT/ls_on_scratch
    -SCRATCH_MNT/ls_on_scratch
    +/mnt/scratch/ls_on_scratch: unknown program 'ls_on_scratch'
    +Try '/mnt/scratch/ls_on_scratch --help' for more information.
    +/mnt/scratch/ls_on_scratch: unknown program 'ls_on_scratch'
    +Try '/mnt/scratch/ls_on_scratch --help' for more information.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/generic/452.out /home/ctheune/results/generic/452.out.bad'  to see the entire diff)
generic/453	 4s
generic/454	 5s
generic/455	[not run] This test requires a valid $LOGWRITES_DEV
generic/456	 4s
generic/457	[not run] This test requires a valid $LOGWRITES_DEV
generic/458	 5s
generic/459	[not run] thin_check utility required, skipped this test
generic/460	 4s
generic/461	 25s
generic/462	[not run] mount /dev/vda4 with dax failed
generic/463	 3s
generic/464	 62s
generic/465	 22s
generic/466	 7s
generic/467	 6s
generic/468	 7s
generic/469	 4s
generic/470	[not run] This test requires a valid $LOGWRITES_DEV
generic/471	 4s
generic/472	 4s
generic/473	- output mismatch (see /home/ctheune/results/generic/473.out.bad)
    --- tests/generic/473.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/generic/473.out.bad	2022-02-01 08:26:46.000979754 +0100
    @@ -6,7 +6,7 @@
     1: [256..287]: hole
     Hole + Data
     0: [0..127]: hole
    -1: [128..255]: data
    +1: [128..135]: data
     Hole + Data + Hole
     0: [0..127]: hole
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/generic/473.out /home/ctheune/results/generic/473.out.bad'  to see the entire diff)
generic/474	 3s
generic/475	 170s
generic/476	 387s
generic/477	 6s
generic/478	 23s
generic/479	 4s
generic/480	 4s
generic/481	 4s
generic/482	[not run] This test requires a valid $LOGWRITES_DEV
generic/483	 9s
generic/484	 4s
generic/485	 5s
generic/486	 3s
generic/487	[not run] This test requires a valid $SCRATCH_LOGDEV
generic/488	 3s
generic/489	 3s
generic/490	 3s
generic/491	 3s
generic/492	 3s
generic/493	 5s
generic/494	 5s
generic/495	 4s
generic/496	 5s
generic/497	 4s
generic/498	 3s
generic/499	 4s
generic/500	 10s
generic/501	 37s
generic/502	 5s
generic/503	 9s
generic/504	 3s
generic/505	 4s
generic/506	 5s
generic/507	 7s
generic/508	 5s
generic/509	 3s
generic/510	 5s
generic/511	 4s
generic/512	 4s
generic/513	 5s
generic/514	[not run] fsgqa user not defined.
generic/515	 4s
generic/516	 4s
generic/517	 4s
generic/518	 4s
generic/519	 4s
generic/520	 36s
generic/521	 1286s
generic/522	 1699s
generic/523	 3s
generic/524	 8s
generic/525	 5s
generic/526	 5s
generic/527	 3s
generic/528	 4s
generic/529	 3s
generic/530	 14s
generic/531	 14s
generic/532	 4s
generic/533	 6s
generic/534	 3s
generic/535	 5s
generic/536	 6s
generic/537	 4s
generic/538	 6s
generic/539	 4s
generic/540	 5s
generic/541	 6s
generic/542	 6s
generic/543	 7s
generic/544	 7s
generic/545	 6s
generic/546	 5s
generic/547	 9s
generic/548	[not run] No encryption support for xfs
generic/549	[not run] No encryption support for xfs
generic/550	[not run] No encryption support for xfs
generic/551	 206s
generic/552	 5s
generic/553	 8s
generic/554	 4s
generic/555	 7s
generic/556	[not run] xfs does not support casefold feature
generic/557	 8s
generic/558	 75s
generic/559	[not run] duperemove utility required, skipped this test
generic/560	[not run] duperemove utility required, skipped this test
generic/561	[not run] duperemove utility required, skipped this test
generic/562	 169s
generic/563	 53s
generic/564	 5s
generic/565	 6s
generic/566	[not run] fsgqa user not defined.
generic/567	 4s
generic/568	 4s
generic/569	 5s
generic/570	[not run] userspace hibernation to swap is enabled
generic/571	 11s
generic/572	[not run] fsverity utility required, skipped this test
generic/573	[not run] fsverity utility required, skipped this test
generic/574	[not run] fsverity utility required, skipped this test
generic/575	[not run] fsverity utility required, skipped this test
generic/576	[not run] fsverity utility required, skipped this test
generic/577	[not run] fsverity utility required, skipped this test
generic/578	 4s
generic/579	[not run] fsverity utility required, skipped this test
generic/580	[not run] No encryption support for xfs
generic/581	[not run] fsgqa user not defined.
generic/582	[not run] No encryption support for xfs
generic/583	[not run] No encryption support for xfs
generic/584	[not run] No encryption support for xfs
generic/585	 4s
generic/586	 6s
generic/587	[not run] fsgqa user not defined.
generic/588	 4s
generic/589	 20s
generic/590	 26s
generic/591	 7s
generic/592	[not run] No encryption support for xfs
generic/593	[not run] No encryption support for xfs
generic/594	- output mismatch (see /home/ctheune/results/generic/594.out.bad)
    --- tests/generic/594.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/generic/594.out.bad	2022-02-01 09:44:44.916917137 +0100
    @@ -5,31 +5,31 @@
     *** Report for group quotas on device SCRATCH_DEV
     Block grace time: DEF_TIME; Inode grace time: DEF_TIME
     *** Report for project quotas on device SCRATCH_DEV
    -Block grace time: 00:10; Inode grace time: 00:20
    +Block grace time: DEF_TIME; Inode grace time: DEF_TIME
     
     2. set group quota timer
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/generic/594.out /home/ctheune/results/generic/594.out.bad'  to see the entire diff)
generic/595	[not run] No encryption support for xfs
generic/596	[not run] accton utility required, skipped this test
generic/597	[not run] fsgqa2 user not defined.
generic/598	[not run] fsgqa2 user not defined.
generic/599	 3s
generic/600	[not run] fsgqa user not defined.
generic/601	[not run] fsgqa user not defined.
generic/602	[not run] No encryption support for xfs
generic/603	[not run] fsgqa user not defined.
generic/604	 5s
generic/605	[not run] mount /dev/vda4 with dax=always failed
generic/606	[not run] mount /dev/vda4 with dax=always failed
generic/607	 5s
generic/608	[not run] mount /dev/vda4 with dax=always failed
generic/609	 5s
generic/610	 40s
generic/611	 6s
generic/612	 8s
generic/613	[not run] No encryption support for xfs
generic/614	 3s
generic/615	 43s
generic/616	[not run] kernel does not support IO_URING
generic/617	[not run] kernel does not support IO_URING
generic/618	 3s
generic/619	 20s
generic/620	 6s
generic/621	[not run] No encryption support for xfs
generic/622	 31s
generic/623	_check_dmesg: something found in dmesg (see /home/ctheune/results/generic/623.dmesg)

generic/624	[not run] fsverity utility required, skipped this test
generic/625	[not run] fsverity utility required, skipped this test
generic/626	 7s
generic/627	 24s
generic/628	 4s
generic/629	 5s
generic/630	 36s
generic/631	 33s
generic/632	 10s
generic/633	[not run] kernel does not support mount_setattr syscall
generic/634	 3s
generic/635	 6s
generic/636	 6s
generic/637	 9s
generic/638	 5s
generic/639	 7s
generic/640	 3s
generic/641	 6s
generic/642	 37s
generic/643	 6s
generic/644	[not run] kernel does not support mount_setattr syscall
generic/645	[not run] kernel does not support mount_setattr syscall
generic/646	_check_xfs_filesystem: filesystem on /dev/vda4 is inconsistent (r)
(see /home/ctheune/results/generic/646.full for details)

generic/647	 5s
generic/648	 124s
generic/649	 4s
generic/650	 74s
generic/651	 4s
generic/652	 4s
generic/653	 4s
generic/654	 5s
generic/655	 9s
generic/656	[not run] kernel does not support mount_setattr syscall
generic/657	 5s
generic/658	 5s
generic/659	 5s
generic/660	 5s
generic/661	 6s
generic/662	 5s
generic/663	 5s
generic/664	 5s
generic/665	 6s
generic/666	 5s
generic/667	 6s
generic/668	 5s
generic/669	 6s
generic/670	 13s
generic/671	 18s
generic/672	 88s
shared/002	 25s
shared/032	 4s
shared/298	 4s
xfs/001	 9s
xfs/002	 18s
xfs/003	 34s
xfs/004	 3s
xfs/005	 3s
xfs/006	 9s
xfs/007	 6s
xfs/008	 5s
xfs/009	 4s
xfs/010	 9s
xfs/011	 19s
xfs/012	 5s
xfs/013	 241s
xfs/014	[not run] fsgqa user not defined.
xfs/015	 23s
xfs/016	 122s
xfs/017	 9s
xfs/018	- output mismatch (see /home/ctheune/results/xfs/018.out.bad)
    --- tests/xfs/018.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/018.out.bad	2022-02-01 10:08:44.814221935 +0100
    @@ -1,17 +1,11 @@
     QA output created by 018
     *** init FS
    -*** compare logprint: 018.op with 018.fulldir/op.mnt-onoalign,logbsize=32k.mkfs-lsize=2000b-llazy-count=1-lversion=1.filtered
    -*** compare logprint: 018.trans_inode with 018.fulldir/trans_inode.mnt-onoalign,logbsize=32k.mkfs-lsize=2000b-llazy-count=1-lversion=1.filtered
    -*** compare logprint: 018.trans_buf with 018.fulldir/trans_buf.mnt-onoalign,logbsize=32k.mkfs-lsize=2000b-llazy-count=1-lversion=1.filtered
    -*** compare logprint: 018.op with 018.fulldir/op.mnt-onoalign,logbsize=32k.mkfs-lsize=2000b-llazy-count=1-lversion=2.filtered
    -*** compare logprint: 018.trans_inode with 018.fulldir/trans_inode.mnt-onoalign,logbsize=32k.mkfs-lsize=2000b-llazy-count=1-lversion=2.filtered
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/018.out /home/ctheune/results/xfs/018.out.bad'  to see the entire diff)
xfs/019	 8s
xfs/020	 10s
xfs/021	 3s
xfs/022	[not run] xfsdump not found
xfs/023	[not run] xfsdump not found
xfs/024	[not run] xfsdump not found
xfs/025	[not run] xfsdump not found
xfs/026	[not run] xfsdump not found
xfs/027	[not run] xfsdump not found
xfs/028	[not run] xfsdump not found
xfs/029	 3s
xfs/030	 12s
xfs/031	_check_xfs_filesystem: filesystem on /dev/vda4 is inconsistent (r)
(see /home/ctheune/results/xfs/031.full for details)
- output mismatch (see /home/ctheune/results/xfs/031.out.bad)
    --- tests/xfs/031.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/031.out.bad	2022-02-01 10:09:47.472542659 +0100
    @@ -38,78 +38,12 @@
     done
     
     === twenty entries (block form)
    +mkfs.xfs: cannot open true: No such file or directory
     Repairing, iteration 1
    -Phase 1 - find and verify superblock...
    -Phase 2 - using <TYPEOF> log
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/031.out /home/ctheune/results/xfs/031.out.bad'  to see the entire diff)
xfs/032	 26s
xfs/033	no qualified output
xfs/034	 4s
xfs/035	[not run] xfsdump not found
xfs/036	[not run] xfsdump not found
xfs/037	[not run] xfsdump not found
xfs/038	[not run] xfsdump not found
xfs/039	[not run] xfsdump not found
xfs/040	[not run] Can't run libxfs-diff without KWORKAREA set
xfs/041	 49s
xfs/042	 23s
xfs/043	[not run] xfsdump not found
xfs/044	[not run] This test requires a valid $SCRATCH_LOGDEV
xfs/045	 5s
xfs/046	[not run] xfsdump not found
xfs/047	[not run] xfsdump not found
xfs/048	 3s
xfs/049	 7s
xfs/050	 18s
xfs/051	[not run] sysfs attribute 'debug/log_recovery_delay' is not supported
xfs/052	 4s
xfs/053	 5s
xfs/054	 3s
xfs/055	[not run] xfsdump not found
xfs/056	[not run] xfsdump not found
xfs/057	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/058	 4s
xfs/059	[not run] xfsdump not found
xfs/060	[not run] xfsdump not found
xfs/061	[not run] xfsdump not found
xfs/062	 19s
xfs/063	[not run] xfsdump not found
xfs/064	[not run] xfsdump not found
xfs/065	[not run] xfsdump not found
xfs/066	[not run] xfsdump not found
xfs/067	 4s
xfs/068	[not run] xfsdump not found
xfs/069	 7s
xfs/070	 4s
xfs/071	no qualified output
xfs/072	 4s
xfs/073	 19s
xfs/074	 4s
xfs/075	 4s
xfs/076	 112s
xfs/077	 12s
xfs/078	 13s
xfs/079	 18s
xfs/080	 7s
xfs/081	[failed, exit status 1]- output mismatch (see /home/ctheune/results/xfs/081.out.bad)
    --- tests/xfs/081.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/081.out.bad	2022-02-01 10:17:14.675684724 +0100
    @@ -1,4 +1,10 @@
     QA output created by 081
     *** init FS
    -*** compare logprint: 081.ugquota.trans_inode with 081.fulldir/trans_inode.mnt-oquota,gquota.mkfs-lsize=2000b-llazy-count=1-lversion=1.filtered
    +
    +*** Cannot mkfs for this test using option specified: -l size=2000b -l lazy-count=1 -l version=1 ***
    +
    +*** compare logprint: 081.ugquota.trans_inode with /home/ctheune/results/xfs/081.fulldir/trans_inode.mnt-oquota,gquota.mkfs-lsize=2000b-llazy-count=1-lversion=1.filtered
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/081.out /home/ctheune/results/xfs/081.out.bad'  to see the entire diff)
xfs/082	- output mismatch (see /home/ctheune/results/xfs/082.out.bad)
    --- tests/xfs/082.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/082.out.bad	2022-02-01 10:17:21.672608619 +0100
    @@ -1,39 +1,14 @@
     QA output created by 082
     *** init FS
     --- mkfs=version=2, mnt=logbsize=32k, sync=sync ---
    -*** compare logprint: 082.trans_inode with 082.fulldir/trans_inode.mnt-ologbsize=32k.mkfs-lsize=2000b-llazy-count=1-lversion=2.sync.filtered
    -*** compare logprint: 082.trans_buf with 082.fulldir/trans_buf.mnt-ologbsize=32k.mkfs-lsize=2000b-llazy-count=1-lversion=2.sync.filtered
    ---- mkfs=version=2,su=4096, mnt=logbsize=32k, sync=sync ---
    -*** compare logprint: 082.trans_inode with 082.fulldir/trans_inode.mnt-ologbsize=32k.mkfs-lsize=2000b-llazy-count=1-lversion=2,su=4096.sync.filtered
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/082.out /home/ctheune/results/xfs/082.out.bad'  to see the entire diff)
xfs/083	 3167s
xfs/084	 64s
xfs/085	 10s
xfs/086	 10s
xfs/087	 12s
xfs/088	 12s
xfs/089	 13s
xfs/090	[not run] External volumes not in use, skipped this test
xfs/091	 12s
xfs/092	 4s
xfs/093	 11s
xfs/094	[not run] External volumes not in use, skipped this test
xfs/095	 4s
xfs/096	[not run] Requires older mkfs without strict input checks: the last supported version of xfsprogs is 4.5.
xfs/097	 11s
xfs/098	 14s
xfs/099	 6s
xfs/100	 8s
xfs/101	 8s
xfs/102	 25s
xfs/103	 4s
xfs/104	 24s
xfs/105	 25s
xfs/106	[not run] fsgqa user not defined.
xfs/108	 6s
xfs/109	 47s
xfs/110	 7s
xfs/111	 8s
xfs/112	 22s
xfs/113	 130s
xfs/114	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/115	 4s
xfs/116	 4s
xfs/117	 14s
xfs/118	 28s
xfs/119	 11s
xfs/120	 6s
xfs/121	 10s
xfs/122	[not run] indent utility required, skipped this test
xfs/123	 4s
xfs/124	 7s
xfs/125	 11s
xfs/126	 10s
xfs/127	 7s
xfs/128	 25s
xfs/129	 10s
xfs/130	 8s
xfs/131	[not run] External volumes not in use, skipped this test
xfs/132	 3s
xfs/133	 5s
xfs/134	 4s
xfs/135	 8s
xfs/136	- output mismatch (see /home/ctheune/results/xfs/136.out.bad)
    --- tests/xfs/136.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/136.out.bad	2022-02-01 12:19:39.365860287 +0100
    @@ -1,5 +1,5 @@
     QA output created by 136
    -inum=67
    +inum=131
     core.format = 2 (extents)
     core.size = 0
     core.extsize = 0
    @@ -7,6 +7,7 @@
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/136.out /home/ctheune/results/xfs/136.out.bad'  to see the entire diff)
xfs/137	 11s
xfs/138	 5s
xfs/139	 21s
xfs/140	 103s
xfs/141	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/142	[not run] External volumes not in use, skipped this test
xfs/143	[not run] External volumes not in use, skipped this test
xfs/144	 8s
xfs/145	 4s
xfs/146	[not run] External volumes not in use, skipped this test
xfs/147	[not run] External volumes not in use, skipped this test
xfs/148	 4s
xfs/149	 2s
xfs/150	 5s
xfs/151	 7s
xfs/152	[not run] kernel does not support mount_setattr syscall
xfs/153	[not run] fsgqa user not defined.
xfs/154	- output mismatch (see /home/ctheune/results/xfs/154.out.bad)
    --- tests/xfs/154.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/154.out.bad	2022-02-01 12:23:24.895399427 +0100
    @@ -1,4 +1,4 @@
     QA output created by 154
     FEATURES: NEEDSREPAIR:YES
    -mount: SCRATCH_MNT: mount(2) system call failed: Structure needs cleaning.
    +mount: SCRATCH_MNT: wrong fs type, bad option, bad superblock on SCRATCH_DEV, missing codepage or helper program, or other error.
     FEATURES: NEEDSREPAIR:NO
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/154.out /home/ctheune/results/xfs/154.out.bad'  to see the entire diff)
xfs/155	 22s
xfs/156	 8s
xfs/157	 7s
xfs/158	- output mismatch (see /home/ctheune/results/xfs/158.out.bad)
    --- tests/xfs/158.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/158.out.bad	2022-02-01 12:24:12.782877275 +0100
    @@ -8,7 +8,7 @@
     Fail partway through upgrading
     Adding inode btree counts to filesystem.
     FEATURES: NEEDSREPAIR:YES INOBTCNT:YES
    -mount: SCRATCH_MNT: mount(2) system call failed: Structure needs cleaning.
    +mount: SCRATCH_MNT: wrong fs type, bad option, bad superblock on SCRATCH_DEV, missing codepage or helper program, or other error.
     Re-run repair to finish upgrade
     FEATURES: NEEDSREPAIR:NO INOBTCNT:YES
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/158.out /home/ctheune/results/xfs/158.out.bad'  to see the entire diff)
xfs/159	 6s
xfs/160	 7s
xfs/161	[not run] quota: bigtime support not detected
xfs/162	 55s
xfs/163	[not run] xfsprogs does not support shrinking
xfs/164	 3s
xfs/165	 2s
xfs/166	 4s
xfs/167	 149s
xfs/168	[not run] xfsprogs does not support shrinking
xfs/169	 40s
xfs/170	 9s
xfs/171	 73s
xfs/172	[failed, exit status 1]- output mismatch (see /home/ctheune/results/xfs/172.out.bad)
    --- tests/xfs/172.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/172.out.bad	2022-02-01 12:30:25.762813308 +0100
    @@ -3,9 +3,5 @@
     # streaming
     # sync AGs...
     # checking stream AGs...
    -+ expected failure, matching AGs
    -# testing 64 16 20 10 1 0 1 ....
    -# streaming
    -# sync AGs...
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/172.out /home/ctheune/results/xfs/172.out.bad'  to see the entire diff)
xfs/173	 47s
xfs/174	 19s
xfs/175	 5s
xfs/176	[not run] xfsprogs does not support shrinking
xfs/177	- output mismatch (see /home/ctheune/results/xfs/177.out.bad)
    --- tests/xfs/177.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/177.out.bad	2022-02-01 12:32:04.098742455 +0100
    @@ -3,13 +3,16 @@
     inodes after creating files is in range
     Round 1
     inodes after bulkstat is in range
    -inodes after expire is in range
    +inodes after expire has value of 1028
    +inodes after expire is NOT in range -2 .. 8
     Round 2
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/177.out /home/ctheune/results/xfs/177.out.bad'  to see the entire diff)
xfs/178	 5s
xfs/179	 6s
xfs/180	 6s
xfs/181	 17s
xfs/182	 16s
xfs/183	 5s
xfs/184	 6s
xfs/185	- output mismatch (see /home/ctheune/results/xfs/185.out.bad)
    --- tests/xfs/185.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/185.out.bad	2022-02-01 12:33:12.312999728 +0100
    @@ -1,2 +1,4 @@
     QA output created by 185
    +No fsmap record for rtdev 7:1 above sector 204808
    +No fsmap record for rtdev 7:1 ending at sector 409599
     Silence is golden
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/185.out /home/ctheune/results/xfs/185.out.bad'  to see the entire diff)
xfs/186	[not run] attr v1 not supported on /dev/vda4
xfs/187	[not run] External volumes not in use, skipped this test
xfs/188	 8s
xfs/189	[not run] noattr2 mount option not supported on /dev/vda4
xfs/190	 3s
xfs/191-input-validation	 7s
xfs/192	 11s
xfs/193	 4s
xfs/194	 5s
xfs/195	[not run] fsgqa user not defined.
xfs/196	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/197	[not run] This test is only valid on 32 bit machines
xfs/198	 10s
xfs/199	 5s
xfs/200	 5s
xfs/201	 14s
xfs/202	 4s
xfs/203	 4s
xfs/204	 11s
xfs/205	 8s
xfs/206	 3s
xfs/207	 4s
xfs/208	 7s
xfs/209	 5s
xfs/210	 5s
xfs/211	 2430s
xfs/212	 5s
xfs/213	[not run] fsgqa user not defined.
xfs/214	[not run] fsgqa user not defined.
xfs/215	 5s
xfs/216	 6s
xfs/217	 5s
xfs/218	 5s
xfs/219	 6s
xfs/220	 4s
xfs/221	 6s
xfs/222	 5s
xfs/223	 5s
xfs/224	 10s
xfs/225	 8s
xfs/226	 7s
xfs/227	 459s
xfs/228	 7s
xfs/229	 38s
xfs/230	 5s
xfs/231	 9s
xfs/232	 9s
xfs/233	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/234	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/235	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/236	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/237	 7s
xfs/238	 6s
xfs/239	 7s
xfs/240	 9s
xfs/241	 11s
xfs/242	 9s
xfs/243	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/244	[not run] 16 bit project IDs not supported on /dev/vda4
xfs/245	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/246	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/247	 4s
xfs/248	 7s
xfs/249	 6s
xfs/250	 23s
xfs/251	 4s
xfs/252	 10s
xfs/253	 5s
xfs/254	 6s
xfs/255	 6s
xfs/256	 7s
xfs/257	 8s
xfs/258	 8s
xfs/259	 7s
xfs/260	 6s
xfs/261	 4s
xfs/262	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/263	 5s
xfs/264	 12s
xfs/265	 27s
xfs/266	[not run] xfsdump not found
xfs/267	[not run] xfsdump not found
xfs/268	[not run] xfsdump not found
xfs/269	 3s
xfs/270	 5s
xfs/271	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/272	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/273	 6s
xfs/274	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/275	[not run] This test requires a valid $SCRATCH_LOGDEV
xfs/276	[not run] External volumes not in use, skipped this test
xfs/277	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/278	 6s
xfs/279	 8s
xfs/280	 4s
xfs/281	[not run] xfsdump not found
xfs/282	[not run] xfsdump not found
xfs/283	[not run] xfsdump not found
xfs/284	 5s
xfs/285	- output mismatch (see /home/ctheune/results/xfs/285.out.bad)
    --- tests/xfs/285.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/285.out.bad	2022-02-01 13:32:40.902065247 +0100
    @@ -1,4 +1,6 @@
     QA output created by 285
     Format and populate
    +Error: /mnt/scratch: Kernel metadata scrubbing facility is not available. (phase1.c line 147)
    +/mnt/scratch: operational errors found: 1
     Concurrent scrub
     Test done
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/285.out /home/ctheune/results/xfs/285.out.bad'  to see the entire diff)
xfs/286	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/287	[not run] xfsdump not found
xfs/288	 5s
xfs/289	 4s
xfs/290	 3s
xfs/291	 151s
xfs/292	 4s
xfs/293	 3s
xfs/294	 147s
xfs/295	 36s
xfs/296	[not run] xfsdump not found
xfs/297	 64s
xfs/298	 13s
xfs/299	 15s
xfs/300	[not run] SELinux not enabled
xfs/301	[not run] xfsdump not found
xfs/302	[not run] xfsdump not found
xfs/303	 2s
xfs/304	 4s
xfs/305	 63s
xfs/306	 32s
xfs/307	 7s
xfs/308	 9s
xfs/309	 27s
xfs/310	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/311	 25s
xfs/312	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/313	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/314	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/315	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/316	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/317	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/318	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/319	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/320	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/321	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/322	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/323	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/324	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/325	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/326	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/327	 5s
xfs/328	 61s
xfs/329	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/330	 5s
xfs/331	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/332	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/333	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/334	[not run] External volumes not in use, skipped this test
xfs/335	[not run] External volumes not in use, skipped this test
xfs/336	[not run] External volumes not in use, skipped this test
xfs/337	[not run] External volumes not in use, skipped this test
xfs/338	[not run] External volumes not in use, skipped this test
xfs/339	[not run] External volumes not in use, skipped this test
xfs/340	[not run] External volumes not in use, skipped this test
xfs/341	[not run] External volumes not in use, skipped this test
xfs/342	[not run] External volumes not in use, skipped this test
xfs/343	[not run] External volumes not in use, skipped this test
xfs/344	 11s
xfs/345	 6s
xfs/346	 57s
xfs/347	 10s
xfs/348	 23s
xfs/349	- output mismatch (see /home/ctheune/results/xfs/349.out.bad)
    --- tests/xfs/349.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/349.out.bad	2022-02-01 13:48:15.470845647 +0100
    @@ -1,3 +1,5 @@
     QA output created by 349
     Format and populate
     Scrub
    +Error: /mnt/scratch: Kernel metadata scrubbing facility is not available. (phase1.c line 147)
    +/mnt/scratch: operational errors found: 1
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/349.out /home/ctheune/results/xfs/349.out.bad'  to see the entire diff)
xfs/350	- output mismatch (see /home/ctheune/results/xfs/350.out.bad)
    --- tests/xfs/350.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/350.out.bad	2022-02-01 14:19:14.296063035 +0100
    @@ -1,4 +1,111 @@
     QA output created by 350
     Format and populate
     Fuzz superblock
    +re-repair failed (1) with qflags = add.
    +offline re-scrub (1) with dirblklog = lastbit.
    +re-repair failed (1) with dirblklog = lastbit.
    +offline repair failed (1) with features_compat = ones.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/350.out /home/ctheune/results/xfs/350.out.bad'  to see the entire diff)
xfs/351	- output mismatch (see /home/ctheune/results/xfs/351.out.bad)
    --- tests/xfs/351.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/351.out.bad	2022-02-01 14:48:26.464173061 +0100
    @@ -1,4 +1,1376 @@
     QA output created by 351
     Format and populate
     Fuzz superblock
    +online repair failed (4) with magicnum = zeroes.
    +offline re-scrub (1) with magicnum = zeroes.
    +online re-scrub (4) with magicnum = zeroes.
    +re-repair failed (1) with magicnum = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/351.out /home/ctheune/results/xfs/351.out.bad'  to see the entire diff)
xfs/352	- output mismatch (see /home/ctheune/results/xfs/352.out.bad)
    --- tests/xfs/352.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/352.out.bad	2022-02-01 15:09:09.645655454 +0100
    @@ -1,4 +1,182 @@
     QA output created by 352
     Format and populate
     Fuzz AGF
    +online re-scrub (4) with magicnum = zeroes.
    +online re-scrub (4) with magicnum = ones.
    +online re-scrub (4) with magicnum = firstbit.
    +online re-scrub (4) with magicnum = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/352.out /home/ctheune/results/xfs/352.out.bad'  to see the entire diff)
xfs/353	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/353.dmesg)
- output mismatch (see /home/ctheune/results/xfs/353.out.bad)
    --- tests/xfs/353.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/353.out.bad	2022-02-01 15:17:11.125042760 +0100
    @@ -1,4 +1,661 @@
     QA output created by 353
     Format and populate
     Fuzz AGF
    +mount failed (32) with magicnum = zeroes.
    +offline re-scrub (1) with magicnum = zeroes.
    +re-mount failed (32) with magicnum = zeroes.
    +re-repair failed (1) with magicnum = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/353.out /home/ctheune/results/xfs/353.out.bad'  to see the entire diff)
xfs/354	- output mismatch (see /home/ctheune/results/xfs/354.out.bad)
    --- tests/xfs/354.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/354.out.bad	2022-02-01 15:21:13.018307144 +0100
    @@ -1,6 +1,57 @@
     QA output created by 354
     Format and populate
     Fuzz AGFL
    +online re-scrub (4) with magicnum = zeroes.
    +online re-scrub (4) with magicnum = ones.
    +online re-scrub (4) with magicnum = firstbit.
    +online re-scrub (4) with magicnum = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/354.out /home/ctheune/results/xfs/354.out.bad'  to see the entire diff)
xfs/355	- output mismatch (see /home/ctheune/results/xfs/355.out.bad)
    --- tests/xfs/355.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/355.out.bad	2022-02-01 15:26:06.871022112 +0100
    @@ -1,6 +1,144 @@
     QA output created by 355
     Format and populate
     Fuzz AGFL
    +online repair failed (4) with magicnum = zeroes.
    +online re-scrub (4) with magicnum = zeroes.
    +online repair failed (4) with magicnum = ones.
    +online re-scrub (4) with magicnum = ones.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/355.out /home/ctheune/results/xfs/355.out.bad'  to see the entire diff)
xfs/356	- output mismatch (see /home/ctheune/results/xfs/356.out.bad)
    --- tests/xfs/356.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/356.out.bad	2022-02-01 15:37:29.618466657 +0100
    @@ -1,4 +1,141 @@
     QA output created by 356
     Format and populate
     Fuzz AGI
    +online re-scrub (4) with magicnum = zeroes.
    +online re-scrub (4) with magicnum = ones.
    +online re-scrub (4) with magicnum = firstbit.
    +online re-scrub (4) with magicnum = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/356.out /home/ctheune/results/xfs/356.out.bad'  to see the entire diff)
xfs/357	- output mismatch (see /home/ctheune/results/xfs/357.out.bad)
    --- tests/xfs/357.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/357.out.bad	2022-02-01 15:43:52.829264208 +0100
    @@ -1,4 +1,482 @@
     QA output created by 357
     Format and populate
     Fuzz AGI
    +mount failed (32) with magicnum = zeroes.
    +offline re-scrub (1) with magicnum = zeroes.
    +re-mount failed (32) with magicnum = zeroes.
    +re-repair failed (1) with magicnum = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/357.out /home/ctheune/results/xfs/357.out.bad'  to see the entire diff)
xfs/358	- output mismatch (see /home/ctheune/results/xfs/358.out.bad)
    --- tests/xfs/358.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/358.out.bad	2022-02-01 15:51:04.579565253 +0100
    @@ -1,4 +1,95 @@
     QA output created by 358
     Format and populate
     Fuzz bnobt recs
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/358.out /home/ctheune/results/xfs/358.out.bad'  to see the entire diff)
xfs/359	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/359.dmesg)
- output mismatch (see /home/ctheune/results/xfs/359.out.bad)
    --- tests/xfs/359.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/359.out.bad	2022-02-01 15:59:29.078048929 +0100
    @@ -1,4 +1,345 @@
     QA output created by 359
     Format and populate
     Fuzz bnobt recs
    +online repair failed (4) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +online re-scrub (4) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/359.out /home/ctheune/results/xfs/359.out.bad'  to see the entire diff)
xfs/360	- output mismatch (see /home/ctheune/results/xfs/360.out.bad)
    --- tests/xfs/360.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/360.out.bad	2022-02-01 16:07:13.678958950 +0100
    @@ -1,4 +1,103 @@
     QA output created by 360
     Format and populate
     Fuzz bnobt keyptr
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/360.out /home/ctheune/results/xfs/360.out.bad'  to see the entire diff)
xfs/361	- output mismatch (see /home/ctheune/results/xfs/361.out.bad)
    --- tests/xfs/361.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/361.out.bad	2022-02-01 16:16:43.947711293 +0100
    @@ -1,4 +1,350 @@
     QA output created by 361
     Format and populate
     Fuzz bnobt keyptr
    +online repair failed (4) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +online re-scrub (4) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/361.out /home/ctheune/results/xfs/361.out.bad'  to see the entire diff)
xfs/362	- output mismatch (see /home/ctheune/results/xfs/362.out.bad)
    --- tests/xfs/362.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/362.out.bad	2022-02-01 16:23:51.552076723 +0100
    @@ -1,4 +1,95 @@
     QA output created by 362
     Format and populate
     Fuzz cntbt
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/362.out /home/ctheune/results/xfs/362.out.bad'  to see the entire diff)
xfs/363	- output mismatch (see /home/ctheune/results/xfs/363.out.bad)
    --- tests/xfs/363.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/363.out.bad	2022-02-01 16:32:34.634373049 +0100
    @@ -1,4 +1,331 @@
     QA output created by 363
     Format and populate
     Fuzz cntbt
    +online repair failed (4) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +online re-scrub (4) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/363.out /home/ctheune/results/xfs/363.out.bad'  to see the entire diff)
xfs/364	- output mismatch (see /home/ctheune/results/xfs/364.out.bad)
    --- tests/xfs/364.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/364.out.bad	2022-02-01 16:42:42.178730778 +0100
    @@ -1,4 +1,115 @@
     QA output created by 364
     Format and populate
     Fuzz inobt
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/364.out /home/ctheune/results/xfs/364.out.bad'  to see the entire diff)
xfs/365	- output mismatch (see /home/ctheune/results/xfs/365.out.bad)
    --- tests/xfs/365.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/365.out.bad	2022-02-01 16:53:23.769559650 +0100
    @@ -1,4 +1,338 @@
     QA output created by 365
     Format and populate
     Fuzz inobt
    +online repair failed (4) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +online re-scrub (4) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/365.out /home/ctheune/results/xfs/365.out.bad'  to see the entire diff)
xfs/366	- output mismatch (see /home/ctheune/results/xfs/366.out.bad)
    --- tests/xfs/366.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/366.out.bad	2022-02-01 17:06:06.509862792 +0100
    @@ -1,4 +1,117 @@
     QA output created by 366
     Format and populate
     Fuzz finobt
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/366.out /home/ctheune/results/xfs/366.out.bad'  to see the entire diff)
xfs/367	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/367.dmesg)
- output mismatch (see /home/ctheune/results/xfs/367.out.bad)
    --- tests/xfs/367.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/367.out.bad	2022-02-01 17:11:01.517558666 +0100
    @@ -1,4 +1,446 @@
     QA output created by 367
     Format and populate
     Fuzz finobt
    +mount failed (32) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +re-mount failed (32) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/367.out /home/ctheune/results/xfs/367.out.bad'  to see the entire diff)
xfs/368	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/369	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/370	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/371	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/372	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/372.dmesg)
- output mismatch (see /home/ctheune/results/xfs/372.out.bad)
    --- tests/xfs/372.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/372.out.bad	2022-02-01 17:23:31.885199930 +0100
    @@ -1,4 +1,102 @@
     QA output created by 372
     Format and populate
     Fuzz refcountbt
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/372.out /home/ctheune/results/xfs/372.out.bad'  to see the entire diff)
xfs/373	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/373.dmesg)
- output mismatch (see /home/ctheune/results/xfs/373.out.bad)
    --- tests/xfs/373.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/373.out.bad	2022-02-01 17:27:25.275748379 +0100
    @@ -1,4 +1,378 @@
     QA output created by 373
     Format and populate
     Fuzz refcountbt
    +mount failed (32) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +re-mount failed (32) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/373.out /home/ctheune/results/xfs/373.out.bad'  to see the entire diff)
xfs/374	- output mismatch (see /home/ctheune/results/xfs/374.out.bad)
    --- tests/xfs/374.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/374.out.bad	2022-02-01 18:00:29.157305558 +0100
    @@ -2,4 +2,418 @@
     Format and populate
     Find btree-format dir inode
     Fuzz inode
    +online re-scrub (4) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = ones.
    +online re-scrub (4) with core.magic = firstbit.
    +online re-scrub (4) with core.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/374.out /home/ctheune/results/xfs/374.out.bad'  to see the entire diff)
xfs/375	- output mismatch (see /home/ctheune/results/xfs/375.out.bad)
    --- tests/xfs/375.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/375.out.bad	2022-02-01 18:27:45.864691865 +0100
    @@ -2,4 +2,1274 @@
     Format and populate
     Find btree-format dir inode
     Fuzz inode
    +online repair failed (4) with core.magic = zeroes.
    +offline re-scrub (1) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = zeroes.
    +re-repair failed (1) with core.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/375.out /home/ctheune/results/xfs/375.out.bad'  to see the entire diff)
xfs/376	- output mismatch (see /home/ctheune/results/xfs/376.out.bad)
    --- tests/xfs/376.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/376.out.bad	2022-02-01 18:59:57.062667459 +0100
    @@ -2,4 +2,418 @@
     Format and populate
     Find extents-format file inode
     Fuzz inode
    +online re-scrub (4) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = ones.
    +online re-scrub (4) with core.magic = firstbit.
    +online re-scrub (4) with core.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/376.out /home/ctheune/results/xfs/376.out.bad'  to see the entire diff)
xfs/377	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/377.dmesg)
- output mismatch (see /home/ctheune/results/xfs/377.out.bad)
    --- tests/xfs/377.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/377.out.bad	2022-02-01 19:28:21.678148459 +0100
    @@ -2,4 +2,1243 @@
     Format and populate
     Find extents-format file inode
     Fuzz inode
    +online repair failed (4) with core.magic = zeroes.
    +offline re-scrub (1) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = zeroes.
    +re-repair failed (1) with core.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/377.out /home/ctheune/results/xfs/377.out.bad'  to see the entire diff)
xfs/378	- output mismatch (see /home/ctheune/results/xfs/378.out.bad)
    --- tests/xfs/378.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/378.out.bad	2022-02-01 20:00:34.318098688 +0100
    @@ -2,4 +2,416 @@
     Format and populate
     Find btree-format file inode
     Fuzz inode
    +online re-scrub (4) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = ones.
    +online re-scrub (4) with core.magic = firstbit.
    +online re-scrub (4) with core.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/378.out /home/ctheune/results/xfs/378.out.bad'  to see the entire diff)
xfs/379	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/379.dmesg)
- output mismatch (see /home/ctheune/results/xfs/379.out.bad)
    --- tests/xfs/379.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/379.out.bad	2022-02-01 20:29:07.499276182 +0100
    @@ -2,4 +2,1273 @@
     Format and populate
     Find btree-format file inode
     Fuzz inode
    +online repair failed (4) with core.magic = zeroes.
    +offline re-scrub (1) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = zeroes.
    +re-repair failed (1) with core.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/379.out /home/ctheune/results/xfs/379.out.bad'  to see the entire diff)
xfs/380	- output mismatch (see /home/ctheune/results/xfs/380.out.bad)
    --- tests/xfs/380.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/380.out.bad	2022-02-01 20:37:21.452098166 +0100
    @@ -2,4 +2,110 @@
     Format and populate
     Find bmbt block
     Fuzz bmbt
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/380.out /home/ctheune/results/xfs/380.out.bad'  to see the entire diff)
xfs/381	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/381.dmesg)
- output mismatch (see /home/ctheune/results/xfs/381.out.bad)
    --- tests/xfs/381.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/381.out.bad	2022-02-01 20:44:19.291667214 +0100
    @@ -2,4 +2,404 @@
     Format and populate
     Find bmbt block
     Fuzz bmbt
    +online repair failed (4) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +online re-scrub (4) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/381.out /home/ctheune/results/xfs/381.out.bad'  to see the entire diff)
xfs/382	- output mismatch (see /home/ctheune/results/xfs/382.out.bad)
    --- tests/xfs/382.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/382.out.bad	2022-02-01 20:49:30.905345712 +0100
    @@ -2,4 +2,71 @@
     Format and populate
     Find symlink remote block
     Fuzz symlink remote block
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/382.out /home/ctheune/results/xfs/382.out.bad'  to see the entire diff)
xfs/383	- output mismatch (see /home/ctheune/results/xfs/383.out.bad)
    --- tests/xfs/383.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/383.out.bad	2022-02-01 20:53:52.823441335 +0100
    @@ -2,4 +2,250 @@
     Format and populate
     Find symlink remote block
     Fuzz symlink remote block
    +online repair failed (4) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +online re-scrub (4) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/383.out /home/ctheune/results/xfs/383.out.bad'  to see the entire diff)
xfs/384	- output mismatch (see /home/ctheune/results/xfs/384.out.bad)
    --- tests/xfs/384.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/384.out.bad	2022-02-01 21:31:07.658705606 +0100
    @@ -2,4 +2,487 @@
     Format and populate
     Find inline-format dir inode
     Fuzz inline-format dir inode
    +online re-scrub (4) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = ones.
    +online re-scrub (4) with core.magic = firstbit.
    +online re-scrub (4) with core.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/384.out /home/ctheune/results/xfs/384.out.bad'  to see the entire diff)
xfs/385	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/385.dmesg)
- output mismatch (see /home/ctheune/results/xfs/385.out.bad)
    --- tests/xfs/385.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/385.out.bad	2022-02-01 22:04:26.599668198 +0100
    @@ -2,4 +2,1520 @@
     Format and populate
     Find inline-format dir inode
     Fuzz inline-format dir inode
    +online repair failed (4) with core.magic = zeroes.
    +offline re-scrub (1) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = zeroes.
    +re-repair failed (1) with core.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/385.out /home/ctheune/results/xfs/385.out.bad'  to see the entire diff)
xfs/386	- output mismatch (see /home/ctheune/results/xfs/386.out.bad)
    --- tests/xfs/386.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/386.out.bad	2022-02-01 22:56:12.461187994 +0100
    @@ -2,4 +2,685 @@
     Format and populate
     Find data-format dir block
     Fuzz data-format dir block
    +online re-scrub (4) with bhdr.hdr.magic = zeroes.
    +online re-scrub (4) with bhdr.hdr.magic = ones.
    +online re-scrub (4) with bhdr.hdr.magic = firstbit.
    +online re-scrub (4) with bhdr.hdr.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/386.out /home/ctheune/results/xfs/386.out.bad'  to see the entire diff)
xfs/387	- output mismatch (see /home/ctheune/results/xfs/387.out.bad)
    --- tests/xfs/387.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/387.out.bad	2022-02-01 23:45:55.092090237 +0100
    @@ -2,4 +2,2608 @@
     Format and populate
     Find data-format dir block
     Fuzz data-format dir block
    +online repair failed (4) with bhdr.hdr.magic = zeroes.
    +offline re-scrub (1) with bhdr.hdr.magic = zeroes.
    +online re-scrub (4) with bhdr.hdr.magic = zeroes.
    +re-repair failed (1) with bhdr.hdr.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/387.out /home/ctheune/results/xfs/387.out.bad'  to see the entire diff)
xfs/388	- output mismatch (see /home/ctheune/results/xfs/388.out.bad)
    --- tests/xfs/388.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/388.out.bad	2022-02-02 00:20:15.869717503 +0100
    @@ -2,4 +2,882 @@
     Format and populate
     Find data-format dir block
     Fuzz data-format dir block
    +online re-scrub (4) with dhdr.hdr.magic = zeroes.
    +online re-scrub (4) with dhdr.hdr.magic = ones.
    +online re-scrub (4) with dhdr.hdr.magic = firstbit.
    +online re-scrub (4) with dhdr.hdr.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/388.out /home/ctheune/results/xfs/388.out.bad'  to see the entire diff)
xfs/389	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/389.dmesg)
- output mismatch (see /home/ctheune/results/xfs/389.out.bad)
    --- tests/xfs/389.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/389.out.bad	2022-02-02 00:54:25.682300056 +0100
    @@ -2,4 +2,1910 @@
     Format and populate
     Find data-format dir block
     Fuzz data-format dir block
    +online repair failed (4) with dhdr.hdr.magic = zeroes.
    +offline re-scrub (1) with dhdr.hdr.magic = zeroes.
    +online re-scrub (4) with dhdr.hdr.magic = zeroes.
    +re-repair failed (1) with dhdr.hdr.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/389.out /home/ctheune/results/xfs/389.out.bad'  to see the entire diff)
xfs/390	- output mismatch (see /home/ctheune/results/xfs/390.out.bad)
    --- tests/xfs/390.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/390.out.bad	2022-02-02 01:15:01.555835077 +0100
    @@ -2,4 +2,255 @@
     Format and populate
     Find leaf1-format dir block
     Fuzz leaf1-format dir block
    +online re-scrub (4) with lhdr.info.hdr.forw = ones.
    +online re-scrub (4) with lhdr.info.hdr.forw = firstbit.
    +online re-scrub (4) with lhdr.info.hdr.forw = middlebit.
    +online re-scrub (4) with lhdr.info.hdr.forw = lastbit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/390.out /home/ctheune/results/xfs/390.out.bad'  to see the entire diff)
xfs/391	- output mismatch (see /home/ctheune/results/xfs/391.out.bad)
    --- tests/xfs/391.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/391.out.bad	2022-02-02 02:21:46.360456689 +0100
    @@ -2,4 +2,992 @@
     Format and populate
     Find leaf1-format dir block
     Fuzz leaf1-format dir block
    +online repair failed (4) with lhdr.info.hdr.forw = ones.
    +offline re-scrub (1) with lhdr.info.hdr.forw = ones.
    +online re-scrub (4) with lhdr.info.hdr.forw = ones.
    +re-repair failed (1) with lhdr.info.hdr.forw = ones.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/391.out /home/ctheune/results/xfs/391.out.bad'  to see the entire diff)
xfs/392	- output mismatch (see /home/ctheune/results/xfs/392.out.bad)
    --- tests/xfs/392.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/392.out.bad	2022-02-02 02:39:53.273685037 +0100
    @@ -2,4 +2,237 @@
     Format and populate
     Find leafn-format dir block
     Fuzz leafn-format dir block
    +online re-scrub (4) with lhdr.info.hdr.forw = zeroes.
    +online re-scrub (4) with lhdr.info.hdr.forw = ones.
    +online re-scrub (4) with lhdr.info.hdr.forw = firstbit.
    +online re-scrub (4) with lhdr.info.hdr.forw = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/392.out /home/ctheune/results/xfs/392.out.bad'  to see the entire diff)
xfs/393	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/393.dmesg)
- output mismatch (see /home/ctheune/results/xfs/393.out.bad)
    --- tests/xfs/393.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/393.out.bad	2022-02-02 02:56:13.691060255 +0100
    @@ -2,4 +2,906 @@
     Format and populate
     Find leafn-format dir block
     Fuzz leafn-format dir block
    +online repair failed (4) with lhdr.info.hdr.forw = zeroes.
    +offline re-scrub (1) with lhdr.info.hdr.forw = zeroes.
    +online re-scrub (4) with lhdr.info.hdr.forw = zeroes.
    +re-repair failed (1) with lhdr.info.hdr.forw = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/393.out /home/ctheune/results/xfs/393.out.bad'  to see the entire diff)
xfs/394	- output mismatch (see /home/ctheune/results/xfs/394.out.bad)
    --- tests/xfs/394.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/394.out.bad	2022-02-02 03:03:22.640407064 +0100
    @@ -2,4 +2,96 @@
     Format and populate
     Find node-format dir block
     Fuzz node-format dir block
    +online re-scrub (4) with nhdr.info.hdr.forw = ones.
    +online re-scrub (4) with nhdr.info.hdr.forw = firstbit.
    +online re-scrub (4) with nhdr.info.hdr.forw = middlebit.
    +online re-scrub (4) with nhdr.info.hdr.forw = lastbit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/394.out /home/ctheune/results/xfs/394.out.bad'  to see the entire diff)
xfs/395	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/395.dmesg)
- output mismatch (see /home/ctheune/results/xfs/395.out.bad)
    --- tests/xfs/395.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/395.out.bad	2022-02-02 03:09:41.191299940 +0100
    @@ -2,4 +2,342 @@
     Format and populate
     Find node-format dir block
     Fuzz node-format dir block
    +online repair failed (4) with nhdr.info.hdr.forw = ones.
    +offline re-scrub (1) with nhdr.info.hdr.forw = ones.
    +online re-scrub (4) with nhdr.info.hdr.forw = ones.
    +re-repair failed (1) with nhdr.info.hdr.forw = ones.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/395.out /home/ctheune/results/xfs/395.out.bad'  to see the entire diff)
xfs/396	- output mismatch (see /home/ctheune/results/xfs/396.out.bad)
    --- tests/xfs/396.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/396.out.bad	2022-02-02 03:15:43.935364049 +0100
    @@ -2,4 +2,81 @@
     Format and populate
     Find freeindex-format dir block
     Fuzz freeindex-format dir block
    +online re-scrub (4) with fhdr.hdr.magic = zeroes.
    +online re-scrub (4) with fhdr.hdr.magic = ones.
    +online re-scrub (4) with fhdr.hdr.magic = firstbit.
    +online re-scrub (4) with fhdr.hdr.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/396.out /home/ctheune/results/xfs/396.out.bad'  to see the entire diff)
xfs/397	- output mismatch (see /home/ctheune/results/xfs/397.out.bad)
    --- tests/xfs/397.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/397.out.bad	2022-02-02 03:20:43.895109289 +0100
    @@ -2,4 +2,296 @@
     Format and populate
     Find freeindex-format dir block
     Fuzz freeindex-format dir block
    +online repair failed (4) with fhdr.hdr.magic = zeroes.
    +offline re-scrub (1) with fhdr.hdr.magic = zeroes.
    +online re-scrub (4) with fhdr.hdr.magic = zeroes.
    +re-repair failed (1) with fhdr.hdr.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/397.out /home/ctheune/results/xfs/397.out.bad'  to see the entire diff)
xfs/398	- output mismatch (see /home/ctheune/results/xfs/398.out.bad)
    --- tests/xfs/398.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/398.out.bad	2022-02-02 03:58:34.267468776 +0100
    @@ -2,4 +2,495 @@
     Format and populate
     Find inline-format attr inode
     Fuzz inline-format attr inode
    +online re-scrub (4) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = ones.
    +online re-scrub (4) with core.magic = firstbit.
    +online re-scrub (4) with core.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/398.out /home/ctheune/results/xfs/398.out.bad'  to see the entire diff)
xfs/399	- output mismatch (see /home/ctheune/results/xfs/399.out.bad)
    --- tests/xfs/399.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/399.out.bad	2022-02-02 04:30:16.665693173 +0100
    @@ -2,4 +2,1496 @@
     Format and populate
     Find inline-format attr inode
     Fuzz inline-format attr inode
    +online repair failed (4) with core.magic = zeroes.
    +offline re-scrub (1) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = zeroes.
    +re-repair failed (1) with core.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/399.out /home/ctheune/results/xfs/399.out.bad'  to see the entire diff)
xfs/400	- output mismatch (see /home/ctheune/results/xfs/400.out.bad)
    --- tests/xfs/400.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/400.out.bad	2022-02-02 05:06:20.550250670 +0100
    @@ -2,4 +2,474 @@
     Format and populate
     Find leaf-format attr block
     Fuzz leaf-format attr block
    +online re-scrub (4) with hdr.info.hdr.forw = ones.
    +online re-scrub (4) with hdr.info.hdr.forw = firstbit.
    +online re-scrub (4) with hdr.info.hdr.forw = middlebit.
    +online re-scrub (4) with hdr.info.hdr.forw = lastbit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/400.out /home/ctheune/results/xfs/400.out.bad'  to see the entire diff)
xfs/401	- output mismatch (see /home/ctheune/results/xfs/401.out.bad)
    --- tests/xfs/401.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/401.out.bad	2022-02-02 05:36:34.422505073 +0100
    @@ -2,4 +2,1620 @@
     Format and populate
     Find leaf-format attr block
     Fuzz leaf-format attr block
    +online repair failed (4) with hdr.info.hdr.forw = ones.
    +offline re-scrub (1) with hdr.info.hdr.forw = ones.
    +online re-scrub (4) with hdr.info.hdr.forw = ones.
    +re-repair failed (1) with hdr.info.hdr.forw = ones.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/401.out /home/ctheune/results/xfs/401.out.bad'  to see the entire diff)
xfs/402	- output mismatch (see /home/ctheune/results/xfs/402.out.bad)
    --- tests/xfs/402.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/402.out.bad	2022-02-02 05:43:37.332901217 +0100
    @@ -2,4 +2,96 @@
     Format and populate
     Find node-format attr block
     Fuzz node-format attr block
    +online re-scrub (4) with hdr.info.hdr.forw = ones.
    +online re-scrub (4) with hdr.info.hdr.forw = firstbit.
    +online re-scrub (4) with hdr.info.hdr.forw = middlebit.
    +online re-scrub (4) with hdr.info.hdr.forw = lastbit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/402.out /home/ctheune/results/xfs/402.out.bad'  to see the entire diff)
xfs/403	- output mismatch (see /home/ctheune/results/xfs/403.out.bad)
    --- tests/xfs/403.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/403.out.bad	2022-02-02 05:49:35.388005590 +0100
    @@ -2,4 +2,340 @@
     Format and populate
     Find node-format attr block
     Fuzz node-format attr block
    +online repair failed (4) with hdr.info.hdr.forw = ones.
    +offline re-scrub (1) with hdr.info.hdr.forw = ones.
    +online re-scrub (4) with hdr.info.hdr.forw = ones.
    +re-repair failed (1) with hdr.info.hdr.forw = ones.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/403.out /home/ctheune/results/xfs/403.out.bad'  to see the entire diff)
xfs/404	- output mismatch (see /home/ctheune/results/xfs/404.out.bad)
    --- tests/xfs/404.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/404.out.bad	2022-02-02 05:54:46.149625304 +0100
    @@ -2,4 +2,70 @@
     Format and populate
     Find external attr block
     Fuzz external attr block
    +online re-scrub (4) with hdr.magic = zeroes.
    +online re-scrub (4) with hdr.magic = ones.
    +online re-scrub (4) with hdr.magic = firstbit.
    +online re-scrub (4) with hdr.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/404.out /home/ctheune/results/xfs/404.out.bad'  to see the entire diff)
xfs/405	- output mismatch (see /home/ctheune/results/xfs/405.out.bad)
    --- tests/xfs/405.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/405.out.bad	2022-02-02 05:59:04.817830516 +0100
    @@ -2,4 +2,212 @@
     Format and populate
     Find external attr block
     Fuzz external attr block
    +online repair failed (4) with hdr.magic = zeroes.
    +offline re-scrub (1) with hdr.magic = zeroes.
    +online re-scrub (4) with hdr.magic = zeroes.
    +re-repair failed (1) with hdr.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/405.out /home/ctheune/results/xfs/405.out.bad'  to see the entire diff)
xfs/406	[not run] External volumes not in use, skipped this test
xfs/407	[not run] External volumes not in use, skipped this test
xfs/408	[not run] External volumes not in use, skipped this test
xfs/409	[not run] External volumes not in use, skipped this test
xfs/410	- output mismatch (see /home/ctheune/results/xfs/410.out.bad)
    --- tests/xfs/410.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/410.out.bad	2022-02-02 06:07:33.546323943 +0100
    @@ -1,4 +1,110 @@
     QA output created by 410
     Format and populate
     Fuzz refcountbt
    +online re-scrub (4) with magic = zeroes.
    +online re-scrub (4) with magic = ones.
    +online re-scrub (4) with magic = firstbit.
    +online re-scrub (4) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/410.out /home/ctheune/results/xfs/410.out.bad'  to see the entire diff)
xfs/411	- output mismatch (see /home/ctheune/results/xfs/411.out.bad)
    --- tests/xfs/411.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/411.out.bad	2022-02-02 06:14:24.829862540 +0100
    @@ -1,4 +1,398 @@
     QA output created by 411
     Format and populate
     Fuzz refcountbt
    +online repair failed (4) with magic = zeroes.
    +offline re-scrub (1) with magic = zeroes.
    +online re-scrub (4) with magic = zeroes.
    +re-repair failed (1) with magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/411.out /home/ctheune/results/xfs/411.out.bad'  to see the entire diff)
xfs/412	- output mismatch (see /home/ctheune/results/xfs/412.out.bad)
    --- tests/xfs/412.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/412.out.bad	2022-02-02 06:46:41.044999631 +0100
    @@ -2,4 +2,419 @@
     Format and populate
     Find btree-format attr inode
     Fuzz inode
    +online re-scrub (4) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = ones.
    +online re-scrub (4) with core.magic = firstbit.
    +online re-scrub (4) with core.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/412.out /home/ctheune/results/xfs/412.out.bad'  to see the entire diff)
xfs/413	- output mismatch (see /home/ctheune/results/xfs/413.out.bad)
    --- tests/xfs/413.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/413.out.bad	2022-02-02 07:14:57.270629197 +0100
    @@ -2,4 +2,1322 @@
     Format and populate
     Find btree-format attr inode
     Fuzz inode
    +online repair failed (4) with core.magic = zeroes.
    +offline re-scrub (1) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = zeroes.
    +re-repair failed (1) with core.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/413.out /home/ctheune/results/xfs/413.out.bad'  to see the entire diff)
xfs/414	- output mismatch (see /home/ctheune/results/xfs/414.out.bad)
    --- tests/xfs/414.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/414.out.bad	2022-02-02 07:45:14.411789578 +0100
    @@ -2,4 +2,392 @@
     Format and populate
     Find blockdev inode
     Fuzz inode
    +online re-scrub (4) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = ones.
    +online re-scrub (4) with core.magic = firstbit.
    +online re-scrub (4) with core.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/414.out /home/ctheune/results/xfs/414.out.bad'  to see the entire diff)
xfs/415	- output mismatch (see /home/ctheune/results/xfs/415.out.bad)
    --- tests/xfs/415.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/415.out.bad	2022-02-02 08:10:29.916205224 +0100
    @@ -2,4 +2,1220 @@
     Format and populate
     Find blockdev inode
     Fuzz inode
    +online repair failed (4) with core.magic = zeroes.
    +offline re-scrub (1) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = zeroes.
    +re-repair failed (1) with core.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/415.out /home/ctheune/results/xfs/415.out.bad'  to see the entire diff)
xfs/416	- output mismatch (see /home/ctheune/results/xfs/416.out.bad)
    --- tests/xfs/416.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/416.out.bad	2022-02-02 08:40:43.787566548 +0100
    @@ -2,4 +2,392 @@
     Format and populate
     Find local-format symlink inode
     Fuzz inode
    +online re-scrub (4) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = ones.
    +online re-scrub (4) with core.magic = firstbit.
    +online re-scrub (4) with core.magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/416.out /home/ctheune/results/xfs/416.out.bad'  to see the entire diff)
xfs/417	- output mismatch (see /home/ctheune/results/xfs/417.out.bad)
    --- tests/xfs/417.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/417.out.bad	2022-02-02 09:06:07.091985295 +0100
    @@ -2,4 +2,1234 @@
     Format and populate
     Find local-format symlink inode
     Fuzz inode
    +online repair failed (4) with core.magic = zeroes.
    +offline re-scrub (1) with core.magic = zeroes.
    +online re-scrub (4) with core.magic = zeroes.
    +re-repair failed (1) with core.magic = zeroes.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/417.out /home/ctheune/results/xfs/417.out.bad'  to see the entire diff)
xfs/418	- output mismatch (see /home/ctheune/results/xfs/418.out.bad)
    --- tests/xfs/418.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/418.out.bad	2022-02-02 09:37:28.159442034 +0100
    @@ -1,4 +1,414 @@
     QA output created by 418
     Format and populate
     Fuzz superblock
    +online re-scrub (4) with magicnum = zeroes.
    +online re-scrub (4) with magicnum = ones.
    +online re-scrub (4) with magicnum = firstbit.
    +online re-scrub (4) with magicnum = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/418.out /home/ctheune/results/xfs/418.out.bad'  to see the entire diff)
xfs/419	[not run] External volumes not in use, skipped this test
xfs/420	 5s
xfs/421	 6s
xfs/422	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/423	[not run] xfs_io scrub support is missing
xfs/424	 7s
xfs/425	[not run] user quota disabled
xfs/426	[not run] user quota disabled
xfs/427	[not run] group quota disabled
xfs/428	[not run] group quota disabled
xfs/429	[not run] project quota disabled
xfs/430	[not run] project quota disabled
xfs/431	 4s
xfs/432	 29s
xfs/433	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/434	[not run] Require module xfs to be unloadable
xfs/435	[not run] Require module xfs to be unloadable
xfs/436	[not run] Require module xfs to be unloadable
xfs/437	[not run] Can't run find-api-violations.sh without WORKAREA set
xfs/438	[not run] fsgqa user not defined.
xfs/439	 4s
xfs/440	[not run] fsgqa user not defined.
xfs/441	[not run] fsgqa user not defined.
xfs/442	 440s
xfs/443	 6s
xfs/444	 30s
xfs/445	 100s
xfs/446	[not run] checkbashisms utility required, skipped this test
xfs/447	[not run] sysfs attribute 'debug/mount_delay' is not supported
xfs/448	[not run] xfs_io scrub support is missing
xfs/449	 4s
xfs/450	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/451	 6s
xfs/452	 6s
xfs/453	 1098s
xfs/454	- output mismatch (see /home/ctheune/results/xfs/454.out.bad)
    --- tests/xfs/454.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/454.out.bad	2022-02-02 10:15:38.920422937 +0100
    @@ -1,4 +1,116 @@
     QA output created by 454
     Format and populate
     Fuzz AGF
    +re-mount failed (32) with magicnum = zeroes.
    +re-mount failed (32) with magicnum = ones.
    +re-mount failed (32) with magicnum = firstbit.
    +re-mount failed (32) with magicnum = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/454.out /home/ctheune/results/xfs/454.out.bad'  to see the entire diff)
xfs/455	 227s
xfs/456	- output mismatch (see /home/ctheune/results/xfs/456.out.bad)
    --- tests/xfs/456.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/456.out.bad	2022-02-02 10:23:04.286564370 +0100
    @@ -1,4 +1,83 @@
     QA output created by 456
     Format and populate
     Fuzz AGI
    +re-mount failed (32) with magicnum = zeroes.
    +re-mount failed (32) with magicnum = ones.
    +re-mount failed (32) with magicnum = firstbit.
    +re-mount failed (32) with magicnum = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/456.out /home/ctheune/results/xfs/456.out.bad'  to see the entire diff)
xfs/457	 364s
xfs/458	 392s
xfs/459	 348s
xfs/460	 252s
xfs/461	- output mismatch (see /home/ctheune/results/xfs/461.out.bad)
    --- tests/xfs/461.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/461.out.bad	2022-02-02 10:49:08.963527657 +0100
    @@ -1,4 +1,70 @@
     QA output created by 461
     Format and populate
     Fuzz finobt
    +re-mount failed (32) with magic = zeroes.
    +re-mount failed (32) with magic = ones.
    +re-mount failed (32) with magic = firstbit.
    +re-mount failed (32) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/461.out /home/ctheune/results/xfs/461.out.bad'  to see the entire diff)
xfs/462	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/463	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/464	- output mismatch (see /home/ctheune/results/xfs/464.out.bad)
    --- tests/xfs/464.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/464.out.bad	2022-02-02 10:51:23.092066243 +0100
    @@ -1,4 +1,80 @@
     QA output created by 464
     Format and populate
     Fuzz refcountbt
    +re-mount failed (32) with magic = zeroes.
    +re-mount failed (32) with magic = ones.
    +re-mount failed (32) with magic = firstbit.
    +re-mount failed (32) with magic = middlebit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/464.out /home/ctheune/results/xfs/464.out.bad'  to see the entire diff)
xfs/465	 1146s
xfs/466	 1170s
xfs/467	 1188s
xfs/468	 282s
xfs/469	 180s
xfs/470	 1372s
xfs/471	 1778s
xfs/472	 1317s
xfs/473	 3461s
xfs/474	 648s
xfs/475	 251s
xfs/476	 200s
xfs/477	 1310s
xfs/478	 1219s
xfs/479	 241s
xfs/480	 175s
xfs/481	[not run] External volumes not in use, skipped this test
xfs/482	[not run] External volumes not in use, skipped this test
xfs/483	 279s
xfs/484	 1125s
xfs/485	 1080s
xfs/486	 1069s
xfs/487	[not run] user quota disabled
xfs/488	[not run] group quota disabled
xfs/489	[not run] project quota disabled
xfs/490	 7s
xfs/491	 7s
xfs/492	 6s
xfs/493	 6s
xfs/494	 3s
xfs/495	 8s
xfs/496	- output mismatch (see /home/ctheune/results/xfs/496.out.bad)
    --- tests/xfs/496.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/496.out.bad	2022-02-02 16:35:43.485528370 +0100
    @@ -2,4 +2,253 @@
     Format and populate
     Find single-leafn-format dir block
     Fuzz single-leafn-format dir block
    +online re-scrub (4) with lhdr.info.hdr.forw = ones.
    +re-repair failed (1) with lhdr.info.hdr.forw = ones.
    +online re-scrub (4) with lhdr.info.hdr.forw = firstbit.
    +re-repair failed (1) with lhdr.info.hdr.forw = firstbit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/496.out /home/ctheune/results/xfs/496.out.bad'  to see the entire diff)
xfs/497	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/497.dmesg)
- output mismatch (see /home/ctheune/results/xfs/497.out.bad)
    --- tests/xfs/497.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/497.out.bad	2022-02-02 17:13:11.251902108 +0100
    @@ -2,4 +2,916 @@
     Format and populate
     Find single-leafn-format dir block
     Fuzz single-leafn-format dir block
    +online repair failed (4) with lhdr.info.hdr.forw = ones.
    +online re-scrub (4) with lhdr.info.hdr.forw = ones.
    +re-repair failed (1) with lhdr.info.hdr.forw = ones.
    +online repair failed (4) with lhdr.info.hdr.forw = firstbit.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/497.out /home/ctheune/results/xfs/497.out.bad'  to see the entire diff)
xfs/498	_check_dmesg: something found in dmesg (see /home/ctheune/results/xfs/498.dmesg)

xfs/499	 4s
xfs/500	 3s
xfs/501	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/502	[not run] XFS error injection requires CONFIG_XFS_DEBUG
xfs/503	 25s
xfs/504	 27s
xfs/505	 3s
xfs/506	- output mismatch (see /home/ctheune/results/xfs/506.out.bad)
    --- tests/xfs/506.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/506.out.bad	2022-02-02 17:45:15.583211371 +0100
    @@ -1,3 +1,17 @@
     QA output created by 506
     Health status has not been collected for this filesystem.
     Please run xfs_scrub(8) to remedy this situation.
    +Error: /mnt/scratch: Kernel metadata scrubbing facility is not available. (phase1.c line 147)
    +/mnt/scratch: operational errors found: 1
    +Health status has not been collected for this filesystem.
    +Please run xfs_scrub(8) to remedy this situation.
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/506.out /home/ctheune/results/xfs/506.out.bad'  to see the entire diff)
xfs/507	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/508	 4s
xfs/509	 50s
xfs/510	 4s
xfs/511	 4s
xfs/512	[not run] fsgqa user not defined.
xfs/513	- output mismatch (see /home/ctheune/results/xfs/513.out.bad)
    --- tests/xfs/513.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/513.out.bad	2022-02-02 17:46:45.776231380 +0100
    @@ -77,7 +77,11 @@
     TEST: "-o usrquota" "pass" "usrquota" "true"
     TEST: "-o quota" "pass" "usrquota" "true"
     TEST: "-o uqnoenforce" "pass" "uqnoenforce" "true"
    +[FAILED]: mount /dev/loop0 /mnt/test/513.mnt -o uqnoenforce
    +ERROR: did not expect to find "uqnoenforce" in "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,usrquota"
     TEST: "-o qnoenforce" "pass" "uqnoenforce" "true"
    +[FAILED]: mount /dev/loop0 /mnt/test/513.mnt -o qnoenforce
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/513.out /home/ctheune/results/xfs/513.out.bad'  to see the entire diff)
xfs/514	 4s
xfs/515	 3s
xfs/516	 27s
xfs/517	[not run] rmapbt not supported by scratch filesystem type: xfs
xfs/518	 5s
xfs/519	 3s
xfs/520	 9s
xfs/521	 5s
xfs/522	 5s
xfs/523	 4s
xfs/524	 5s
xfs/525	 5s
xfs/526	 4s
xfs/527	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/528	 7s
xfs/529	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/530	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/531	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/532	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/533	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/534	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/535	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/536	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/537	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/538	[not run] Require XFS built with CONFIG_XFS_DEBUG
xfs/539	- output mismatch (see /home/ctheune/results/xfs/539.out.bad)
    --- tests/xfs/539.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/539.out.bad	2022-02-02 17:48:53.138846974 +0100
    @@ -1,2 +1,4 @@
     QA output created by 539
     Silence is golden.
    +Should not be able to find deprecation warning for attr2
    +Should not be able to find deprecation warning for noikeep
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/539.out /home/ctheune/results/xfs/539.out.bad'  to see the entire diff)
xfs/540	- output mismatch (see /home/ctheune/results/xfs/540.out.bad)
    --- tests/xfs/540.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/540.out.bad	2022-02-02 17:49:00.612765718 +0100
    @@ -2,4 +2,5 @@
     Format and mount
     Misconfigure the root directory
     Detect misconfigured directory
    +repair did not catch error?
     Repair misconfigured directory
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/540.out /home/ctheune/results/xfs/540.out.bad'  to see the entire diff)
xfs/541	[not run] External volumes not in use, skipped this test
xfs/542	_check_xfs_filesystem: filesystem on /dev/vda4 is inconsistent (r)
(see /home/ctheune/results/xfs/542.full for details)
- output mismatch (see /home/ctheune/results/xfs/542.out.bad)
    --- tests/xfs/542.out	1970-01-01 01:00:01.000000000 +0100
    +++ /home/ctheune/results/xfs/542.out.bad	2022-02-02 17:49:08.393681121 +0100
    @@ -1,2 +1,3 @@
     QA output created by 542
     sync_file_range: Input/output error
    +cp: failed to clone '/mnt/scratch/file3' from '/mnt/scratch/file2': Structure needs cleaning
    ...
    (Run 'diff -u /tmp/xfstests.wVf4eh/tests/xfs/542.out /home/ctheune/results/xfs/542.out.bad'  to see the entire diff)
Ran: generic/001 generic/002 generic/003 generic/004 generic/005 generic/006 generic/007 generic/008 generic/009 generic/010 generic/011 generic/012 generic/013 generic/014 generic/015 generic/016 generic/017 generic/018 generic/019 generic/020 generic/021 generic/022 generic/023 generic/024 generic/025 generic/026 generic/027 generic/028 generic/029 generic/030 generic/031 generic/032 generic/033 generic/034 generic/035 generic/036 generic/037 generic/038 generic/039 generic/040 generic/041 generic/042 generic/043 generic/044 generic/045 generic/046 generic/047 generic/048 generic/049 generic/050 generic/051 generic/052 generic/053 generic/054 generic/055 generic/056 generic/057 generic/058 generic/059 generic/060 generic/061 generic/062 generic/063 generic/064 generic/065 generic/066 generic/067 generic/068 generic/069 generic/070 generic/071 generic/072 generic/073 generic/074 generic/075 generic/076 generic/077 generic/078 generic/079 generic/080 generic/081 generic/082 generic/083 generic/084 generic/085 generic/086 generic/087 generic/088 generic/089 generic/090 generic/091 generic/092 generic/093 generic/094 generic/095 generic/096 generic/097 generic/098 generic/099 generic/100 generic/101 generic/102 generic/103 generic/104 generic/105 generic/106 generic/107 generic/108 generic/109 generic/110 generic/111 generic/112 generic/113 generic/114 generic/115 generic/116 generic/117 generic/118 generic/119 generic/120 generic/121 generic/122 generic/123 generic/124 generic/125 generic/126 generic/127 generic/128 generic/129 generic/130 generic/131 generic/132 generic/133 generic/134 generic/135 generic/136 generic/137 generic/138 generic/139 generic/140 generic/141 generic/142 generic/143 generic/144 generic/145 generic/146 generic/147 generic/148 generic/149 generic/150 generic/151 generic/152 generic/153 generic/154 generic/155 generic/156 generic/157 generic/158 generic/159 generic/160 generic/161 generic/162 generic/163 generic/164 generic/165 generic/166 generic/167 generic/168 generic/169 generic/170 generic/171 generic/172 generic/173 generic/174 generic/175 generic/176 generic/177 generic/178 generic/179 generic/180 generic/181 generic/182 generic/183 generic/184 generic/185 generic/186 generic/187 generic/188 generic/189 generic/190 generic/191 generic/192 generic/193 generic/194 generic/195 generic/196 generic/197 generic/198 generic/199 generic/200 generic/201 generic/202 generic/203 generic/204 generic/205 generic/206 generic/207 generic/208 generic/209 generic/210 generic/211 generic/212 generic/213 generic/214 generic/215 generic/216 generic/217 generic/218 generic/219 generic/220 generic/221 generic/222 generic/223 generic/224 generic/225 generic/226 generic/227 generic/228 generic/229 generic/230 generic/231 generic/232 generic/233 generic/234 generic/235 generic/236 generic/237 generic/238 generic/239 generic/240 generic/241 generic/242 generic/243 generic/244 generic/245 generic/246 generic/247 generic/248 generic/249 generic/250 generic/251 generic/252 generic/253 generic/254 generic/255 generic/256 generic/257 generic/258 generic/259 generic/260 generic/261 generic/262 generic/263 generic/264 generic/265 generic/266 generic/267 generic/268 generic/269 generic/270 generic/271 generic/272 generic/273 generic/274 generic/275 generic/276 generic/277 generic/278 generic/279 generic/280 generic/281 generic/282 generic/283 generic/284 generic/285 generic/286 generic/287 generic/288 generic/289 generic/290 generic/291 generic/292 generic/293 generic/294 generic/295 generic/296 generic/297 generic/298 generic/299 generic/300 generic/301 generic/302 generic/303 generic/304 generic/305 generic/306 generic/307 generic/308 generic/309 generic/310 generic/311 generic/312 generic/313 generic/314 generic/315 generic/316 generic/317 generic/318 generic/319 generic/320 generic/321 generic/322 generic/323 generic/324 generic/325 generic/326 generic/327 generic/328 generic/329 generic/330 generic/331 generic/332 generic/333 generic/334 generic/335 generic/336 generic/337 generic/338 generic/339 generic/340 generic/341 generic/342 generic/343 generic/344 generic/345 generic/346 generic/347 generic/348 generic/349 generic/350 generic/351 generic/352 generic/353 generic/354 generic/355 generic/356 generic/357 generic/358 generic/359 generic/360 generic/361 generic/362 generic/363 generic/364 generic/365 generic/366 generic/367 generic/368 generic/369 generic/370 generic/371 generic/372 generic/373 generic/374 generic/375 generic/376 generic/377 generic/378 generic/379 generic/380 generic/381 generic/382 generic/383 generic/384 generic/385 generic/386 generic/387 generic/388 generic/389 generic/390 generic/391 generic/392 generic/393 generic/394 generic/395 generic/396 generic/397 generic/398 generic/399 generic/400 generic/401 generic/402 generic/403 generic/404 generic/405 generic/406 generic/407 generic/408 generic/409 generic/410 generic/411 generic/412 generic/413 generic/414 generic/415 generic/416 generic/417 generic/418 generic/419 generic/420 generic/421 generic/422 generic/423 generic/424 generic/425 generic/426 generic/427 generic/428 generic/429 generic/430 generic/431 generic/432 generic/433 generic/434 generic/435 generic/436 generic/437 generic/438 generic/439 generic/440 generic/441 generic/442 generic/443 generic/444 generic/445 generic/446 generic/447 generic/448 generic/449 generic/450 generic/451 generic/452 generic/453 generic/454 generic/455 generic/456 generic/457 generic/458 generic/459 generic/460 generic/461 generic/462 generic/463 generic/464 generic/465 generic/466 generic/467 generic/468 generic/469 generic/470 generic/471 generic/472 generic/473 generic/474 generic/475 generic/476 generic/477 generic/478 generic/479 generic/480 generic/481 generic/482 generic/483 generic/484 generic/485 generic/486 generic/487 generic/488 generic/489 generic/490 generic/491 generic/492 generic/493 generic/494 generic/495 generic/496 generic/497 generic/498 generic/499 generic/500 generic/501 generic/502 generic/503 generic/504 generic/505 generic/506 generic/507 generic/508 generic/509 generic/510 generic/511 generic/512 generic/513 generic/514 generic/515 generic/516 generic/517 generic/518 generic/519 generic/520 generic/521 generic/522 generic/523 generic/524 generic/525 generic/526 generic/527 generic/528 generic/529 generic/530 generic/531 generic/532 generic/533 generic/534 generic/535 generic/536 generic/537 generic/538 generic/539 generic/540 generic/541 generic/542 generic/543 generic/544 generic/545 generic/546 generic/547 generic/548 generic/549 generic/550 generic/551 generic/552 generic/553 generic/554 generic/555 generic/556 generic/557 generic/558 generic/559 generic/560 generic/561 generic/562 generic/563 generic/564 generic/565 generic/566 generic/567 generic/568 generic/569 generic/570 generic/571 generic/572 generic/573 generic/574 generic/575 generic/576 generic/577 generic/578 generic/579 generic/580 generic/581 generic/582 generic/583 generic/584 generic/585 generic/586 generic/587 generic/588 generic/589 generic/590 generic/591 generic/592 generic/593 generic/594 generic/595 generic/596 generic/597 generic/598 generic/599 generic/600 generic/601 generic/602 generic/603 generic/604 generic/605 generic/606 generic/607 generic/608 generic/609 generic/610 generic/611 generic/612 generic/613 generic/614 generic/615 generic/616 generic/617 generic/618 generic/619 generic/620 generic/621 generic/622 generic/623 generic/624 generic/625 generic/626 generic/627 generic/628 generic/629 generic/630 generic/631 generic/632 generic/633 generic/634 generic/635 generic/636 generic/637 generic/638 generic/639 generic/640 generic/641 generic/642 generic/643 generic/644 generic/645 generic/646 generic/647 generic/648 generic/649 generic/650 generic/651 generic/652 generic/653 generic/654 generic/655 generic/656 generic/657 generic/658 generic/659 generic/660 generic/661 generic/662 generic/663 generic/664 generic/665 generic/666 generic/667 generic/668 generic/669 generic/670 generic/671 generic/672 shared/002 shared/032 shared/298 xfs/001 xfs/002 xfs/003 xfs/004 xfs/005 xfs/006 xfs/007 xfs/008 xfs/009 xfs/010 xfs/011 xfs/012 xfs/013 xfs/014 xfs/015 xfs/016 xfs/017 xfs/018 xfs/019 xfs/020 xfs/021 xfs/022 xfs/023 xfs/024 xfs/025 xfs/026 xfs/027 xfs/028 xfs/029 xfs/030 xfs/031 xfs/032 xfs/033 xfs/034 xfs/035 xfs/036 xfs/037 xfs/038 xfs/039 xfs/040 xfs/041 xfs/042 xfs/043 xfs/044 xfs/045 xfs/046 xfs/047 xfs/048 xfs/049 xfs/050 xfs/051 xfs/052 xfs/053 xfs/054 xfs/055 xfs/056 xfs/057 xfs/058 xfs/059 xfs/060 xfs/061 xfs/062 xfs/063 xfs/064 xfs/065 xfs/066 xfs/067 xfs/068 xfs/069 xfs/070 xfs/071 xfs/072 xfs/073 xfs/074 xfs/075 xfs/076 xfs/077 xfs/078 xfs/079 xfs/080 xfs/081 xfs/082 xfs/083 xfs/084 xfs/085 xfs/086 xfs/087 xfs/088 xfs/089 xfs/090 xfs/091 xfs/092 xfs/093 xfs/094 xfs/095 xfs/096 xfs/097 xfs/098 xfs/099 xfs/100 xfs/101 xfs/102 xfs/103 xfs/104 xfs/105 xfs/106 xfs/108 xfs/109 xfs/110 xfs/111 xfs/112 xfs/113 xfs/114 xfs/115 xfs/116 xfs/117 xfs/118 xfs/119 xfs/120 xfs/121 xfs/122 xfs/123 xfs/124 xfs/125 xfs/126 xfs/127 xfs/128 xfs/129 xfs/130 xfs/131 xfs/132 xfs/133 xfs/134 xfs/135 xfs/136 xfs/137 xfs/138 xfs/139 xfs/140 xfs/141 xfs/142 xfs/143 xfs/144 xfs/145 xfs/146 xfs/147 xfs/148 xfs/149 xfs/150 xfs/151 xfs/152 xfs/153 xfs/154 xfs/155 xfs/156 xfs/157 xfs/158 xfs/159 xfs/160 xfs/161 xfs/162 xfs/163 xfs/164 xfs/165 xfs/166 xfs/167 xfs/168 xfs/169 xfs/170 xfs/171 xfs/172 xfs/173 xfs/174 xfs/175 xfs/176 xfs/177 xfs/178 xfs/179 xfs/180 xfs/181 xfs/182 xfs/183 xfs/184 xfs/185 xfs/186 xfs/187 xfs/188 xfs/189 xfs/190 xfs/191-input-validation xfs/192 xfs/193 xfs/194 xfs/195 xfs/196 xfs/197 xfs/198 xfs/199 xfs/200 xfs/201 xfs/202 xfs/203 xfs/204 xfs/205 xfs/206 xfs/207 xfs/208 xfs/209 xfs/210 xfs/211 xfs/212 xfs/213 xfs/214 xfs/215 xfs/216 xfs/217 xfs/218 xfs/219 xfs/220 xfs/221 xfs/222 xfs/223 xfs/224 xfs/225 xfs/226 xfs/227 xfs/228 xfs/229 xfs/230 xfs/231 xfs/232 xfs/233 xfs/234 xfs/235 xfs/236 xfs/237 xfs/238 xfs/239 xfs/240 xfs/241 xfs/242 xfs/243 xfs/244 xfs/245 xfs/246 xfs/247 xfs/248 xfs/249 xfs/250 xfs/251 xfs/252 xfs/253 xfs/254 xfs/255 xfs/256 xfs/257 xfs/258 xfs/259 xfs/260 xfs/261 xfs/262 xfs/263 xfs/264 xfs/265 xfs/266 xfs/267 xfs/268 xfs/269 xfs/270 xfs/271 xfs/272 xfs/273 xfs/274 xfs/275 xfs/276 xfs/277 xfs/278 xfs/279 xfs/280 xfs/281 xfs/282 xfs/283 xfs/284 xfs/285 xfs/286 xfs/287 xfs/288 xfs/289 xfs/290 xfs/291 xfs/292 xfs/293 xfs/294 xfs/295 xfs/296 xfs/297 xfs/298 xfs/299 xfs/300 xfs/301 xfs/302 xfs/303 xfs/304 xfs/305 xfs/306 xfs/307 xfs/308 xfs/309 xfs/310 xfs/311 xfs/312 xfs/313 xfs/314 xfs/315 xfs/316 xfs/317 xfs/318 xfs/319 xfs/320 xfs/321 xfs/322 xfs/323 xfs/324 xfs/325 xfs/326 xfs/327 xfs/328 xfs/329 xfs/330 xfs/331 xfs/332 xfs/333 xfs/334 xfs/335 xfs/336 xfs/337 xfs/338 xfs/339 xfs/340 xfs/341 xfs/342 xfs/343 xfs/344 xfs/345 xfs/346 xfs/347 xfs/348 xfs/349 xfs/350 xfs/351 xfs/352 xfs/353 xfs/354 xfs/355 xfs/356 xfs/357 xfs/358 xfs/359 xfs/360 xfs/361 xfs/362 xfs/363 xfs/364 xfs/365 xfs/366 xfs/367 xfs/368 xfs/369 xfs/370 xfs/371 xfs/372 xfs/373 xfs/374 xfs/375 xfs/376 xfs/377 xfs/378 xfs/379 xfs/380 xfs/381 xfs/382 xfs/383 xfs/384 xfs/385 xfs/386 xfs/387 xfs/388 xfs/389 xfs/390 xfs/391 xfs/392 xfs/393 xfs/394 xfs/395 xfs/396 xfs/397 xfs/398 xfs/399 xfs/400 xfs/401 xfs/402 xfs/403 xfs/404 xfs/405 xfs/406 xfs/407 xfs/408 xfs/409 xfs/410 xfs/411 xfs/412 xfs/413 xfs/414 xfs/415 xfs/416 xfs/417 xfs/418 xfs/419 xfs/420 xfs/421 xfs/422 xfs/423 xfs/424 xfs/425 xfs/426 xfs/427 xfs/428 xfs/429 xfs/430 xfs/431 xfs/432 xfs/433 xfs/434 xfs/435 xfs/436 xfs/437 xfs/438 xfs/439 xfs/440 xfs/441 xfs/442 xfs/443 xfs/444 xfs/445 xfs/446 xfs/447 xfs/448 xfs/449 xfs/450 xfs/451 xfs/452 xfs/453 xfs/454 xfs/455 xfs/456 xfs/457 xfs/458 xfs/459 xfs/460 xfs/461 xfs/462 xfs/463 xfs/464 xfs/465 xfs/466 xfs/467 xfs/468 xfs/469 xfs/470 xfs/471 xfs/472 xfs/473 xfs/474 xfs/475 xfs/476 xfs/477 xfs/478 xfs/479 xfs/480 xfs/481 xfs/482 xfs/483 xfs/484 xfs/485 xfs/486 xfs/487 xfs/488 xfs/489 xfs/490 xfs/491 xfs/492 xfs/493 xfs/494 xfs/495 xfs/496 xfs/497 xfs/498 xfs/499 xfs/500 xfs/501 xfs/502 xfs/503 xfs/504 xfs/505 xfs/506 xfs/507 xfs/508 xfs/509 xfs/510 xfs/511 xfs/512 xfs/513 xfs/514 xfs/515 xfs/516 xfs/517 xfs/518 xfs/519 xfs/520 xfs/521 xfs/522 xfs/523 xfs/524 xfs/525 xfs/526 xfs/527 xfs/528 xfs/529 xfs/530 xfs/531 xfs/532 xfs/533 xfs/534 xfs/535 xfs/536 xfs/537 xfs/538 xfs/539 xfs/540 xfs/541 xfs/542
Not run: generic/010 generic/019 generic/077 generic/079 generic/093 generic/123 generic/125 generic/128 generic/193 generic/219 generic/230 generic/231 generic/233 generic/235 generic/241 generic/256 generic/270 generic/305 generic/314 generic/317 generic/326 generic/327 generic/328 generic/355 generic/362 generic/363 generic/364 generic/365 generic/366 generic/367 generic/368 generic/369 generic/370 generic/378 generic/381 generic/382 generic/395 generic/396 generic/397 generic/398 generic/399 generic/413 generic/419 generic/421 generic/429 generic/435 generic/440 generic/455 generic/457 generic/459 generic/462 generic/470 generic/482 generic/487 generic/514 generic/548 generic/549 generic/550 generic/556 generic/559 generic/560 generic/561 generic/566 generic/570 generic/572 generic/573 generic/574 generic/575 generic/576 generic/577 generic/579 generic/580 generic/581 generic/582 generic/583 generic/584 generic/587 generic/592 generic/593 generic/595 generic/596 generic/597 generic/598 generic/600 generic/601 generic/602 generic/603 generic/605 generic/606 generic/608 generic/613 generic/616 generic/617 generic/621 generic/624 generic/625 generic/633 generic/644 generic/645 generic/656 xfs/014 xfs/022 xfs/023 xfs/024 xfs/025 xfs/026 xfs/027 xfs/028 xfs/035 xfs/036 xfs/037 xfs/038 xfs/039 xfs/040 xfs/043 xfs/044 xfs/046 xfs/047 xfs/051 xfs/055 xfs/056 xfs/057 xfs/059 xfs/060 xfs/061 xfs/063 xfs/064 xfs/065 xfs/066 xfs/068 xfs/090 xfs/094 xfs/096 xfs/106 xfs/114 xfs/122 xfs/131 xfs/141 xfs/142 xfs/143 xfs/146 xfs/147 xfs/152 xfs/153 xfs/161 xfs/163 xfs/168 xfs/176 xfs/186 xfs/187 xfs/189 xfs/195 xfs/196 xfs/197 xfs/213 xfs/214 xfs/233 xfs/234 xfs/235 xfs/236 xfs/243 xfs/244 xfs/245 xfs/246 xfs/262 xfs/266 xfs/267 xfs/268 xfs/271 xfs/272 xfs/274 xfs/275 xfs/276 xfs/277 xfs/281 xfs/282 xfs/283 xfs/286 xfs/287 xfs/296 xfs/300 xfs/301 xfs/302 xfs/310 xfs/312 xfs/313 xfs/314 xfs/315 xfs/316 xfs/317 xfs/318 xfs/319 xfs/320 xfs/321 xfs/322 xfs/323 xfs/324 xfs/325 xfs/326 xfs/329 xfs/331 xfs/332 xfs/333 xfs/334 xfs/335 xfs/336 xfs/337 xfs/338 xfs/339 xfs/340 xfs/341 xfs/342 xfs/343 xfs/368 xfs/369 xfs/370 xfs/371 xfs/406 xfs/407 xfs/408 xfs/409 xfs/419 xfs/422 xfs/423 xfs/425 xfs/426 xfs/427 xfs/428 xfs/429 xfs/430 xfs/433 xfs/434 xfs/435 xfs/436 xfs/437 xfs/438 xfs/440 xfs/441 xfs/446 xfs/447 xfs/448 xfs/450 xfs/462 xfs/463 xfs/481 xfs/482 xfs/487 xfs/488 xfs/489 xfs/501 xfs/502 xfs/507 xfs/512 xfs/517 xfs/527 xfs/529 xfs/530 xfs/531 xfs/532 xfs/533 xfs/534 xfs/535 xfs/536 xfs/537 xfs/538 xfs/541
Failures: generic/035 generic/050 generic/452 generic/473 generic/594 generic/623 generic/646 xfs/018 xfs/031 xfs/033 xfs/071 xfs/081 xfs/082 xfs/136 xfs/154 xfs/158 xfs/172 xfs/177 xfs/185 xfs/285 xfs/349 xfs/350 xfs/351 xfs/352 xfs/353 xfs/354 xfs/355 xfs/356 xfs/357 xfs/358 xfs/359 xfs/360 xfs/361 xfs/362 xfs/363 xfs/364 xfs/365 xfs/366 xfs/367 xfs/372 xfs/373 xfs/374 xfs/375 xfs/376 xfs/377 xfs/378 xfs/379 xfs/380 xfs/381 xfs/382 xfs/383 xfs/384 xfs/385 xfs/386 xfs/387 xfs/388 xfs/389 xfs/390 xfs/391 xfs/392 xfs/393 xfs/394 xfs/395 xfs/396 xfs/397 xfs/398 xfs/399 xfs/400 xfs/401 xfs/402 xfs/403 xfs/404 xfs/405 xfs/410 xfs/411 xfs/412 xfs/413 xfs/414 xfs/415 xfs/416 xfs/417 xfs/418 xfs/454 xfs/456 xfs/461 xfs/464 xfs/496 xfs/497 xfs/498 xfs/506 xfs/513 xfs/539 xfs/540 xfs/542
Failed 94 of 1216 tests


[-- Attachment #1.3: Type: text/plain, Size: 321 bytes --]



Kind regards and stay safe,
Christian

--
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Help deciding about backported patch (kernel bug 214767, 19f4e7cc8197 xfs: Fix CIL throttle hang when CIL space used going backwards)
  2022-02-17  9:22 Help deciding about backported patch (kernel bug 214767, 19f4e7cc8197 xfs: Fix CIL throttle hang when CIL space used going backwards) Christian Theune
@ 2022-02-19 21:14 ` Dave Chinner
  2022-02-23  6:17   ` Christian Theune
  0 siblings, 1 reply; 3+ messages in thread
From: Dave Chinner @ 2022-02-19 21:14 UTC (permalink / raw)
  To: Christian Theune; +Cc: linux-xfs

On Thu, Feb 17, 2022 at 10:22:49AM +0100, Christian Theune wrote:
> Hi,
> 
> I’ve been debugging an elusive XFS issue that I could not track
> down to any other parameters than it being an xfs internal bug.
> I’ve recorded what I’ve seen so far in
> https://bugzilla.kernel.org/show_bug.cgi?id=214767 and Dave
> recommended that "19f4e7cc8197 xfs: Fix CIL throttle hang when CIL
> space used going backwards” is likely the issue. AFAICT this was
> not backported to the 5.10 branch and we’ve been updating to
> vanilla kernels diligently and still keep seeing this issue.
> Unfortunately within a fleet of around 1k VMs it strikes about
> once every week or so and there’s no way to predict when and
> where.
> 
> So, I took Dave’s pointer and applied the patch to our 5.10 series
> (basd on 5.10.76 at that point) and it applied cleanly. The
> machine boots fine and I ran the XFS test suite. However, I
> haven’t done any tests using the test suite before and I’m getting
> a number of errors where I don’t know how to interpret the
> results. Some of those seem to be due to not having the DEBUG flag
> set in the kernel, others … I’m not sure.

Run the "auto" group tests ('-g auto') only, which will weed out
tests that are broken, likely to fail or crash the machine (i.e.
test-to-failure scenarios). You can ignore "not run" reports - they
aren't failures, just indicative of the kernel not supporting that
functionality (like not being built with DEBUG functionality).

Then run the tests across an unmodified kernel 2-3 times to get a
baseline set of results (should be identical each run), then do the
same thing for the patched kernel.

Now compare baseline vs patched results, looking for things that
failed in the patched kernel that didn't fail in the baseline kernel
- those are the regressions that need more investigation. If there
are no regressions (very likely), you are good to go.

> I’m attaching the test runner output, unfortunately I lost the
> actual outputs as the test ran quite long and the outputs where
> cleaned up by the tempfile watcher faster than I could retrieve
> them. I can run them again, my estimation currently is it takes
> around 3-4 days to complete them, though.

The auto group tests should take ~3-6 hours to run a full cycle
depending on storage config.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Help deciding about backported patch (kernel bug 214767, 19f4e7cc8197 xfs: Fix CIL throttle hang when CIL space used going backwards)
  2022-02-19 21:14 ` Dave Chinner
@ 2022-02-23  6:17   ` Christian Theune
  0 siblings, 0 replies; 3+ messages in thread
From: Christian Theune @ 2022-02-23  6:17 UTC (permalink / raw)
  To: Dave Chinner; +Cc: linux-xfs

[-- Attachment #1: Type: text/plain, Size: 5148 bytes --]

Hi Dave,

thanks a lot - those are the instructions I was missing!

Unfortunately I mixed up the results directories and the test baseline output tat goes to the /tmp directory, so I don’t have the true full diffs available, only what’s in the stdout log. In any case, here’s what I found:

Baseline
--------

On my vanilla kernel (5.10.76) I get between 18 and 20 test failures when running auto mode as you instructed. The affected tests on vanilla are:

generic/035 generic/050 generic/388 generic/452 generic/594 generic/623 generic/646 generic/670* xfs/031 xfs/033 xfs/071 xfs/154 xfs/158 xfs/177 xfs/185 xfs/506 xfs/513 xfs/539 xfs/540 xfs/542

Notable here is generic/670 which only failed in the baseline but not the patched kernel:

generic/670 10s ... - output mismatch (see /home/ctheune/fc-nixos/results/generic/670.out.bad)
     Reflink and mmap reread the files!
    +61 61 61 61 61 61 61 61 61 62 62 62 62 62 62 62
     Finished reflinking

Patched
-------

On my patched kernel I get between 18 and 22 test failures. I’m listing only the ones not failing in the baseline:

generic/471
    -RWF_NOWAIT time is within limits.
    +RWF_NOWAIT took 0.2517 seconds

generic/475
     Silence is golden.
    +your 131072x1 screen size is bogus. expect trouble
    +your 131072x1 screen size is bogus. expect trouble
    +your 131072x1 screen size is bogus. expect trouble
    +your 131072x1 screen size is bogus. expect trouble
    +your 131072x1 screen size is bogus. expect trouble
    ...

generic/648
     Silence is golden.
    +your 131072x1 screen size is bogus. expect trouble

Also notable is that only the second run on the patched kernel contained additional failures compared to the baseline. The first and third run were “clean” compared to the baseline.

I’m guessing that the “screen size is bogus” messages are due to me running the tests in a ‘screen’ + sudo environment. Leaves the generic/471 which doesn’t sound too bad, but honestly I have no idea … :)

Also, I’d be happy to pay back a bit by adding your instructions to the documentation or wiki (or wherever googling has a higher chance of finding them).

Kind regards,
Christian

> On 19. Feb 2022, at 22:14, Dave Chinner <david@fromorbit.com> wrote:
> 
> On Thu, Feb 17, 2022 at 10:22:49AM +0100, Christian Theune wrote:
>> Hi,
>> 
>> I’ve been debugging an elusive XFS issue that I could not track
>> down to any other parameters than it being an xfs internal bug.
>> I’ve recorded what I’ve seen so far in
>> https://bugzilla.kernel.org/show_bug.cgi?id=214767 and Dave
>> recommended that "19f4e7cc8197 xfs: Fix CIL throttle hang when CIL
>> space used going backwards” is likely the issue. AFAICT this was
>> not backported to the 5.10 branch and we’ve been updating to
>> vanilla kernels diligently and still keep seeing this issue.
>> Unfortunately within a fleet of around 1k VMs it strikes about
>> once every week or so and there’s no way to predict when and
>> where.
>> 
>> So, I took Dave’s pointer and applied the patch to our 5.10 series
>> (basd on 5.10.76 at that point) and it applied cleanly. The
>> machine boots fine and I ran the XFS test suite. However, I
>> haven’t done any tests using the test suite before and I’m getting
>> a number of errors where I don’t know how to interpret the
>> results. Some of those seem to be due to not having the DEBUG flag
>> set in the kernel, others … I’m not sure.
> 
> Run the "auto" group tests ('-g auto') only, which will weed out
> tests that are broken, likely to fail or crash the machine (i.e.
> test-to-failure scenarios). You can ignore "not run" reports - they
> aren't failures, just indicative of the kernel not supporting that
> functionality (like not being built with DEBUG functionality).
> 
> Then run the tests across an unmodified kernel 2-3 times to get a
> baseline set of results (should be identical each run), then do the
> same thing for the patched kernel.
> 
> Now compare baseline vs patched results, looking for things that
> failed in the patched kernel that didn't fail in the baseline kernel
> - those are the regressions that need more investigation. If there
> are no regressions (very likely), you are good to go.
> 
>> I’m attaching the test runner output, unfortunately I lost the
>> actual outputs as the test ran quite long and the outputs where
>> cleaned up by the tempfile watcher faster than I could retrieve
>> them. I can run them again, my estimation currently is it takes
>> around 3-4 days to complete them, though.
> 
> The auto group tests should take ~3-6 hours to run a full cycle
> depending on storage config.
> 
> Cheers,
> 
> Dave.
> --
> Dave Chinner
> david@fromorbit.com

Liebe Grüße,
Christian Theune

--
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2022-02-23  6:17 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-17  9:22 Help deciding about backported patch (kernel bug 214767, 19f4e7cc8197 xfs: Fix CIL throttle hang when CIL space used going backwards) Christian Theune
2022-02-19 21:14 ` Dave Chinner
2022-02-23  6:17   ` Christian Theune

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).